Yes it means the early years of the 20th Century. Typically it would be from 1901 to about 1925 or a bit earlier. 1999 would be in the "later years of the 20th Century".

The turn of the 20th century is the beginning of the 20th century.

20th century begins January 1, 1901. "Shortly after" the turn of the 20th century generally means anytime after this period to about 1905 as you say. But 'shortly after' is very subjective - I would even extend it myself another year or two until 1907 or so. Also, I think the more time passes, the longer this period 'shortly after' becomes. If it were 1910, I would personally say shortly after the turn of the 20th century was 1901-1903 or so. As the years pass I allow a little more, but that is me.

Oh, really? Doesn't it refer to periods like 1901, 1902?

Actually I know for a fact what this means, as I use this expression myself very very often. It means the end of 20th Century and beginning of the 21st. To be exact it means something like December 1999 to January 2000.