It would sound better if you flipped it round like "They are not my world, you are."

It is a bit strange, but 'not they' is grammatically correct, strictly. However, in normal speech, you would never hear it. People would always say 'not them'. Use 'not them'. I can guarantee nobody will say anything.

You are my world not they. is correct grammatically. In actual speech most of the people tend to use 'them'. There has been a big debate going on in what is correct :D P.S. Just change the word order and see :D They are my world not you !