You could also say - "don't worry about me" or "I do worry about you"
Someone's sake implies more of a sense of their well-being though. Perhaps this is a more accurate alternative - "don't worry about my well-being".
Does that make sense? I hope so - it's quite hard to explain written down! :)