Often used when talking of love,
You belong to me is a possessive statement, for example: Your my wife, you belong to me, your mine.(negative)
You belong with me - example: You belong with me, I belong with you, we belong with each other - I want us to be together for ever.
You belong with me is often used in song lyrics when expressing love to another person.(positive)