The use of 'would' is hypothetical or conditional. You would use it in sentences like this:
I would go with you if I could invite a friend.
I wouldn't go with you if you were the last man on earth! (harsh)
Therefore, the correct response is: 'I don't want to go with you.' (harsh)
However, people (at least in the cultures I know personally) do not reject someone so directly. They're more likely to make an excuse like, "Oh, sorry, I have to study for an exam/have dinner with my parents, etc. OR Thanks, but I've already made plans, maybe another time.