The use of 'would' is hypothetical or conditional. You would use it in sentences like this: I would go with you if I could invite a friend. I wouldn't go with you if you were the last man on earth! (harsh) Therefore, the correct response is: 'I don't want to go with you.' (harsh) However, people (at least in the cultures I know personally) do not reject someone so directly. They're more likely to make an excuse like, "Oh, sorry, I have to study for an exam/have dinner with my parents, etc. OR Thanks, but I've already made plans, maybe another time.

thank you very much!谢谢！

Yikes, either way it's a harsh rejection! Thank you for the invite, but I already have other plans. Sorry, but I don't really want to go with you to the prom. Both of your examples are perfectly correct. I don't want to go with you. = Very clear, direct and no-nonsense. I wouldn't like to go with you. = On its own this sounds a little strange, but as a direct answer to the question as phrased, this answer sounds just fine. It is perhaps a little less harsh than the first response, but really is equally unequivocal.