Either is fine, but I think "with you" is more natural. Wait and see how much agreement or disagreement there is amongst English speakers.
for "mad" it tends to be when you are really exceptionally angry.
for "angry" it tends to be when less annoyed, but you will have to judge by the tone of voice, and the level of shouting and saucepan flying.