"angry with me " is correct, because your object is animate.

You can be angry with somebody (or an animal) You can be angry at a situation. He was angry with Fred for being late. He was angry at having been kept waiting.