A direct translation of "I'm angry with you" would be "watashi-wa okotte imasu" or "watashi-wa ikatte imasu," but it'd sound very strange if you actually said them to a person. Among friends, you might say "atamani kichau," meaning "I am upset." In a slightly more matured conversation, you might say "mattaku haradatashii," meaning "it's upsetting." As for "You are being silly," you might say "baka mitai," meaning "you are foolish." You also might say "waru fuzake ga sugiru wayo," meaning "you are being too silly." Just, please be careful when you use negative phrases against Japanese people if you don't know them very well unless you wish to end your relationship with them. Direct exchange of negative expressions in public is not well received, at least in my generation (yes, I'm old) .