'Aim' can be very complex because it may be a general thing/ability (I have good aim, in general), or a specific thing after the fact (He took his shot, but his aim was bad). Sometimes 'shot' will be better to use as it's a closely related concept. For your example: 'I tried to shoot him with my toy gun, but I had bad aim.' Or: ",but my aim was bad." Both are fine. Instead of aim, you could say: "But I had a bad shot," that will basically mean the same, the outcome of the process whether the 'shot' itself or the 'aim' that led up to the shot was bad. For 'took,' this verb will be used with 'shot' when discussing an 'after-the-fact' thing. So "He took his shot" is more of a perfective whereas "He took his aim" is describing more of a process that led up 'to the shot.' For 'too much aim' said by the YouTuber, this is more or less him playing with words. Maybe he took too long to aim or he didn't automatically aim/track the target quick enough and it led to a bad result. Normally 'too much aim' isn't really a thing it's more of creative word play or a sort of memetic slang. 'Aim' is not a quantity/countable, but instead it is discussed with regard to quality...so the YouTuber was being creative.

In regard to your #1 option, you would say: “I tried to shoot him with my toy gun but I had really bad aim” - or - “I tried to shoot him with my toy gun but I had really poor aim”. The #2 option, in all of its variations, is incorrect. A (casual) slang option (in the US) would be: “I tried to shoot him with my toy gun but my aim sucks.” (Natives don’t always use correct grammar) 🌈 Nicky