In this case, "to" suggests some kind of transition, and it goes with certain verbs such as go, come, give, talk, and so on. The phrase "come at" exists, and it means "attack". ("Come at me, bro!!") "At" suggests a point or target, and goes with verbs such as look, laugh, throw and so on.