It's not really impolite; just a little too direct.
The reasonable way to ask someone's name is to offer your name. "Hello, my name's Yancey." The other person would offer their name in return.
If you missed the chance when you first met, or if the other person already knows your name, then there are a few other phrases you can use:
"Sorry, I didn't catch your name earlier."
"I'm afraid I don't know your name. How may I call you?" (quite polite)