You could say the person jostled you. But "jostle" is a fairly unusual word and not that common in conversation.
Another choice is, "He brushed by/past me," which is less strong.
If it is very crowded you can say, "He squeezed past/by me."
If the action is stronger, close to a push, "He elbowed by me."
I think "shove" is as strong as push. "Nudge" often means to touch someone to get their attention.