The words aren't synonyms, but they are often (but not always) interchangeable.
If something is common, examples of it are numerous easy to find. It's the opposite of 'rare'.
If something is ordinary, it's not special. If something's common, then each example's also ordinary by definition.
For example, apples are very common ie. there are lots of apples and they're easy to find. If an apple is just like any other apple, then it's an ordinary apple. You couldn't say that a single apple is common (outside of British(?) slang, where it can mean 'cheap').
If I said "apples are ordinary", I wouldn't be saying that apples are common, but that apples aren't special compared to other kinds of fruit (or some other context specific category).