Both are things that are commonly done.
A convention is the way something is normally done in regard to a procedure.
eg. The convention at a restaurant is that you eat your food, and then you pay.
A custom is a common way of doing something in regard to a society or culture.
eg. In the United States, the custom is to tip at least 15% of the total bill at a restaurant.
Does that clear things up?