A course is a class that you sign up for. It covers a specific topic and lasts for a specific period of time (one semester, for example) — "Art of the High Renaissance," or "Introduction to Photography". A subject is a general field of study — art history, chemistry, European history, etc.
Sometimes they are synonyms, especially when you are talking in general about your classes:
- I am taking five subjects this semester. / I am taking five courses this semester.
- I failed every subject in my first semester / I failed every course in my first semester.
In both of those sentences, "course" is more common.