Patrick, from the point of view of an English-based learner, the most efficient mindset is NOT to think of 去 and 来 as complements of direction. Two important things to remember about Chinese grammar are:
1. Knowing the part of speech (whether something is a noun, a verb, a gerund, etc) is not essential to the understanding of meaning. Chinese children do not learn Chinese with concepts such as "complements of direction". So most of the time you can, and should, forget about this kind of terminology. The important thing is to grasp the essential meaning of each word.
2. Chinese grammar is not necessarily sequential or linear like English grammar. It is more like pieces of fruits in a salad bowl. Once you have grasped the meaning of every word, you have cracked the sentence. In this way it is more like Latin.
So, for these two words, you only need to know:
去: going; moving away from oneself.
e.g. 回家去 = go home (move away from here towards home)
去年= last year (the year that has moved away from me; it's gone)
去掉 = get rid of (remove it; take it away)
来: coming; coming towards oneself.
e.g. 过来! = Come here!
来年 = next year (the coming year)
来往 = keeping in touch socially; vehicular traffic (the essence of this is "coming and going")
These two words are grammatical chameleons and multi-functional: adjective, adverb, verb etc.
It is very important to dispense with the concept of linear sequential linkage of words. Just remember the salad bowl concept.
At this stage of your progress, just concentrate on the comprehension. Don't bother to learn how to construct sentences yet.
Use the official HSK textbooks. Don't use any of the Chinese grammar textbooks written in English by so-called sinologists, whether from Yale or Harvard or other universities; these really are not so helpful.