How did we drift?
We drifted apart.
Apart is modifying the verb "drift", so it is an adverb.
Alternatively, you could think of "drift apart" as a phrasal verb, so that "apart" has no meaning by itself.
"far" is modifying "apart". Adverbs modify verbs, adjectives, AND other adverbs, so "far" is an adverb.