Rather than - shows preference. This expression is generally used in 'parallel' structures. e.g - with two nouns, adjectives, adverbs, infinitives or -ing forms.
e.g
1). We ought to invest in machinery rather than buildings.
2). I prefer starting early rather than leaving things to the last minute.
When the main clause has a to - infinitive, rather than is usually followed by an infinitive without to or -ing form.
e.g - I decided to write rather than phone/phoning.
Instead of - suggests that one person, thing or action replaces another. Instead is not used alone as a preposition; we use the two words instead of.
Instead of is not usually followed by an infinitive.
e.g
1). I'll have tea instead of coffee, please.
2). I stayed in bed all day instead of going to work.
3). John was invited to the reception, but he was ill, so Jimmy went instead of him.
Note - Instead (without of) is an adverb. It begins or ends a clause usually.
e.g - She didn't go to Greece after all. Instead , she went to America.