If you treat them both as nouns, there's a neat little way to know which to use.
Look at the other words and match them based on where they come from: Latin-based words with the Latin-based "response", and Germanic-based words with the Germanic "answer". So you provide a response to an invitiation, but you make an answer to a call. Good writers often choose their collocations based on the history of the words.
A simpler explanation is that you'd use "response" in more formal situations and "answer" in common, familiar situations.