If your friend was a native English speaker, I would suggest phrases like, "I can't make it" or "I have to beg off." However, if you are talking to a non-native speaker, "cancel" is fine (it isn't too formal, and isn't reserved for Business English).