It's polite to explain why you are unable to accept an invitation or offer. For example: I'd love to but I'm busy, I'm visiting my sister this weekend.