When the verb comes directly after the word 'rather', it has to be in the base form (bare infinitive). This is when the subject which the verb refers to is also the subject of the sentence: Would you rather stay at home? Yes, I'd rather stay at home. We use the past form (which is actually a subjunctive indicating an unreal situation) when there is another noun between the word 'rather' and the verb. This is when the verb refers to a person or thing which is different from the subject of the sentence: I'd rather YOU stayed at home = I don't want you to go out.

Would you rather stay at home? (asking now, about the future; or generally, about habits.) Would you rather have stayed at home? (asking after the event: did you regret not staying at home?) Yes, I would rather have stayed at home.