"Are you a friend of Rachel's?" is correct. It's what we call a double genitive. Look at this:
a friend of mine
a friend of yours
a friend of his
a friend of hers
a friend of ours
a friend of theirs.
As you can see, in each case we use the possessive pronoun after 'a .. of ...'. We don't say 'a friend of me' or a 'friend of him', for example, and we don't usually say 'a friend of Rachel': we use the genitive (possessive) form of the name.
"Are you Rachel's friend?" is also correct, but it has a slightly different meaning. We would say this when we are referring to a specific friend who has already been mentioned. For example, let's say Rachel had sent a friend of hers to call at your house. When a stranger knocks on your door, you'd then say "Are you Rachel's friend?". This means 'the particular friend that Rachel told me about'.