They both mean "yes." However, "yep" is modern and colloquial, and Americans use it everyday. It is a very casual way of saying "yes." On the other hand, "aye" is an old-fashioned way of saying "yes." You would sound like a character from a Jane Austen novel if you said "aye" in everyday speech. It is, however, still used in parliamentary procedure, as in "all in favor, say 'aye'."

In informal speech, most Americans will typically use "yeah" instead of "yep", although there may be some regional variations in that usage. "yes" is used in formal writing (excluding "parliamentary procedure" pointed about above) and by many people all the time during informal speech too. You can hardly ever go wrong with "yes!" Except for usage by politicians in their work, "aye" is such an unusual word that if you said it, the average person might think that you meant "eye".

yep = yes aye = yes (used by sailors)

Aye is still used by the scots and the Irish in everyday speech, but their speech is universally sneered upon.