Tricky here means complicated and controversial. Ross is about to discuss the history of monogamy, which didn't exist until only the past few thousand years. Despite the fact that monogamy is seen as normal, monogamy was (and some would argue still is) a very odd and very recent social construct. However, the vast majority of Americans practice and aspire to monogamy and some might find the idea of questioning monogamy to be an uninteresting exercise, hence the characters "pretend to fall asleep." Rather, Ross is ABOUT to say that monogamy is tricky because for the millions of years leading up to modern humanity, monogamy did not really exist, thus monogamy is a social convention invented relatively recently. Furthermore, there is plenty of evidence that paints the original styles of monogamy as abusive, manipulative, demeaning, and so on.. Even modern monogamy is under siege in today's world with increasing rates of divorce in many countries. This is what Ross means by ticky.

Its use in this dialogue is meant to convey sarcasm.

Yes, 'tricky' in the sense of a difficult, awkward subject. In other words, they feel uncomfortable and want to avoid talking about this any more. That's why they pretend to fall asleep.

It's misspelled. It should be "tricky" and it means it's difficult or controversial.