Think of it as a set phrase. If you make the comment 'You had to be there!' you're saying that it was a unique experience. The full form of the sentence is something like 'You had to be there to appreciate it' or 'You had to be there to know how funny it was'. Or, in other words, someone who hadn't been there couldn't imagine what it was like. Yes, this expression is used fairly frequently, especially if someone is recounting an experience that was funny or embarrassing or surprising in some way. When you realise that simple words can't explain what it was like, you can say 'You had to be there!'.

"had to" is similar to modal verbs which express possibility, suggestion, certainty, obligation and necessity. -I had to clean my room. (necessity or obligation) -You had to be there/It was necessary for you to be there.

"Had to" is the past form for "Have to" As you can see that the use is in Past Tense, right? The meaning of "have to" is more or less the same like "must" in a degree of necessity/obligation. But please notice that they ('have to' and 'must') are used differently depending on who imposes the obligation.

It's a way of saying that you wouldn't understand the story or joke unless you were there at the time. Hence "You had to have been there" or the more colloquial "You had to be there".