Think of it as a set phrase.
If you make the comment 'You had to be there!' you're saying that it was a unique experience. The full form of the sentence is something like 'You had to be there to appreciate it' or 'You had to be there to know how funny it was'. Or, in other words, someone who hadn't been there couldn't imagine what it was like.
Yes, this expression is used fairly frequently, especially if someone is recounting an experience that was funny or embarrassing or surprising in some way. When you realise that simple words can't explain what it was like, you can say 'You had to be there!'.