That's half the fun is an expression you say when someone doesn't understand why you are doing something. For example, when someone says "I would never learn a second language because I don't like learning new words." You could say "But that's half the fun!" There are many other reasons why you like learning a second language, but because they commented on that one aspect, this can be a deflection of sorts. Something you can say with a smile without seeming to argue with the person who doesn't understand.
I hope this helps!