It means something like to try to. A similar expression exists in English, 'out to' in the phrase 'he is out to get you', but you can't really use it in as many situations as in Dutch I think. This phrase means something like 'if someone is trying to be witty/funny, once in a while they might lie just a bit', but it is difficult to translate it literally.

I am not 100% certain, but looking it up, the translation seems to be "out." So, the first part would be something like "If someone is out," which I assume means "to allow to be known." This is only my guess, based on context, but I think the whole phrase means: "If someone tells you that they are witty, then it is likely that they are lying just a bit." I'll leave it to a native speaker to answer this definitively, but that is how it seems to me. :)