It doesn't really mean anything. Like many Family Guy quotes, this is a made-up phrase. That program is full of made-up phrases, obscure references, and euphemisms! In this case, I think that Peter has Irish heritage, so the 'Irish' is referring to his genetics. A wisp is something light and thin. So, he could be suggesting that his leg hair is light and thin, or more likely, he is using 'irish' to ironically mean it isn't really wispy, something like 'British brashness'.

He is saying that Irish men have a certain amount of thin hair (usually red) on their bodies. In this case, it sounds like he is referring to not having much hair, but just a small amount of thin hairs. Always be careful with Family Guy, many bad words and phrases on there! (not this one though)