I tend to agree with the other answers: "Aren't you ashamed of lying?" is probably what one means most of the time.
"Ashamed of lying" means that one is ashamed because of lying itself.
In contrast, "ashamed to lie" fundamentally means that somehow one is already ashamed, probably due to doing something else, and the shame makes lying problematic (maybe because it would compound the shame one already feels, or because the consequences of this lying can cause further shame).
Another difference worth mentioning here, especially since it provides a good segue to everyday usage, is that "ashamed of lying" does not commit the subject of the sentence into the action of "lying". In other words, "ashamed of lying" may mean "ashamed of my lying" or "ashamed of your lying", or even "his lying". One is ashamed of lying in general, of the concept of lying, if you want. However, "I am ashamed to lie" makes sure that one speaks about the subject, "I" in this case: it states clearly that one speaks of his own lying, instead of the general concept of lying.
Anyway, in real life, away from the grammar textbooks, one often uses the latter to mean the first. This practice goes as far as to generally regard the use of the present participle as a noun to be a bit affected if not a tad obsolete. Contemporary colloquial seems to favor the infinitive over present participle in this type of construction.
There is another way of having the cake and eating it too. If one must use the infinitive, you are better off with "Doesn't it shame you to lie?".