They are both grammatically correct. I think the difference is something like this (but I'm not sure if it's a real rule or not): Aren't you ashamed of lying? = Don't you feel shame because you lied. (That's the more common one.) The politicians aren't ashamed of lying about how the money was spent. = They lied about it already and we found out, but they are not ashamed. Aren't you ashamed to lie? = Aren't you reluctant or unwilling to lie because it will bring you shame / you will be ashamed? The politicians aren't ashamed to lie about how the money will be spent. (They would not hesitate to lie about it. They aren't worried about being shamed.) Ashamed to = I'm hesitant because I will feel shame. I'm ashamed to admit that I like Justin Bieber's music. (OK...it's just an example.) Ashamed of = I feel the shame now for what I did. I'm ashamed of telling those dirty jokes to the nun. (Really...just another example) Maybe somebody knows a real rule.

both are correct

I tend to agree with the other answers: "Aren't you ashamed of lying?" is probably what one means most of the time. "Ashamed of lying" means that one is ashamed because of lying itself. In contrast, "ashamed to lie" fundamentally means that somehow one is already ashamed, probably due to doing something else, and the shame makes lying problematic (maybe because it would compound the shame one already feels, or because the consequences of this lying can cause further shame). Another difference worth mentioning here, especially since it provides a good segue to everyday usage, is that "ashamed of lying" does not commit the subject of the sentence into the action of "lying". In other words, "ashamed of lying" may mean "ashamed of my lying" or "ashamed of your lying", or even "his lying". One is ashamed of lying in general, of the concept of lying, if you want. However, "I am ashamed to lie" makes sure that one speaks about the subject, "I" in this case: it states clearly that one speaks of his own lying, instead of the general concept of lying. Anyway, in real life, away from the grammar textbooks, one often uses the latter to mean the first. This practice goes as far as to generally regard the use of the present participle as a noun to be a bit affected if not a tad obsolete. Contemporary colloquial seems to favor the infinitive over present participle in this type of construction. There is another way of having the cake and eating it too. If one must use the infinitive, you are better off with "Doesn't it shame you to lie?".

the second sounds better.

I think the correct one is your second option ^^