The first two sentences are not grammatically correct. This is because after the verb "hear," the bare infinitive is used (this means the infinitive without the "to"). So, in these sentences, just use "say" instead of "to say." This is true for a small number of verbs, such as hear, see, watch, help, let, make. So the correct forms are
Never once did I hear you say that to me.
I never once heard you day that to me.