Correction in question : Which is the correct sentence ? ( I didn't understand why you have written "By right" there)
"Yesterday you told me (that) you never wrote a letter (?) " is correct
Do you want to say,
"you had not written any letter"
"you had never written a letter"