You would not say "If my memory serves right." (missing the word "me")
You would say,
"If memory serves me right."
"If memory serves." <---this can be used informally.
"If memory serves me correctly,..."
Informally you could also say,
"If I'm remembering this right, ..."
"If I'm remembering correctly,..."
"Unless I'm mistaken,..."
"My memory is (kind of) foggy now, but what I remember is..."
"If I've got this right, ..."
"As far as I can remember, ..."