I would have remembered if you told me that = I would have remembered if you told me, at any point in time
I would have remembered if you told me that = I would have remembered if you had told me, at some certain, specific time in the past
They are almost exactly the same meaning, but adding the "had" makes it refer to a more specific point in the past.
When we use this tense, we often add some other information, to help identify this particular time:
"If you had told me before I went to work this morning."
You're probably being confused with one of the most strange verb tenses in English - the past perfect. See here for some clear explanations of the difference between "You had told me" and "You told me":
http://web2.uvcs.uvic.ca/elc/studyzone/410/grammar/pastpf.htm
Below I will show you the important explanation that I copied from this page:
Using the Past Perfect
The past perfect is used to show you which of two events happened first. Imagine that two things happened in the past:
"I went to see the movie. We discussed the movie in class."
Here, we don't know which order the events happened in. That may be important -- perhaps I went to see the movie after the discussion, or maybe I saw the movie before the discussion. There are many ways to make this sequence clear, and the past perfect is one of them. This is how we do it:
"I went to see the movie. We had discussed it in class."