Hi - both are okay. Don't worry about taking the "that" out, in my opinion. I know LOADS of natural English speakers that would include "that" every time that they used this phrase (including me, in fact).
"Talked" suggests to me that you had a longer conversation about the thing.
"Told" suggests to me it was just a quick exchange of words.
Tom