Yes, you could say: "Me olvido de las palabras/ Olvido las palabras", but you would be transforming a passive sentence into an active one. Pay attention to the fact that in the first sentence the subject is "las palabras", while in the second one, the subject is "YO".
So, the first sentence is in passive voice, the passive subject is "palabras" and "se" is in this case a passive particle, the sentence being equivalent to: "Las palabras son olvidadas por mi".
So: Las palabras son olvidadas --- las palabras se olvidan --- se me olvidan las palabras ("me" gives information about the active subject) ---"A MI se ME olvidan las palabras" (this duplication is common in spanish sentences, to put the stress on the active subject).