While the definitions of these words is almost the same, there are subtle differences in use. A souvenir would normally be something you kept in memory of an event or a trip, rather than of a person. Technically it is not wrong and I think it would be understood but it would be an unusual way to use that word. A remembrance is also technically correct although that word is most often used in the context of remembering someone who has passed away. Again, I think it would be understood perfectly but depending on the context, it might sound a little odd. I think Keep it as a memory of me would sound most natural and correct. hope that helps.

The short answer is no. I don't what "it" is that you are referring to but... "Keep it to remember me by." is probably the best option. A souvenir is something you buy when you are away from home to take back with you to either remind you of your trip or as a gift to someone back at home. The term "in remembrance of" is usually reserved for talking about someone that has passed on. And even "keep it as a memory of me" still brings to mind the image of someone that has passed.