I'm not an expert, but here's how I would define them:
"Experiment", as a noun, simply describes a test under controlled conditions. You could say, "the laboratory performed experiments on guinea pigs."
"Experimentation" is the act of experimenting. You could say, "after a long day of experimentation, I finally managed to bake the perfect cake."