You can say both. "I failed in the examination (exam)" is more formal. "I failed the exam" is usually what you hear in daily conversations and is more commonplace.

fail the exam or fail on the exam

you can say either one. If you say "I failed in the exam." it sounds much more formal. It is not what you would expect an ordinary person to say, but it might be said by someone who communicates at a higher register of English-- like someone with an Ivy League education, or possibly someone from Britain. It is not the typical American way to say it, but to me it sounds pleasant.