מבחן - is the literal translation of test.
בחינה - Is more like an exam. It's plural form could also be used to talk about tests and exams in general, but then you'd say בחינות (exams in plural).
בוחן - quiz
The same way that they're interchanged a lot in English, they can be switched around in Hebrew.