The teacher would ask them to hand in their TESTS.
You need a plural form here, because what you really mean by 'tests' in this sentence is 'test papers'. 'Tests' here refers to the physical pieces of paper
involved, so these should obviously be in the plural.
This is different from saying, for example, 'All the students are taking a test tomorrow.' Here we use a singular form, because they are all taking the same test.