His jaw would set means that his face adopted a determined expression. I guess this is still from P.G. Wodehouse. He uses a lot of slang specific to the British upper-classes in that period that is not widely used or even understood now. However, it basically means it was clear he was thinking hard. The onion is his head. The hamster wheels are his brain.

It's not an idiom. Either it's the writer's colorful invention or it could POSSIBLY BE A TRANSCRIPTION ERROR. Pet hamsters (rodents) have wire-frame wheels in their cages. They like to run around in them and make the wheel turn. "Like a hamster in a wheel" can be a colorful phrase for "obsessive activity." It is a fairly common image to say "I could see the gears turning in his head" or "I could see the wheels turning in his head" meaning "I could see that he was thinking." "Hamster wheels" is just a colorful and original variation. "Onion" is a mystery to me. It probably means his head, but I've never heard it before. Because an onion is in layers, "onion" is often a metaphor for something deep and complex, however much you expose there is always more underneath.

His jaw would set -- describes a expression with his mouth that shows he is determined. I've never heard of "turning in the onion". It probably means in his mind. The whole expression means he is thinking. Like the "gears are grinding" -- his brain, the machine, is working.

It is definitely a metaphor; can I ask where you are reading this from? It basically means that you could tell he was thinking.