I think I would normally expect to see a grain of barley but both are ok I suppose. As this is such a rare thing to see written, I doubt anyone has a strong opinion on it!

A grain of barley stresses the amount - not very much barley, just this one grain - while 'a barley grain' is a descriptor whose focus is on the grain - it is a grain, and specifically it is of the barley plant. But it's not a BIG difference and either way is fine for most things in a technical sense. It depends on how other things are being described in the neighboring sentences, too, which style means what. Neither is more bookish though, that's not a difference, and neither is a scientific term, it just depends on what it being stressed.

They are almost the same. “A grain of barley” is a single piece of barley. “A barley grain” can mean 2 things, either a single grain of barley OR the concept of barley as a grain, An example for this last use would be “I made beer out of a barley grain” or “I made bread using a barley grain.”