As Ellen says, it refers to the flower. It may help to think about it this way: "Only one blossom OF this flower grows." "This is the flower OF WHICH only one blossom grows."
This is sort of a different usage of "of which," I guess. I don't know the correct grammatical description for it, but it doesn't work with a verb the same way as sentences like these:
"It's a topic IN WHICH many people are interested." (Many people are "interested IN" the topic.)
"This is the novel ABOUT WHICH I wrote my essay." (I "wrote ABOUT" this novel in my essay.)
Instead, it refers back to the original noun and describes instances of that noun. For example:
"We have two cats at home, both OF WHICH are Siamese." (Both OF our cats are Siamese cats.)
"There are many types of tea, one OF WHICH is Assam." (Assam is one OF the types of tea.)