I have a big bookshelf with four shelves.
The top two are fiction. The bottom two shelves are non-fiction. Books that I've already read are stashed behind books that are not yet read or completed. They are in order to "look good".
So for example all of my Colleen McCulloughs are grouped together because they are paperbacks and all around the same size, height-wise. Then I have about three Barry Eisler books that don't quite make it to the end of the shelf if they're stacked with the spines vertically, so I stack those ones sideways, so the spines are horizontal. Some books are hardcover and might be raised higher, so I leave those ones on the ends so that visually, they don't just stick out in the middle, grouped by author if necessary. Laying them flat and stacked or grouping them together upright entirely depends on how it pleases me when I look at the bookshelf.
Non-fiction is an entirely different section to organize. Sometimes I find myself lingering more in front of the bookshelf, so I chose to have these books on the bottom so I can sit down in front of them. For fiction, usually I take what I want and leave. Non-fiction, I'm usually picking at a few and looking at them right then and there.
The other thing about my non-fiction books, is that they are in sections of the subject. Now, I still group larger hardcovers on the ends (such as my atlases, or coffee table books), but my other books are grouped by subject and then arranged the same way I arrange the fiction. They need to look esthetically pleasing to me. Some are grouped together, and some flat and stacked, depending on the flow of what looks nice. Also, within each subject if I can keep the same authors together, I will.
I never gave any of this much thought before this discussion! But I seem to always do this the same way, every time I move and need to reorganize my bookshelf when I unpack. Great topic! :)