This is an American English idiom, though rarely used, referring to an awkward or difficult situation. Using "fine kettle of fish" has the same meaning. Here is a sample usage:
"After walking into the classroom and realizing that there was a pop quiz, he realized that his decision not to study last night had put him in a pretty [fine] kettle of fish."