Case has a few meanings. One is as a shortening of briefcase or suitcase. Here, it means something like 'situation' or 'problem'. In fact, when I was answering this question, my first way of writing it was:
"In this case, it means something like..."
A very common usage for 'case' in this way is for detectives or policemen: If they are investigating something, it is their current 'case'. For example, some Sherlock Holmes' books are called "The case of..."
In this example, the hubble scientists had trouble figuring something out (because two things look very similar). So, the 'case' is that distinguishing the difference is hard. This was a very intriguing problem for them to solve.