Yes, you're right. 싫은데? in that context means exactly that. You can think of it as (그렇게 하기) 싫은데? (I don't want to do that). The "?" at the end also plays an interesting role. Without it (or if written as 싫은데 ...) it might sound like "Um, I don't feel like doing that, but ...", just wanting to avoid it. Adding the question mark makes it defiant, like "I don't want to do that. So? What are you going to do about it?".