You could say something like 'that sounds like it's right up my street' if someone has asked you if you want to visit a museum, and you would like to go. The expression shows that the topic is something that interests you a lot.
If someone just asks if you like museums then the phrase doesn't really fit that situation. We also don't normally use it in the negative like in your original question.
'Take it or leave it' means you have no strong feelings about the subject, as Bob said you don't really care about it.
Person A: What do you think about going to the cinema?
Person B: I can take it or leave it. It's nice to go sometimes, but I'm not really a big movie fan.