Sofi, your use of "in that case" is correct.
You could say "in this case" if you're, say, describing various possible situations.
"It's out of context" could make sense, but I think I'd more frequently hear something like "It was taken out of context."
"Put me in context" is not something I've ever heard. If you're telling someone to tell you the full context of something, you might say "In what context?" or a question like that.
You might more often hear "To put it in context, ..." as a lead into an explanation of how the topic fits into a broader topic.