The first one is your more casual way of saying it, but the correct way will definitely be the second way of saying it.

"What's the matter" is a common expression so native speakers might sometimes say "I don't know what's the matter" in casual conversation, and not realise their mistake. In other situations, we would feel the mistake more obviously and so I don't recommend saying it more generally. For example: "Who is that guy?" "I don't know who that guy is." Correct. If you said, "I don't know who is that guy", it would sound strange, even in casual conversation.

The first one is more natural