Matter refers to subject or question but problem is something that needs to be solved.

Depending on context. When you ask someone "What is the matter?" you are asking about their emotions which replaces longer more direct questions such as "What has caused you to be upset?" or "What has caused you to cry?". When asking "What is the problem?" you're asking about an issue they may be struggling with that has a solution such as struggling with understanding or performing an action. Matter is also what all things are made of