Usually, the more proper way to ask the question is "What is the matter?" as opposed to "What is your matter?" (considered not as polite). Also, the main difference between "What's your trouble?" vs. "What's the matter?" (in my opinion) is that one is more severe than the other. Usually, if you ask "What's the matter?", it's about the person going through a minor issue/problem, while asking about one's trouble is more about a major issue.

What's troubling you? What's the matter?