"It doesn't make sense" means something is logically incorrect. e.g. "Two plus two doesn't make five. That makes no sense!" "I don't understand" means that you haven't fully comprehended a concept. e.g. "This topic is very advanced. I don't understand it yet."

"I don't understand" has a broad meaning; it can refer to people speaking foreign languages, or to a complex situation (such as a scientific concept, news item...) "It doesn't make sense" means that you don't understand the logic behind "it". It can even mean that you understand, but you disagree with it. So, it is a much more specific comment than "I don't understand"

Here's another way to think of it: If I say I don't understand something, I probably want you to explain it to me. If I say something doesn't make sense, I would probably proceed to explain to you why it doesn't make sense.