"It's not a big deal" means "it's not particularly important". In context, that may sometimes be very similar to "it's not a problem", but often could be very different.
For instance:-
"My car caught fire today, but it wasn't a big problem."
"My car caught fire today, but it wasn't a big deal."
(Both are similar.)
"My friend got a promotion today, but she doesn't think it's a big deal."
(Only "deal" would be appropriate here, because a promotion is a positive thing.)
"Can you afford the cost of your son's medical treatment?"
"It's not a big problem."
(Saying "it's not a big deal" would be inappropriate here because obviously the medical treatment is important.)