Hi Daniel!
Normally, we can say "a big deal" but we can't say "(a) big of a deal" since it wouldn't be correct.
So, for example, "it was a big deal" is correct, but "it was a big of a deal" would be wrong.
There is, however, one situation I can think of where you may hear the second one and I've heard it quite a lot. Let me give you an example:
A: I can't stop panicking about yesterday's test! What if I did really badly?
B: Oh, don't worry about it. It's not that big of a deal.
Here, instead of "it's not that big of a deal", B could have said "it's not a big deal", which has more or less the same meaning. B is just saying "it's not a big problem, it's not worth worrying about". I wonder if this is what you were thinking of?
I hope this was helpful! :)