It's just an adjective and a noun. That's an extremely common sequence in English. Either the person who told you that it's incorrect doesn't know much about English grammar or you misunderstood what they said. Since "brother" is a singular countable noun, it needs a determiner. See how Claire said "my big brother" and "my little brother?" Did you use "big brother" without a determiner? Could that be what the person was trying to correct?

To mean your older brother? Yes. e.g. I get on well with my big brother but my little brother is annoying.

Si