Chris and Bramwell have given you the correct technical terms : paternal grandparents and maternal grandparents. However, you need to realise that these are quite formal terms: they're more appropriate in a court of law or an academic essay than in general conversation. In everyday conversation, or in an informal piece of writing, they might sound odd. If a young person is just chatting to a friend, they'd probably say 'We lived with our grandparents - I lived with my dad's parents and my brother lived with my mom's [parents]".

When I was a child my brother and I did not live together. I lived with my paternal grandparents, my brother lived with my maternal grandparents.

When I was a child, my brother and I did not live together. I lived with my paternal grandparents My younger brother lived with my maternal grandparents