Both are okay. 「ご」is a word to make the words politer. and ご祖父母／祖父母are fine to use for this situation, too. There are no special ways to call other person's grand parents.