It has all to do with the origin of the words. You have to remember that when "avós" means grandparents it is a masculine noun though:
Os meus dois avôs. = both my grandfathers, from my mother's and father's side.
As minhas duas avós. = both my grandmothers, from my mother's and father's side.
Os meus quatro avós. = males and females grandparents alike.
As for the reason, it is due its etymology. In Old Portuguese the words were avoo (grandfather) and avoa (grandmother). By this time, a linguistic phenomenon was taking place: the "ó" sounds were raised to "ô" sounds when next to another "o" in the singular. Note that it is NOT uncommon in Portuguese for a masculine word to have an "ô" sound in the singular but and "ó" sound in the plural, eg: fôgo, fógos, pôrto, pórtos, ôsso, óssos. (accents added for clarity). That's what happened to avoo. It was probably pronounced avôo in the singular, but avóo in the plural, while grandmother was avoa (sg) and avoas (pl).
Finally, Portuguese contracted a lot of vowels in hiatuses, eg: marea became maré, coor became cor, maa became má. Avoo was no different and contracted into avô in the singular for its close ô sound but avós in the plural for its open ó sound, just like in fôgo, fógos. Coincidentally, avoa was contracted into avó and avoas became avós as well. Later "avôs" must have emerged as an analogy.
So there you go:
Old Portuguese > Modern Portuguese
O avôo, os avóos > o avô, os avós
A avóa, as avóas > a avó, as avós
Sources: theory based on the etymology of the words and my knowledge of linguistic phenomena in Old Portuguese. I study Linguistics in college by the way.