I never thought for once i'd have the chance to help you back for all the help you've given me, Yan!
I really wanted to know the answer to this question... and as luck has it, the latest one of the Pimsleur's lessons I heard addresses exactly this issue. I'll transcribe it all:
"'go' and 'o' as polite prefixes. When you refer to your own family you say 'kazoku', while you call someone else's family 'gokazoku'. When you talk about your own house you say 'watashi no uchi', 'my house', however, you say 'anata no ouchi', 'your house', when you talk about someone else's house.
The rule concerning when to use 'go' and 'o' is quite hard, even for native speakers to explain clearly. You just have to know it, since there is no clear-cut straight rule. It might be somewhat similar to 'a' and 'the' in English. Principally, 'go' precedes a word that originally comes from the Chinese language, and 'o' is used for those that are originally Japanese. Whether the word comes from Chinese or Japanese is a very difficult question, even for the Japanese. as a non-native speaker, you do not need to worry too much about making a mistake between 'go' and 'o', although you should show your politeness by making the attempt to add the correct prefix."