In the US, it’s referred to as Police Station. I’ve never heard anyone say Police Office. I suppose if there was some separate administrative building, maybe someone would refer to the Police Office?

I'm a US native speaker. I've never heard the phrase "police office." You might be thinking of "police officer." An "officer" is a person with an official duty. In the United States, "policeman" or "policewoman" is a slightly informal word; "police officer" is the correct, respectful term; "cop" is informal.