Either is fine but I think it's much more common to say "at the office"; you might only hear someone say "in the office" if you're talking with someone outside of the building/department & telling them you left something "in the office."
I think of "in the office" as being more exact in location; like, that's where you happen to be sitting/standing at that moment, while "at the office" just means you're inside the building or inside the department/cubicle that you work in.