In governments across the English-speaking world, these terms are often used interchangeably. In fact, sometimes a body changes its name from one to another. "service" emphasises the idea that it is serving the public's needs e.g. health service, fire service, "department" indicates a unit of a bigger administrative whole e.g. the Department for Rural Affairs "agency" often indicates that it has been entrusted with a task by another body and acts on their behalf e.g. the Environmental Protection Agency "office" indicates a body with an administrative role e.g. the Cabinet Office, the Patent Office. In the USA, they have police departments, whereas in England we have "constabularies" and in Northern Ireland, they have a "service". In England, many of the departments of state are called "office" e.g. Home Office, Foreign Office; some are called "ministry" e.g. Ministry of Defence; and others are called "departments" e.g. the Department of Work and Pensions.

Thank you for the complement :-)

You left out bureau, board, administration, and commission. I believe that department heads are in the cabinet. But honestly, I'm not sure and it has never mattered that much.