As Paul says, it's a (not especially common) GB English expression meaning 'Go away!' It's rude, but it's not as vulgar as actually swearing at the person. The advantage of saying 'Bog off!' to someone is that it SOUNDS vulgar, and feels vulgar, but actually isn't. It's a perfectly innocent word. You could say 'Bog off!' in the script of a children's movie and still get a family certificate from the Board of Censors for it. That counts for a lot.

It's a slang way of saying 'go away' in the UK. Basically it's a more polite way of telling someone to fuck off/piss off. The example on the Cambridge Online Dictionary website is "Bog off and leave me alone." It's slang and very informal. People would say it in irritation or anger if someone won't leave them alone. To be honest, I don't hear it very often where I live. I normally hear people say 'go away' or one of the cruder alternatives to convey the same meaning.