In that context, it probably would be rude because the two speakers have been arguing. It would be said angrily and would be intended rudely. However, it isn't intrinsically rude. It could equally be said in a friendly manner. By the way, be careful of 'cabinet'. This is a false friend. In English, a cabinet is a small cupboard. A doctor works from an office or a surgery.

I would say it is rude, because that expression "off you go" is typically used from an adult to a child, or children.