To drive simply means the action of driving a car.

To drive off means to leave. It means that you are leaving (not arriving) and your mode of transport for leaving is by car.