Da haben Sie recht, (Herr XX)! You can't mix formal and informal. Either you say Herr/Frau XX, or you leave it out. Anyway you use the formal way of expression, whereas in English the verb and 'you' do not change, so you have to use a 'Sir' to make it clear. But just 'Herr' without name is not possible.