Actually you can use 'who' or 'whom' or nothing. Whom is only used when referring to the object of the main clause. Here the subject is 'He' the verb is 'was' the object 'the man' 'Whom' refers to the object 'The man' and can therefore be used. But the use of 'whom' is becoming less common. Many native speakers now tend to use 'who' instead of 'whom' because they feel 'whom' is too formal or outdated.