It means that they would not be expecting it and would not know how to react as they were surprised and will not have a prepared or standard response.

When a soldier is ready for an attack, he is 'ON guard'. If you attack him when he is not expecting you, you 'catch him OFF guard'. In a situation where someone is expecting a fight, or an argument perhaps, a tactic is to approach them in a surprising way - with a smile perhaps, or at a surprising time. The expression is generally used with the verb 'to catch'. eg. I want to catch him off guard, so I will call him at the weekend. eg. I didn't expect him to be friendly - he caught me off guard

On (one's) guard means to be alert, cautious, aware of what is going on around you. Off (one's) guard is the opposite - inattentive, unprepared, unsuspecting. Catching someone off their guard (or off guard) means you have you have found them/put them/caught them in a situation they were not prepared for - which usually results in the feeling of surprise by the person who is "caught off guard". I hope that makes it a bit more clear?