I agree with Phil. If the more formal, but less common, 'his' is used then it would appear to be a gerund as it's a noun. But it could also be taken as a reduced participle clause.
I seem to recall his mentioning a brother once. - As a gerund, this would focus on the process of the mentioning, so the way he did it or something like this.
I seem to recall him mentioning a brother once. - I seem to recall that he mentioned a brother once. - This would focus on the fact or the result of what he mentioned - I think he has a brother.
I think the second situation is more likely, but we can't tell from the sentence alone. We would rely on things like context, tone of voice and body to language to understand what was actually meant, so sentences like these are practically impossible to be certain about without this information.
A more subtle example could be:
I remember him/his ordering a meal in Japanese. - As a gerund, this would describe the process and the person could be implying that it was funny or embarrassing or something like this.
I remember him/his ordering a meal in Japanese. - I remember that he ordered a meal in Japanese. - As a participle, this would imply we're interested in the fact or result. Maybe they're saying that he must be able to speak Japanese.
With the above examples, the way it was said should tell us if it was a gerund or a participle.