I think it means that he is not a very imposing man - he sort of blends into the scenery, so no-one remembered that he was coming , and, probably, even when he was there, no-one really noticed him. I think if it meant that he came without being invited, it would have said "no-one remembered having invited him to dinner'.