I'll take Keith's sample sentence as it sounds more natural. You're right that you'll hear native speakers say both 'He was upset by me being there' and 'He was upset by my being there'.
There isn't a difference in meaning, as they are both variations on the same phrase. The former is more common, but the latter is more grammatically correct. It's correct to say 'my being there' because 'being' is a gerund ( a verb functioning as a noun), so it needs a possessive adjective before it.