I'll take Keith's sample sentence as it sounds more natural. You're right that you'll hear native speakers say both 'He was upset by me being there' and 'He was upset by my being there'. There isn't a difference in meaning, as they are both variations on the same phrase. The former is more common, but the latter is more grammatically correct. It's correct to say 'my being there' because 'being' is a gerund ( a verb functioning as a noun), so it needs a possessive adjective before it.

Both sound fine to this American ear. I use either one, and to me they both have the same meaning.

They're a little bit different situations. 'My going there upset him' sounds ok to me, but a little too 'official - sounding'. In that case I would probably say 'He was upset by me being there' ('me going there' isn't good in this case, maybe because 'being there' isn't your action/fault). :)

I think it depends on what you are trying to emphasize as the problem: the action (the first sentence) or the person (the second sentence).