Hi there. You use the first example to thank someone for asking you to accompany him/her on an outing. The second example is used to thank someone for inviting you to his house. Hope this helps you!

You say "Thanks for inviting me out." to someone who has taken you out for a meal, to the pub, concert etc and "Thanks for inviting me over." when you have been invited to someone's place for a meal, party etc