It depends a little on context. I would probably use the first one, because it specifies that you're inviting the person to your home. (Apartment, by the way, rather than partment.) "You're cordially invited" is a very formal way of saying this, and -- at least in the U.S. -- is more common on written invitations than spoken ones. If you're not giving a written invitation, you might say something like, "I'd like to invite you to my apartment for lunch."
I hope that helps!