In the phrase "I'm going to breakfast", it's a noun. Same as "I'm going to the city centre" or "I'm going to my friend's place." You are moving from this place to another place (for breakfast).
If you say "I'm going to eat my breakfast", it's a little odd. What else would you do with breakfast? :)
It is very rare to use "breakfast" as a verb, so for the moment don't worry about it.