From a dictionary:
breakfast:
noun
1. the first meal of the day; morning meal: A hearty breakfast was served at 7 a.m.
2. the food eaten at the first meal of the day: a breakfast of bacon and eggs.
verb (used without object)
3. to eat breakfast: He breakfasted on bacon and eggs.
verb (used with object)
4. to supply (someone) with breakfast: We breakfasted the author in the finest restaurant.
James' answer seems to be the most accurate.
In Canada and the US we use breakfast as a noun - maybe 99% of the time (definition #1 and 2)
It is very unusual for someone to use breakfast as a verb, although it's not incorrect (definition #3 and 4).
"and sat down to breakfast" means "and sat down to eat the breakfast".
"to breakfast" would usually be interpretted as a preposition + noun, however, it could also be considered to be the infinitive form of the verb breakfast (i.e., to breakfast).