It means that you, in a given moment in the past, were asking yourself whether the person you are talking to, had the intention to come home for dinner, or was in her/his way to come home for dinner (the home where you, who are speaking, are, or where you were at the moment of asking it to yourself. But "was wondering" in this case could also be a "past" not referred to "the past", but to the "present": It could be a "hesitant", a "tentative", way of asking questions. In other words, a "polite" way of asking questions, referred to the present. Like saying: "I'm asking myself if you are coming home for dinner". Or : "do you mind", or "do I disturb you if I ask you if you are coming home for dinner?" Of course, since "I was wondering" is in the past (in spite of referring to the present, as i've said), as a result, for a fact of "grammar agreement", "were coming" is in the past form as well.

The asker wants to know if the person they are asking is coming home to eat dinner that evening.