You can either say "which was made by my mother" or "which my mother made". (In the first case the pronoun is a subject, and in the second, an object.) It does indeed refer to the most recent noun, so we are talking about the dinner, not the spaghetti specifically. Moreover, as Javie notes, you probably want a comma. If you do not have it, then it becomes a restrictive modifier, and you are identifying the dinner which your mother made in distinction to dinners made by other people. (That doesn't make a lot of sense in your case, but think about "I enjoyed the dinner which was delicious", but perhaps not the one which was disgusting.)

Yep, you can use 'which' there, you just need to add 'was' after it. I ate cream spaghetti with my family at dinner, which WAS made by my mother. Think of 'which' coming after the noun 'spaghetti': I ate cream spaghetti (with my family at dinner) which was made by my mother. I'd say the sentence is correct but am not sure what you are referring to by 'cream spaghetti'. Is that a special type of spaghetti?

You can use 'which' this way! I would suggest you use a comma before 'which' if you are adding more information. For example: I have a new hat, which is a lot better than my other hats.