It's a playful word that is used in the context of casual conversation, and I've heard it many times.
Children, mothers with children, casual arts and crafts - It's not a word brawny men would use, unless they were allowing themselves to become playful like kids.
The spelling is also playful. It kind of takes itself from a variety of other similar words and combines them in whatever way the user desires - mash, smash, slush, moosh, mush, shmush, shmoosh, etc..
However, were I to have to choose exactly how I wanted to spell it, I agree with you that they might have a certain connotation for each spelling.
I think of smush is more of a physical smashing of something playful and moist - Playdough, cookie dough, pie dough, mixing of salves, that sort of thing.
I think that smoosh does seem like it has more of a dry and squiggly feel to it. Just was mentioned with the dog face rumpling.
As you can see, since there is no real definition, you can probably relax on using the exact one. We all might use a different variation as well, depending on how we feel. :)