As in spoken English and in a private informal communication you can say what you feel like :)
It is not common to say 'sweet dream' though , "Sweet dreams" is the commonly used expression .
Scientifically an average person has between 3 to 5 dreams a night ,if not more !
That is probably why it is not accurate to say ' have a sweet dream ' !
Otherwise in a flirtatious way you might even say " Have a sweet dream of me " :)