"Don't be silly" is not supposed to be really offensive, as long as you only use it in an informal setting, around friends and family. Tone of voice matters a lot with this one, too. "Don't be silly" is usually followed by a response that corrects the other person's misguided comment or question.
"Are we walking to the museum?"
"Don't be silly -it's much too far, we will take a taxi."
"Can I help you with the dishes?
"Don't be silly -you're our guest, just relax and enjoy the party!"
The only way I could see this being offensive is if you said it with a harsh/angry tone, or if you said it to someone whom you should respect, like your boss.
Hope that helps!