"Don't be silly" is not supposed to be really offensive, as long as you only use it in an informal setting, around friends and family. Tone of voice matters a lot with this one, too. "Don't be silly" is usually followed by a response that corrects the other person's misguided comment or question. Example: "Are we walking to the museum?" "Don't be silly -it's much too far, we will take a taxi." or "Can I help you with the dishes? "Don't be silly -you're our guest, just relax and enjoy the party!" The only way I could see this being offensive is if you said it with a harsh/angry tone, or if you said it to someone whom you should respect, like your boss. Hope that helps!