Matt and Tim are right. Just for further information, "to hell with [X]" can also be used for cursing or dismissing other things or ideas, not just people. For example:
A: "I don't think we should get married. It could ruin your career."
B: "To hell with my career. It's not important."
You can also just say "to hell with it" as a general way of giving up on a situation, a plan, etc.:
"I can't get this key to work. To hell with it, I'd better just break the lock."
"I waited for you until midnight, but when you still didn't show up, I finally decided, to hell with it, I'm going to bed."