You use whether when the sentence has the sense of 'something may or may not be true or happen.'
You use if with conditional clauses.
So: 'I don't know whether I should invite her to my party,' but 'If I invite her to my party, then I fear that she would ruin it.'