I would leave off the "then I will act accordingly". It sounds normal and natural with just the "If you don't want to come along then just tell me." If you really want to sound "natural" and if you are annoyed with this person, then say: If you don't want to come along then just tell me! My god! What's the big deal?. Then I will go without you! :o) Yes, I know it's on the rude side, but it's something I can hear myself saying.