Great question! There really is no difference. But if I had to say something was different, I'd say that perhaps "to sit down," sounds a bit stronger, more exasperated. Perhaps the scene is like this. You enter a room looking for a place to sit. You turn to your friend and say, "There's no place to sit." You look some more and turn to your friend again, frustrated, "There's just no place to sit down!" But really, you can use either.