"not to late" is wrong in every situation. There is no verb "to late".

Im not native English speaker but in my opinion "not to be late" is correct, because "late" is an adjective so it should be with "be", after preposition "to" is normally a verb so "to be" is a correct one however in spoken English you can use both I think

I agree with that " not to be late" is right.

I am not native English speaker but i think "to late" is correct.