I would understand "hardly late" to mean only a few minutes late on one occasion. I would understand "hardly ever late" to mean almost never late, generally speaking.

hardly late for work- means you are not late for work almost every day hardly ever late for work- is almost the same but means you are late almost no days. So, they are almost the same but i think there is a tiny tiny difference. "hardly ever" is just used to put more emphasis. I mean, you can say one or the other. It means the same really. Adding the "ever" just puts more emphasis to make seem more often than normal.

"Hardly" can mean the same as "barely" or "not very" * "I hardly have any money" = "I barely have any money." = "I have very little money." * "I'm hardly late for work." = "I'm barely late." = "I'm just a little late, only three minutes." "Hardly ever" means the same thing as "almost never." * "I hardly ever have any money" = "I almost never have any money" = "I'm usually broke." * "I'm hardly ever late for work." = "I'm almost never late." = "I'm almost always on time."

Hardly is like "几乎不“ whereas hardly ever is like "几乎从不“. The first refers to one instance, the second to a general pattern.