hardly late for work- means you are not late for work almost every day
hardly ever late for work- is almost the same but means you are late almost no days.
So, they are almost the same but i think there is a tiny tiny difference. "hardly ever" is just used to put more emphasis.
I mean, you can say one or the other. It means the same really. Adding the "ever" just puts more emphasis to make seem
more often than normal.