Actually, in come cases, "late evening" can loosely be considered a point in time, relatively speaking. Consider the following sentence: "Some flowers only flower at late evening and night and others, particularly several white varieties seem to actually glow under a moonlight." Neal is correct in that you would hear or see "at" used much less often in this case.

Only the second of your 2 alternatives is correct. You could also use 'in the late evening' or 'during the late evening'. As fdmaxey says, 'at' refers to a specifc moment in time, which 'late evening' is not!

"at" states that you arrived at exactly a given time. "by" states that you got there at or before the given time. Since you talking about a period of time (late evening) the meaning is fairly clear both ways, though "by" might imply a stretch of time somewhat earlier that "at". If you were talking about a time, it would be different. "I reached home at 8 o'clock" means that I got there at exactly 8. "...by 8 o'clock" would imply that I got there either at 8 or some time before. The difference is more distinct in this case.