It would be more natural to say "I got here at 5am" instead of "I came here at 5am".
To my ears it sounds unnatural to omit "here" in these two examples, you could say "came in" or "got in" though if you really didn't want to use "here".
It is also fine to change the main verb to arrived, in this case "here" can be omitted: "I arrived at 5am" and "I arrived here at 5am" are both OK.