Here is a paragraph that uses all of them:
At that time, I was a young man. That's when I went I went to University. That time was a great time. At the time, I didn't have a care in the world. Back then, I felt like I could do anything. Back in the day, we all acted like kings and queens. In the past, I was able to speak more Korean than I can now.
Ultimately they are very similar expressions, but some of them have nuance differences. Particularly "Back in the day" has more of a nostalgic feel. "In the past" can refer to someone's past OR refer to the distant past (e.g. In the past, Sejong the Great invented Hangeul). "At that time" references something you've already discussed. (I went to University a while ago. At that time, grunge music was cool.)
Hope that helps!