you have to understand = (please understand)
a day does not go by when = (everyday)
that = (any thought, memory, past experience)
does not weigh heavily = every day I think about that thing that ...
on my conscience = (right and wrong) this speaker feels guilty about something.