'In the back of ones mind' is an idiom meaning: if a thought that worries you is at the back of your mind, it is always in your mind although you do not spend time thinking about it.