"I will come back in an hour" is a promise to return in approximately an hour, perhaps any time between 50 and 70 minutes [if you're German, it might mean any time between 59 and 61 minutes ;) ] "I will come back within an hour" is a promise to return in less than an hour.

An hour is one hour ( 60 minutes), between an hour is the time start counting from 0 till to the end of 60 minutes, any time within this period. In an hour, meaning about one hour, you don't know the exact time, but close to the end at 60 minutes.