The correct sentence is this: "I asked my mother if my brother had come back home." You need past perfect (had come) because the brother's return possibly happened before the question. Also, you don't put "to" before "home" in this sentence.