I am going to my brother's - meaning I am going to my brother's house. We often drop the "house" because it's commonly assumed that you are going there.