It is to their home, and romantically so. If it were your home, it would be "I know who I want to take home".
In this case, you can safely assume that the performer of the action is also the person who has the home in question. In the original lyric, the "taking home" is done by someone else.