Listen is to hear. Listen in or listen in on is to hear someone without them knowing or without consent.

"Listen in" or "listen in on" are colloquial expressions used when an existing conversation is already taking place or going to take place, and you want to participate and get some information out of it. I don't necessarily agree with Badria that these expressions imply surreptitious listening; the listening could be out in the open. You could say to a co-worker: "Would you mind if I listened in on your conversation with Julia? I want to know what she thinks of this proposal."