It's just a way of starting a conversation. It basically means "what are you doing?" or "have you got any news/gossip for me?".

E.g. If I was walking up the town and bumped into my friend John and he said "hey, what's up?".

I could reply "Alright John, I'm just going to buy a new pair of shoes".

Or I could reply "Nothing much. What's up with you?"

Or I could reply "Well, did you hear what happened last night?"

"Alright" and "well" are used as greetings here in Ireland.

Hope that helps.

"What's up" is essentially a short way to say "what have you been up to recently" or depending on the context it can have similar meanings like "what's new" or "what's happening". The most basic response is either going to say something like "not much" or as Mano pointed out reply in kind with "hey, what's up" as well, but you can also respond by talking about something that recently happened or is going on right now in your life. It's used kind of like "how are you", but you don't respond the same way. It actually sounds kind of funny when someone responds "I'm fine, thank you". 

A similar phrase that I probably use about as much would be "how's it going?" It's basically the same as "how are you", but is less focused on the person's health and more on what's new or interesting in that person's life. Like always in English though, people aren't actually expecting a long response (they won't be offended if you do give them a long one, but they may be surprised). A simple "pretty good" usually suffices, especially if you ask another question in return. 

Basically, it is a rude and indifferent  expression,  which shifts the burden of responsibility for communication  entirely upon the person being greeted. That is why  this idiomatic expression is  associated with casuistry and recent social conventions. It is not  really "friendly" and that is why you are entirely confused as to what the other party "wants" to hear.

  "Hello. How are you?"   is better

    You can respond anyway you want. You can be kind in such circumstances and say  "Hey!" or something equally vacuous. You can smile and/or respond with an equivalent facial expression as the person greeting you.

  When people greet me this way,  I generally trust them about as far as I can thrown them,  as we say in the USA.

Sometimes I answer roof or sky depending upon where I am :)

I almost forgot, here's an old commercial that makes fun of the saying while showing you how to use it at the same time.

https://www.youtube.com/watch?v=tauYnVE6ykU