I would almost never use it, except as a joke.
It can refer to trash, something that you don't need. However, it can refer to two or more people who are unimportant and of low social standing. It is an insult when applied to people.
You might hear it from other people.
"I thought you said important people were going to be at this meeting, but all I see here is riffraff"
"I usually don't associate with them on a social basis. Let's face it - they're basically just riffraff."