It could mean "don't hang out", but it is more general. It means to have anything to do with. So it also means that the speaker would not work with or have any business dealings with such people.