This means you said something that made the speaker feel bad. He might have cared very much about you, and you said "I don't want to see you any more." The speaker could hear that from other people, and not be bothered, but because YOU said it, he replied, "You're the last person I want to hear that from", meaning he wished you never said that to him.