There's a saying, "You can't make a silk purse out of a sow's ear." It means if you start with poor quality materials, you can't make something nice. The person speaking and Lula were both adopted and Tony apparently considered them "poor quality material", perhaps because of race, in Lula's case. In Tony's opinion, even though they were raised by presumably good quality people, they couldn't hope to overcome their inferior beginnings.