“Sweep me off” here is related to the more common expression “sweep me off my feet” — an expression used to mean that you very quickly became infatuated with (fell in love with) another person. Example: “The handsome stranger quickly swept Emily off her feet, and they eloped together.” Here, the idea seems to be that, in a burst of romance, the narrator would quickly fall in love with the tracksuit man and he would take her away from wherever she was to a (slightly) better life.

John's answer is completely correct, in that this is a reference to the romantic term 'sweep someone off their feet'. For context though, the references to 'most velour tracksuit' and 'an apartment shared by only four families' is meant to tell the reader that this woman/girl is very poor. In England there is a stereotype that people who wear Tracksuits (outside the Gym/Sports) are not very wealthy and so 'the most velour' tracksuit would make this man/boy a richer poor person. The fact she is romanticising an apartment with ONLY(!) four families also implies she's currently living with more than four, so she'd have to be very poor herself. In short a lot of the text is making sure you know that she is as poor as she can be, but still romantic (sort of) and hopeful that something will change her circumstances.

What is this from?