To close out means to end. If I move to another country, I'd close out my account at the local bank. If a store is going out of business, it might have a closeout sale. I can't think of a connection to a university, unless it means the class was full and no one else could register for it -- registration was closed out. But that's just a guess. Normally we'd say registration was closed.

I can't remember the context, I think it was about something related to college or university

In what context is it being used?