I'm going to take a stab and say that they're talking about holding the poker game at the other guy's house.
"Class it up" means to "up the classiness" essentially, so I'm assuming the guy has a nice home.
"Holding it here" refers to holding the poker games at his place, to have it there.
"When you're up" refers to when it is his responsibility to host - I'm g uessing they have a rotation for who hosts the games, so when "he's up", it's his turn.
A "buy in" in poker is the entry fee for the game.
Fifty is the "buy in", $50.
"We're small time" is him saying that the $50 is a small "buy in". "Small time", as opposed to "big time", refers to a lower magnitude affair.