"Charla pequeña" doesn't make any sense.
Unfortunately, I can't come up with any words to express this idea concisely. I would just say "charla" or "conversación trivial" / "conversación sobre cosas sin importancia" if you want to emphasise that it was a trivial conversation.
I could also say "hablar del tiempo" to refer metaphorically to all these conversations, even though they're not about weather.