You can say "We've known each other since we were just babies" or even something like "I've known him since I was a baby"
Also, If it's alright, can I correct your question as well?
I have some friends that lived in the same village as me. Someone asked me how long we've known each other. I wanted to tell him that we have known each other since we were babies.
^^ You're doing a great job :)