Both, but they are slightly different. The only thing you need to remember is that 'off' is the opposite of 'on', while 'out' is the opposite of 'in'. So if it is a simple chair without arms, and the baby was on the chair, you could say that he fell off the chair. But if it's a big armchair, or a special type of baby chair which encloses the baby, you could say that the baby is in the chair and that he falls out (of) it.

You can say either "My baby fell off the chair" or "My baby fell out of the chair." (Add "of" into your 2nd sentence.)