I'm not sure if that is technically 100% grammatically correct or not, but the meaning is very clear either way. However, I would say "I'm not a native English speaker of either English or French" instead. The way that you wrote it sounds less natural.