I would go with : You can get by with simple English if you speak it as a second language. Professional teachers and ordinary native speakers will understand you then : )

It sounds perfectly natural to me to say "ESL speaker". I do not know where the other commenters come from, but the state I am from in the US has a large foreign-born population, so most schools have ESL classes separate from the regular English classes, so the majority of people are familiar with the term here. Even people who were not in the ESL classes themselves knew the name of the class.

"ESL speaker" isn't incorrect at all, but as others have mentioned some people may not be familiar with the term "ESL". If you're speaking to someone who knows that term I think it's a perfectly natural sentence, but otherwise it may be more natural to say "as a non-native speaker", or "You can get by with simple English if it's not your first language."

I agree with Harry... You could say: "...non-native speaker..."

From what I am aware I don't think many people know what the term ESL speaker means. That said I think your context would have provided the information to work it out and the way you said it was perfect.