I wouldn't use "of" or "in." I would say: He has experience teaching English. (You don't need to say "language" because it's assumed). An engineer has experience solving equations. (He solved more than one equation, so you would use the plural form). You could also use "in" but that wouldn't be my first choice. "Of" is not grammatically correct.

"in" sounds more natural. You may use "with" also