The correct way of saying it is "My English is not fluent." You're right - 'fluency' is a noun. Therefore, we cannot use it too describe 'English', which is also a noun. We need an adjective, like 'fluent'. Hope this helps. :)

My English is not fluent

you can say my i am not fluent in English or my English is not fluent.