Yes, grammatically, it's correct. But it sounds not comfortable.
The reason for that, is:
"老师的话我听懂得很" up until here, a native speaker would think it's a positive sentence, but very abruptly "少" change the the meaning to negative. so it's not so comfortable.
The better way to say it would be:
"老师的话我听不太懂"
“老师的话我只听得懂很少”