그만하다 & 그만두다 are very similar and  have same meaning of stopping what u r doing something.

but 그만두다 has much more meaning of what u've decided to quit because of other people or circumstances around you.

그만하다 is more tended to quit because of your willing.

 listeners r not able to know exactly why you stop) 

그만두다

ex) 1. 매일 야근 때문에 그만두기로 했어. (express  much more by others/circumstances)

       (But you can also say 매일 야근 때문에 그만하게)   

그만하다

ex) 1. 그만해주세요.  참 괴롭습니다...  ㅠ