The past and past participle of 'quit' are the same as the infinitive - quit. The older regular form 'quitted' is very rare nowadays.
"My father decided that I should have quit the course" sounds better, but in fact doesn't make a lot of sense.
Do you mean "My father decided that I should quit the course", meaning that he has decided that I should not continue with the course that I'm currently doing? Or "My father said that I should have quit the course", meaning that he told me, at the end of the course, that I was wrong to follow it through to the end and ought to have left part-way through it?
And no, we don't usually use 'quit' in British English. This is an American phrasing. In British English we'd say 'leave' or 'give up'.