He told me that it was silly of me to have spoken to my boss in that way.

In reported speech you could say something like: He spoke to his boss in an inappropriate way. Or He realised that it was silly to speak to his boss that way.

Do you mean in reported speech, or a more diplomatic version of the sentence?

In reported speech