To be off is a colloquial phrase meaning the start of. So if he’s off to a rocky start it means that things have started poorly. In horse racing they say “and they’re off” which is equivalent to saying they “and they started”. Hope this helps!

He has started badly. ‘Off to’ just means started.