I would say, "我们在香港的驾驶方向是相反的". To put it more precisely, I would say "我们在香港的驾驶方向是左上右落的. “左上右落”, literally "left up, right down", means "drive/walk on the left".
In HK, "靠左行驶" means to drive in the leftmost lane, i.e. the slow lane., or to keep to the left.
These wordings are applicable in HK. They may be different in other Chinese speaking regions.