To me, you find things irritating (irritating noise, irritating vibration) but people or animal pests annoying ( annoying child, annoying mosquito). Not sure if that is a hard and fast rule though.