The first thing you need to know about the word 'loo', and to a lesser extent the word 'toilet', is that they are used in British English. Speakers of American English never say 'loo', and only say 'toilet' to refer to the actual object. US English speakers use the term 'bathroom' or 'restroom' in most situations. If you are among speakers of British English, though, 'loo' is the best word to use. It isn't rude at all - quite the opposite. It is friendly, casual, non-offensive, informal and perfectly polite. In fact, 'toilet' is more embarrassing than 'loo', because it's seen as cruder in some ways. Most people in Britain say 'loo' all the time. It's the word I use on all occasions, with family, friends and colleagues. If I'm in a business meeting and we take a break, I would say 'I'm just going to the loo, I'll be back in a minute'. Of course, the two words do have the same meaning. You can say 'toilet' if you want, but it is 'loo' is a nicer word to say in polite company. Toilet is OK, but it's more 'basic'. We are less embarrassed about saying 'toilet' than Americans are, but we still avoid it if we can. If I were you, I'd use 'loo' when you are with British colleagues.

Yes your work colleagues are right, you can use both. You can use some euphemisms if you feel uncomfortable using these words. For example I am going to powder my nose or I need to spend a penny.