Imagine you are on a boat and you plan your trip very carefully. And sudddenly, a thunderstorm arrives and knocks your boat off course. After the storm, you realize you don't know where you are but you know one thing, you must get back to your destination.
This is a literal example of being in uncharter waters.
Now imagine, you work for a company and you plan a big presentation for the President of the Beijing office. Before the President arrives, you realize that you are missing some very important material to make your presentation perfect. You don't know what to do do because you have no experience with figuring out this type of problem.
This is a figurative example of "being in unchartered waters."
So in this situation, you would turn to your co-worker and say, I'm in unchartered waters; which means I have no map to help me take the correct road
It implies you have a problem and you have no experience fixing the type of problem you are having.