They both mean you're wrong, but they differ in degree. Let's say you're multiplying 4 x 4, if your answer is 17, you are wrong if your answer is 9302, you are way off

Yes, it's more or less synonymous with someone/something being incorrect. For example, you could say "you're way off the mark" or "that answer is way off."