Actually, the sentence "Wow, that's colder outside" sounds like it’s comparing the current temperature to another temperature, either stated or implied (e.g., "colder than before" or "colder than inside"). If you mean "very cold" without comparing, you would say: "Wow, it's so cold outside!" or "Wow, it's very cold outside!" The word "colder" by itself always implies comparison (e.g., colder than earlier, colder than inside, etc.). So, if you want to avoid comparison, use "cold," "so cold," or "very cold."

If you're talking about the weather then it should be "Wow, it's colder outside" (than indoors). If you are talking, say, about a flask with hot coffee inside then it would be colder outside the flask than inside. It is comparing the colder temperature to the warmer one.