The sentence is perfect because whom is in the object case, and the object case is triggered by "to." Most commonly people do say "who you go to" like Joe said. You can say to whom you go, of course, but that's a little haughty sounding to me and most people don't talk that way. Or even write that way. That's that old "never end a sentence with a preposition" rule someone invented someplace that kids are taught in school and promptly forget because it bears no relation to how people really speak. What I find a bit confusing is the English "I see who is now responsible." I see whom you are talking about, would be right, but to say "I see who is responsible" has to be in the indicative case standing alone, yet one would think the verb "to see" should take an object and the only one in that sentence is "who", but "is responsible" needs a subject in the indicative case. So "who" can't be in both cases. The problem is the sentence is so short and not divided by a comma we really don't see it as two separate sentences. I don't know. English is just weird like that. Other languages, seem to avoid this confusion by breaking this into two easily identifiable sentences. That seems clearer to me.

Yes, your phrase is formal and technically correct, but this one is better: 'I know to whom you go.' The more common, though informal, form of the phrase would be, "I know who you go to."