This is a really interesting question!
What you have here is the so-called "bare infinitive." A bare infinitive is when the infinitive, in this case "to go," drops the "to" but still functions as an infinitive. The bare infinitive always follows verbs like would, shall, may, and must, but as in this case, can also follow certain other verbs like let, make, and to hear. Go seems to have become one of the those verbs. It seems to happen a lot in conditional cases like the one you ask about.; for example, "I would rather eat nails than study more English!"
Here are some more examples both formal and informal. Consider
I bade him go.
You must learn English.
He had better ask permission.
Let her stay there.
I heard him speak the words.
I would definitely say and write "How about if we go see."