Think of conmigo, contigo, and consigo as irregular. All the other ones are regular: con él, con ella, con usted, con vosotros, con vosotras, con ellos, con ellas. If you’re curious, the “-go” was originally the Latin preposition “cum” (from which we get Spanish “con”) used as a postposition. So in Latin, we had: mecum, tecum, secum, vobiscum, nobiscum. So, in Church Latin, they use the phrase “pax vobiscum” (peace be with you), a translation of the Hebrew shalom aleikhem, cognate with Arabic as-salaamu aleikum. As is usual in going from Latin to Spanish, case endings were simplified, the intervocalic /k/ changed to /g/, the final (unaccented syllable) /m/ disappeared, and the post-tonic /u/ changed to /o/. This gave us: migo, tigo, nosco, and vosco. The preposition con (from “cum”) was then placed (redundantly) in front of the pronouns that already had the postposition. In modern Spanish, “connosco” and “convusco” are no longer used (although their cognates are used in Portuguese), but conmigo, contigo, and consigo remain as the only way of combining “con” with the 1st and 2nd person singular pronouns, and the 3rd person reflexive pronoun.