"Look."
It comes from the days when ordinary people bought and sold horses. Just as you might check a used car for rust spots or loose ball joints, people knew how to judge the quality of a horse.
According to an agricultural college, "The art of determining the age of horses by inspection of the teeth is an old one. It can be developed to a considerable degree of accuracy in determining the age of young horses. The probability of error increases as age advances..."
If you were buying a horse, you would inspect the teeth carefully to determine how old the horse was. You would "look the horse in the mouth."
In the saying, "a gift horse" means a horse that someone is giving you as a gift, not a horse that you are buying. The saying means that it is rude and improper to judge the quality of a gift.
For example, suppose someone gave you an iPhone. It would be polite to say "Oh, how wonderful! Just what I wanted!" It would be impolite to say "Hmmm... I want at least 128GB of storage, let me turn it on so that I can check to see how much you gave me." That would be "looking a gift horse in the mouth."