'come by' is an idiom. As with many idioms, they just have to be learnt as they are, without trying to make logical sense of them. 'come by' means 'to visit' something and can imply more politeness than 'come to' in the context of your sentence. 'Come to' can sometimes indicate an 'instruction' to come to a place.

Both are grammatically correct, but there is a subtle difference in the meaning. The preposition “to” can indicate a destination or objective. If your friend’s dorm is your destination, there is an implication that you might stay there for some time. The preposition “by” indicates proximity and can have the meaning of “to approach and move beyond.” The implication with “by” is that you will go to a place but only briefly before going somewhere else. I recommend the Merriam Webster Learner Dictionary (learnersdictionary.com) and The Free Dictionary by Farlex as resources. They have lots of examples to help you understand how prepositions are used. Getting a good grasp of prepositions takes some time, but it is possible. Try not to treat it as a brute memorization process. The meanings of prepositions are subtle but they can be mastered.