You’re right — in actual practice, the meanings are quite similar.
“X is attracted to Y.” This describes X using “attracted” as an adjective. The prepositional phrase is used as an adverb “to Y” giving us specific information.
“X is attracted by Y.” This is the passive voice. The active version would be “Y attracts X.” The passive construction is preferable if were are primarily interested in X, rather than Y.