There's almost no difference. "I am attracted to her" puts the emphasis on me as the active one. The mental picture is of me moving in her direction. "I am attracted by her" puts the emphasis on her as the active one. The mental picture is of her somehow pulling me toward her, by the force of her magnetic personality.

You’re right — in actual practice, the meanings are quite similar. “X is attracted to Y.” This describes X using “attracted” as an adjective. The prepositional phrase is used as an adverb “to Y” giving us specific information. “X is attracted by Y.” This is the passive voice. The active version would be “Y attracts X.” The passive construction is preferable if were are primarily interested in X, rather than Y.

Generally you are attracted to something and attracted by someone. I may be attracted to the light (like moth!) or attracted to my husband.