With whom I had a lot of differences". > This is correct I had a boyfriend with whom we were very different", X I had a boyfriend who is "very different" from me ( to mean "differences in character") After "with whom" , you can use I or any other pronouns , but you cant include him as in "we"

Speaking naturally, you might say : "I had a boyfriend who was very different to / than me." You can't be "different with someone". In your case, you are actually referring to at least 3 people "we" (2+ people) were different to another person (your boyfriend). This is obviously not what you mean. The second one is better "with whom I had a lot of differences" but it's more formal, less unusual and there is an ambiguity; "differences" can also mean "disagreements." So I don't recommend it.