Speaking naturally, you might say : "I had a boyfriend who was very different to / than me."
You can't be "different with someone". In your case, you are actually referring to at least 3 people "we" (2+ people) were different to another person (your boyfriend). This is obviously not what you mean.
The second one is better "with whom I had a lot of differences" but it's more formal, less unusual and there is an ambiguity; "differences" can also mean "disagreements." So I don't recommend it.