I would not say "agreement ON an idea" I think most American English speakers would say: in spite of agreeing with someone/someone in spite of agreeing to something In spite of agreeing with the general solution, John had some reservations about the cost. In spite of + gerund

"In spite of my agreement on this idea," is correct but it is not a complete sentence. You would have to say something like, "In spite of my agreement on this idea, I still have some reservations." And mostly in America we just say, " In spite of my agreement, I still have some concerns." You don't need to say, "on this idea," that is implied.