Johnson is insulting the Earl of Chesterfield for his early lack of support to Johnson's Dictionary.
He is referring to the poet Virgil's story of Damon in the 8th Eclogue, who sings a sad song of love and heartbreak. In the quote, "Love" is a personification of the emotion, who, by living among the rocks, is proven to be a cruel and heartless character.
The fact that Johnson wrote the line in English is taken to be an insult in itself, as educated men would have discussed the line in the original Latin.