Is this sentence acceptable?
The relationship of the masses of the three balls is m1＞m2＞m3

Question

Answer

Yup. Although I would add a colon after 'is', to give greater clarity.

Answer

When used in physics, "mass" is uncountable so you would not make it plural even it refers tothe mass of more than one thing.

http://learnersdictionary.com/definition/mass

[noncount] physics : the quantity of matter in something
    a star's mass