You can say "They ordered her to return the book." When you do that, you are not changing the mood of any verb. Rather, you are replacing one noun clause ("that she return the book") with a different noun clause ("to return the book"). Infinitives act as nouns, adjectives, or adverbs. They serve very well to create noun clauses. The one thing infinitives are unable to do is to act as verbs. An infinitive never has a subject, whereas verbs do. "That" has the ability to transform any sentence into a noun clause: "I like that she sings well." Now, "she return the book" is not a good sentence because the subjunctive mood needs some verb that fits it, but "that she return the book" is a good noun clause using subjunctive mood. Other nouns work just as well: "They ordered her cooperation." Present participles can also be used this way: "They suggested her going to school" in which "her" is an indirect object and "going to school" is a noun phrase acting as the direct object.

Verbs like "suggest," "demand," and "insist" usually require "that + base verb" (subjunctive): * "They demanded that she return the book." Using "to" works with some verbs, like "order" or "request," but not with "suggest" or "insist": * "They requested him to leave." (correct) * "She suggested him to leave." (incorrect).

Not mandatory, but certainly sounds better. I can only speak to us here in England (as opposed to Americans) but we rarely use the subjective anyway, especially in conversation. We’d usually use “should”.

They suggested/ordered/demanded/requested/insisted (that) she return the book. (Good, ‘that’ is optional) They ordered/requested/demanded her to return the book. (Good) They suggested to her to return the book. (Ok) They insisted to her to return the book. (No, only the subjunctive sounds right.) If you said ‘returns’ in the first case, it wouldn’t sound terrible but could be unclear. For example, ‘They insisted that she returns the book.’ has an indicative meaning: ‘They are sure that she returns the book. (habitually)’ American English

The 'to' form ie the infinitive is usually used where the subject 'they' is performing the action : 'They demanded to return the book'. The subjunctive is used when the subject making the action is part of a subsidiary clause forming the object of the principal clause : 'They demanded (that) she return the book'. It's more elegant than using the infinitive 'They demanded her to return the book'.