I feel that there is no need to use "the" because we are not referring to specific underground hot water in a specific area of Iceland. It is just underground hot water in general.
May I suggest that we shift the part "in the future" towards the end of the sentence as it will not break up the flow of ideas between (i) what they believe and (ii) making use of the ground water, which is a description of what they believe.
Therefore, I will write:
The Icelanders believe that they can make use of underground hot water for their benefit in the future.
Hope this helps.