There's the explanation that this phrase is from the usual meaning of Devil, i.e. the supreme spirit of evil. If it's that Devil we are talking about, then the origin of this idiom is straightforward - the Devil is bad and falling in the deep sea is bad, so when caught between the two , we would be in difficulty.