I've taught many Spanish-speaking students over the years, and I am almost certain that you are making the mistake of pronouncing the word 'worm' with an 'or' sound as you would in Spanish - like the English word 'or'. Am I correct?
Are you pronouncing the word 'worm' to rhyme with 'form' and 'dorm' ? If you are, you need to realise that you are pronouncing it wrong. Don't let the spelling confuse you. Here's a tip - the letter 'w' represents a semi-vowel sound, which means that the actual vowel sound which comes after probably has an irregular pronunciation.
In fact, the word 'worm' ( along with 'word', 'world', 'worse', 'work' and other common words beginning 'wor..') has an UR sound, not an 'or' sound. The vowel in 'worm' is same sound as in 'earth', 'turn' and 'girl'. The word 'worm' rhymes with 'term' and 'firm'.
So what about 'warm'? Well, confusing as it might seem to a Spanish-speaker, it is the word 'warm' which actually rhymes with 'form' and 'dorm' (not 'worm').
Try practising with the following pairs of rhyming words. Say the first word in the pair, and then force yourself to say the second (your target word) with exactly the same sound. Forget about the spelling and just concentrate on the sounds!
I hope that helps!