Either; ...I don't have your number. (I didn't write it down, or I lost it) ...I don't remember your number (remember is better than memorize here) Actors 'memorize' there lines, so they can say them without reading them.

Yes, the sentence is grammatically correct and sounds natural. However, to make it slightly more conversational, you could also say: "I couldn't call you because I haven't memorized your phone number." "I couldn't call you because I don’t have your phone number saved in my memory." The original is perfectly fine as it is!

It’s great. Note that ‘Having a number’ ‘Remembering a number’ and ‘Having a number memorized’ have three different meanings. Yours is very good, if that is what you meant!