"He can't have my phone number." ...has two meanings:
1. I refuse t give him my phone number.
2. It is highly unlikely (deduction) that he has my number.
The meaning is taking from the context.
By the way, it won't help if you classify modals into groups. It just clutters up your mind with useless information! Learn the modals one by one, exactly as I have shown you above. It is much quicker, and more natural.