I completely understand this! I think the only way to overcome this is to regularly interact with native speakers in these scenarios - either naturally or via role play.