Grammatically “that” is used to single something out from a group so it sounds natural in defining relative clauses. It sounds less natural in non- defining relative clauses when non-essential information is introduced, however I would imagine there are a huge amount of native English speakers who get this wrong in everyday speech. My advice is not to worry too much about formality or “why” and focus on learning a trick/ rule you can always remember. For example: use “that” for essential information and “which” for extra information.