Myriad English words have Latin/French origin, tracing roots of the origin and meaning behind is rather an historical task at hand. Doesn't help in language learning. Well, on a humourous note, the name develops some sense as a squirrel eats the nut ,holding it close to its chest! Soo chestnut!

According to Oxford languages, here is how the word evolved: early 16th century: from Old English chesten (from Old French chastaine, via Latin from Greek kastanea ) + nut.

Agreed with what Jonathan said. When it's a colour, they're referring to the colour of a chestnut.