The expression comes from the way that some surfaces (fur, suede maybe wood) are smooth when you rub them one way, but have a rougher texture when rubbed in the other direction. Going against the grain means going against the way that feels natural and comfortable.