'Fans' are supporters. Sports teams and celebrities have fans. If you say 'I'm a fan of ..' , it just means that you like that thing or that person. For example 'I'm not a fan of tofu' mean that I don't like it much. Being a 'fan' is about personal preference.
'Buffs' are both interested and knowledgeable about a topic. For example, a 'cinema buff' is a fan of the cinema, but he/she also knows a lot about the history of the film industry, for example. Being a 'buff' is about knowledge.