A set of keys is specifically a collection of keys that is meant to be together, probably on some sort of "keyring". You might have a set of house keys which contain the keys for all the doors in your house on a keyring. Or a set of shed keys that included all the keys for your garage, sheds, etc. I think that a group of keys is just some keys put together - grouped. They may not have anything in common. You wouldn't often say a group of keys in normal conversation.

Group is a more ambiguos term. You are more likely to use set on a group of non living things than a group of living things. A chess set. A steak knife set. A set of tools, or a set of keys. Unlike a group of people or living things(which there are soooo many different words for groups of animals, native speakers dont know them)