The term "fair game" isn't from playing a game, it's from hunting. "Fair game" were animals that you had permission to hunt (as opposed to animals that you could not kill because they were on someone else's land or were protected for some other reason).
When you say something is "fair game" you are saying that it is ok to attack (or target or scrutinize) that thing. (e.g., "The politician's voting record is fair game. His children are not.")