In my experience "of the world" is more metaphorical and not really common. You might say "he's a man of the world" to refer to someone who is well-traveled. In the world might be to explain the uniqueness of something. Like "this is the best cheese in the whole world!" to express that you really, really like that cheese.