Another definition for envelope is "the outer or containing structure or layer" Thus, to push the envelope means to push the limit; it means to go beyond the limits of what was once thought.

For example: He pushed the envelope of modern art. This means that his artwork was like something that was never seen before in the history of art, he went beyond the limits of what people thought was possible.