It means "He ruptured himself while he was playing football". He was playing football, and at some point during the game he ruptured himself.
I'm not quite sure what you mean by "He ruptured himself in the process of playing football". Isn't that the same as 'while playing football'?
"He ruptured himself by playing football" is possible, but less likely. Football is generally a relatively safe game, and the mere fact of playing football should not lead to a rupture. This would only be the case if he had a previous injury or wound, and it was the simple activity of running and kicking that was to blame for the rupture.