Had been working :) This works best because the action is ongoing. Hope this helps!

Both are correct when put into context. He had been working for half an hour when he noticed he was painting the wrong wall. He kept on painting but changed to the correct wall. Later he explained his mistake and his wife forgave him. He had worked for half an hour when he noticed he was painting the wrong wall. He was so disgusted with himself that he stopped working and went to the pub. His wife was very upset and made him sleep with the dog that night. Context is always stronger then grammar. It is also more fun, interesting and creative. I hope this helps. Have fun

Had been working is correct. You could say "He had worked for half an hour before he realized...." to make the other option correct. But with the conjunction "when" you need the perfect tense, "had been working".