First of all, "balls" can be a rude word, and it is a rude word in this context. It means "testicles." In the United States it was taboo when I was a kid in the 1950s. Now it is more acceptable, but it is still rude.

To "bust your balls" is actually a fairly violent, angry, hostile phrase. It means to hit someone in the testicles, perhaps with a knee, causing intense pain if not injury.

Like some very hostile phrases, it can be a joke under the right circumstances. One might imagine a conversation like the following, between co-workers. But for it to friendly or joking conservation, the co-workers had better be very good friends, and had better speak in just the right tone of voice. Even then, it isn't very friendly. 

"I'm going to kill Jake when he comes back. He left his sandwich in the refrigerator again. Oh, there he is. Jake, I'm going to strangle you if you don't start being careful about leaving food in the refrigerator. Look, it's gotten moldy and the refrigerator stinks."

"Wow, yuck! That is gross. I'm sorry!"

"You should be sorry! This isn't the first time!"

"All right, all right, don't bust my balls over it. I'll clean the refrigerator right now and I'll bring a package of baking soda to put in the refrigerator tomorrow."

Of course we know that the first speaker isn't literally going to kill Jake, strangle him, or hit him in the testicles.