Take your sweet time is a way of expressing annoyance because, like you guessed, they are doing sth too slowly.
Whereas 'take your time' can seriously mean 'no hurry', this is more like 'you are taking a ridiculous amount of time'.
1: He's taking his sweet time getting here!
2: I'm sure there's a good reason, maybe traffic's bad.
1: We've been waiting half an hour!
It means Take your time + some added honey :D
You might want to talk to like this to your girlfriend or anyone you want to sound too sweet to.