Take your sweet time is a way of expressing annoyance because, like you guessed, they are doing sth too slowly.

Whereas 'take your time' can seriously mean 'no hurry', this is more like 'you are taking a ridiculous amount of time'.

e.g.

1: He's taking his sweet time getting here!

2: I'm sure there's a good reason, maybe traffic's bad.

1: We've been waiting half an hour!

It means Take your time + some added honey :D 

You might want to talk to like this to your girlfriend or anyone you want to sound too sweet to.