"get down to" basically means to get started on something. "get down to party" or "get down to improving" don't really make sense. Usually you hear it as "get down to business," meaning that you need to stop procrastinating and start working.

I agree with Taylor, if you want to use it as "get down to." However, to 'get down,' is a very old slang term and could mean "party" or "dance." Example: "Let's get down tonight." No one uses it this way any more :)