Depends on when you're talking to someone. "Guten Rutsch" means you wish someone a good transition ("slide") to the new year, so it's only appropriate at the end of the old year. As soon as the year has started, "frohes neues Jahr" is most common, just like in English.

A native German speaker will hopefully give you a more definitive answer but my German friends always say "guten Rutsch" for Happy New Year, so I think that must be the most common informal/spoken version. However, I have seen "gutes Neues Jahr" in cards. Of course, you can combine them into a longer greeting too - "Einen guten Rutsch ins neue Jahr!"