It has a formal tone, so it's not frequently used in casual speech. Other options like "rarely" or "not...often" are used more. Sometimes you'll hear "seldomly" in spoken English, But it's not really a proper word, so it should never be used in formal speech or writing.

It is common. It is not formal. It's a good word - use it.

I use it from time to time

There's no reason to avoid it. There's no need to use it. English is rich in synonyms, and they slowly come in and out of fashion. I have a quick test I use to see if some word is too formal or has fallen out of use, and that is to search Google News. Because it is news, most of it is contemporary English--and most of it is standard English that is grammatically correct. I quickly found the following, and am convinced that the word is commonly used in everyday English. LeBron James, seldom labeled a pure scorer, eyes the NBA points summit Dear Abby: My parents insist we visit, but they seldom visit us How a Company Recycles Seldom Used World’s Largest Passenger Plane “Ringtails are fairly widespread throughout Utah but seldom seen,” the department said on Facebook A Brief Guide To Often-Heard But Seldom-Known Aviation Jargon Caribbean Mosquitoes Carry a Seldom Discussed Travel Disease

In my experience, seldom is more commonly used by non-natives than natives. It's fine though, but, as Jonathan says, it has a formal tone.