I cannot think of an example where we would say “few years ago” without also using the “a”. It sounds wrong without the “a”.

few and a few have very different meanings, they are not interchangeable . few = an insufficent number i.e. not enough a few = some

I think few years ago could be used in short answers and in the beginning of sentences (I'm not sure about it, though). Example: - When you moved here? - Few years ago - How you decided to live here? - I had the decision talking to my husband a few years ago