It would be more correct to say, "May I use you as a reference?" You could also add the word "please" to add a touch of politeness. "May I please use you as a reference?" It does not sound like you are taking advantage of the person.

Usually people say: Can I use you as a referee for a job that I am applying for?