"At the doctor" sounds really strange to me. "Where have you been"? - I've been at the doctor's/dentist's/hairdresser's/tailor's/optician's. None of these examples would sound right to me without the "s".

Yes; both are correct."At the doctor''s" perhaps suggests that you have been at the place (his/her consulting room or surgery) at which the doctor is located whereas "at the doctor" is not so specific as to the location.

I agree with Andy. 'At the doctor' sounds very strange to me, too. 'The doctor' is a person, and 'the doctor's' is a place. If you go to the doctor's, you go to his/her surgery (GB) or office (US). You can also go to the doctor, which means that you go to see this person. If you are 'at the doctor's', you are at the doctor's surgery/office. But I have never heard anyone say 'at the doctor'. To my mind, this makes no sense. You can be at a place, but you can't be at a person.

Both are correct. With "doctor's," the word "office" or "surgery" is omitted but understood. "I've been to the doctor's [office]." The same usage occurs in other contexts. "I'm going over to Kate's to return a book" means "I'm going over to Kate's [house] to return a book." "Dinner at Antoine's" means "Dinner at Antoine's [restaurant]."