It's funny simply because it sounds strange, because we just don't use the word "the" when we talk about family members. It sounds like you're talking about a specific grandpa, but we have no idea WHICH specific grandpa you mean. You can say "I played with the dog," or "I talked to the mailman," but you can't say "I played with the son" or "I talked to the sister." You need to say "my son" or "your son" or "our son," etc. If you're talking about your own family, or the family of another person who is part of your conversation, you simply can't say "the." You can say "the grandpa/mother/father/sister/aunt/etc." if you're talking about a different family, especially if you don't know the family (so you can't use their names). For example, "I saw a family with a new baby playing at the park yesterday. The mother was wearing a blue dress just like mine." You can say "the mother," because you're referring to a specific mother, but you don't know her, and you don't know whose mother she is -- to you, she's just "the mother I saw at the park." If you can identify her as a specific person's mother, then you should: "Mary's mother," "my friend's mother," "my boss's mother," "the mailman's mother," etc.)

Also "the grandpa" can mean a very old person or some person who acts like an old person, e.g. "The grandpa was the cause of the accident". Usually with a negative connotation about the person.

“The” is an article that indicates only one kind of that noun or reference to a specific item. So, for example, in “Can you hand me the notebook in my bag?”, the speaker only has one notebook in their bag. When it comes to your question, I think it depends on what you were referencing. If it was your own grandfather, it would be “my grandpa” so using “the” might make him laugh, since you’re basically saying there is only one grandpa in all the people in the world. If you are talking about his father, then it would be “your grandpa,” since the grandpa belongs to him. Does that make sense? If not, just let me know.