The meaning in this context is "you are no fun!". One meaning is literal like the one above, that was posted by Gary. The one in Family Guy meant that Brian is no fun. Example: You prank your friend by telling him his important watch is missing. You then tell him after he is looking for 10 minutes, that you took it. He laughs it off. He is considered a "good sport". The opposite would be that he yells at you or starts complaining about the joke. He would then be a "bad sport". Basically, it was meant in the context that he was not being fun. I can't find the video exactly but based on what you provided, this seems like the usage intended. Meaning: "You are not fun". It can only be used after something "fun" was done or attempted, but the other person reacted in a somewhat negative manner.

a 'good sport' is someone who plays fairly, and is respectful to their competitors. A poor sport would be the opposite. I don't know (and don't want to look up!) the context to know if this is appropriate. It's not a common usage.