As the commenter said, it can mean a "gay" (as in effeminate or homosexual) way of speaking. However, it can also have a broader meaning, such as peculiar or odd or dumb. "Gay" is a very broad and common insult in America, which nowadays can mean almost anything.
Although it is never a compliment, it reflects badly on the speaker, not on his target. Only a rude and low-class person would seriously insult another person this way.
There is one exception I know of. Among groups of friends, especially teenagers or young people, it can sometimes be a friendly and harmless insult. One friend might say to another, "that hair style looks gay", and both people would laugh.
The word has become so commonly over used in America, you pretty much have to judge by context. It is, as the commenter said, never a compliment, but how serious or playful an insult it is depends on context.