In that case, yeah, hell, you better pay some attention when you are speaking to some one.
Although they can understand you to the most part, if your grammar is not too broken to understand, you better forget about the bs "grammar is nothing, don't give it a damn".
That is because, you don't yet know what grammar really is.
A quick question:
Do you pay much (or even a damn) "attention" to the Chinese grammar when you are speaking Chinese?