>Would japanese people consider me as a rude gaijin?
Yes, I am sure they do.
>I know all the textbooks say I HAVE TO use polite Japanese (for example masu instead of ru) when speaking to senpai or strangers,
First, I don't agree with you and those textbooks that say "masu" form is a polite Japanese.
I think it is not polite but USUAL Japanese. Look at all the answers and comments here written in Japanse by Japanse people. They all are written in "Desu/Masu" style, so-called "polite way". Then are we all so polite? I don't think so. We only use "desu/masu" because it is usual.
Japanese speech style has 3 layers, "vulgar-usual-polite", like
”してもいい(か）？“, "してもいいですか？“, "さしつかえなければ～してもよろしいでしょうか？” for saying "can I do it ?". You don't have to use the polite one. But I am sure you have to use the usual one (because it is usual).
>I feel much more confident and comfortable speaking casual japanese and it always takes me some time to think of the polite version how to say a certain thing.
Gotcha, but still you have to use "desu/masu", again because it is the usual one.