Small and little are the same. If you said "we're having small party" or "we're having a little party", it means the same thing. A bit or a little bit, refers to taking something from a larger thing. Think of it like a piece of cheese. You have the whole cheese, but if you cut off a piece, you have "a little bit" of cheese. "A bit" and a "little bit" are used to describe amounts. The best rule I could think of for you to use is to only use "a bit" or "a little bit" when talking about food.

I'd say that you can use both of them almost in all cases.