1) Раньше у меня было две кошки. Can I use были instead of было here?
Both are acceptable, but "было" sounds much better. I don't know if there is a strict rule about it, but in real speech both forms are used, thougt in specific context one may sound better than the other.
2) У меня будет долгий отпуск следующий год. Why is будет used here, not буду?
Because it's not "I will", it's "the leave/vacation will be". The subject is "отпуск". 3rd person.
I will = я буду, he/she/it will = он/она/оно будет.
And it would be better to say "У меня в следующем году будет длинный отпуск" or "На будущий год у меня будет длинный отпуск". "длинный" is a more casual form, "долгий" sounds a little pompous or obsolete.
3) In which situation do people use Я занята or Я занятая?
Usually when somebody asks you to do something while you are busy, or you have something urgent to do. For example your friend is calling you and is going to tell how she spent her holiday, and you have to finish you work and have no time to listen to her, you may say "Извини, я занята, мне нужно доделать работу".
4) Я учу матемАтику. Does it mean "I teach maths" or "I learn maths"?
I learn. A person who teaches would say "я преподаю математику".
5) Ты слУшаешь менЯ? Does it mean "are you listening to me" or "do you listen to me"?
It may have both meanings, depending on context.
6) не могли бы вы помочь мне? Is this sentence more polite than Вы мОжете мне помОчь? Why is "не" needed here?
Yes, it's more polite. It's a polite request, while the second is rather a neutral question (when you don't ask to help you but really want to know if a person can help you) and in certain situations it may sound like an insistent demand. To use "Не" in such situations is a common custom, I think it's purpose is to emphasize the fact that you are ready to accept any answer. Is not it so in English? "Won't you help me please?"