Hi Rous,
There are a few ways of crafting a sentence with "despite".
1. We managed to sleep despite the loud music.
Notice that we can use a noun phrase (the loud music) after the word "despite".
However, if we are using "in spite", then it has to be used with "of".
2. We managed to sleep in spite of the loud music.
We do not use "despite of".
Hope this helps you.
Cheers, Lance