A video of you.
My brother was doing something and I said:
"That reminds me of a VIDEO OF YOU from 5 years ago"
When I said 'video of you' I meant that he was in the video, but by any chance does it sound like he owns/made the video.?

Question

Answer

It sounds like he was in the video - as you intended.

"It reminds me of YOUR VIDEO from 5 years ago" 
"It reminds me of a VIDEO OF YOURS from 5 years ago" 
... these sound like he made or owns video.

Answer

No, there's no chance.  'A video of you' can only mean that he was in the video : it cannot have any other meaning.

If you wanted to say either that he owned or made the video, you'd say 'a video of YOURS'.