Hi Alexandros, good question!
In the second sentence, 'dat' means 'those', so it could be literally translated to 'those are Jan's books'. However, if you would like to say 'those books', it would be 'die boeken', so I can see how it is confusing. 'That book', on the other hand, would be 'dat boek'. The reason 'dat' is used here is because of how the sentence is structured. For instance, you could say 'Die boeken zijn van Jan' (Those books are Jan's). In the same fashion, the first sentence could be written 'Zij zijn Jan en Peter' (They are Jan en Peter). 'Het zijn Jan en Peter' rather means 'It's Jan and Peter', which I think you could say in English too. Like, hey look, it's Jan and Peter. 'Dat zijn de boeken van Jan' could be an answer to the question 'What is that?' or 'What are those?' Okay, I am not sure this cleared anything up, but I hope it helped a little, haha.