He is agreeing with Tom. He's saying 'No, it's not a good idea.'
That's how we agree in English. The 'yes' or 'no' word goes with the grammar of the sentence you're agreeing with.
For example 'It's not very warm today, is it?'
'No, it isn't'
The second person is agreeing with the first.