I rather hear and use ‘I didn’t go anywhere’

It would be natural to give the one-word answer 'Nowhere' but 'I went nowhere' sounds unnatural. As Patrycja says, 'I didn't go anywhere' is more natural.