If you say 'I went to class', this is similar to saying 'I went to school'. It means that you left your house and went to your class. If you are 'in class' this tells us where you are - you are not 'at home', you're 'in class'. 'I went to classes' means that, at some time in the past, you attended several different classes in different subjects, or that you attended a course that consisted of several classes.

class is a singular noun classes is plural

'Class' refers to a place or group of people, whereas when you speak about 'classes' you have in mind learning activity

class -> singualar classes -> plural