You would say it is dinner time, and it has no 's' in it.
You would use an 's' when there are numbers involved. For example, there are 26 dinners needed for the family meal. But it would be more natural to say meals than dinners when counting or using numbers in a sentence. We had to prepare 26 meals last night. I prepared 6 meals at lunch time. There is no apostrophe needed.