Yes, it is fine grammatically - it means that you and three others (who are either present, or known to the person you are speaking to) have dinner together on a regular basis.