If you don't have an specific cafe in mind, then "Let’s go to a cafe." would be the correct sentence. In this example you could say, "Let’s go to a mart." but I think it would be more natural to say, "Let’s go to the mart." even if you don't have a specific one in mind. Let me know if you'd like a little more explanation :)

With no specific cafe in mind, you have to say "Let's go to a cafe". For mart both 'a' or 'the' are possible when you aren't thinking of a particular mart. The reason is, some people will think of 'mart' in the general sense like you think of 'shops'. So similar to saying, "Let's go to the shops", they say "Let's go to the mart". But others will still say "Let's go to a mart" when they have no special mart in their mind. Myself, I prefer using 'a mart' when I have no particular mart in mind. This is probably because 'mart' isn't commonly used where I am, so I think of it in the same way I think of a 'cafe' rather than in a more general sense.