You shouldn't avoid it. It isn't an English letter, but there are certain words in the English language which we've taken directly from French and so they include this letter. It's not a question of choice or preference - the correct way to write this word is 'café'. It's pronounced 'ay', like in the word 'say'. Other words which have this letter are fiancé , cliché and blasé.

I would say that both café and cafe are correct. Most keyboards in the US do not have a way to type é, so I find many, many places that call themselves a cafe. Some places try to make up for the accent with an apostrophe: cafe'. To me, this looks ridiculous and uneducated, but I like both café and cafe.