There's very little difference between these words when applied, for example, to a scene, place, mood or atmosphere. In fact, they are often used together for added emphasis. But please note that while a calm person may be described as 'serene', you can't use 'tranquil' in this way.