It means you travelled to a place, probably the suburb, which is only 5 mins away from the city, and most of it is nice and calm.

5 minutes from the city, it’s very nice and calm. (There is an unstated assumption that, in the city, it’s not nice and calm. The speaker and listener already know that. ‘For the most part’ qualifies the statement. So occasionally the negative factors associated with being ‘in the city’ show up here.)

I interpret "for the most part" as an adverbial phrase that modifies "really". (Adverbs can modify each other.). "Really" is a word that emphasizes. "For the most part" lowers the intensity of that emphasis. The meaning does not change if you move the phrase anywhere else close to "really": It’s really, for the most part, nice and calm to be five minutes from the city. It’s for the most part, really nice and calm to be five minutes from the city. It’s really, for the most part, nice and calm to be five minutes from the city. I deleted "just" and "pretty". You are using too many emphasizers and de-emphasizers ("just", "really", "pretty", "for the most part"). Allow your words to speak for themselves.