"Smeti" and "moći" are modal verbs and in this case both have an infinitive form. "Smeti" means "to dare" or it indicates a permission to do something. "Moći" indicates ability, possibility or a request. Examples: Smem li da koristim tvoje rukavice? (May I use your gloves?/Am I allowed to use your gloves?); Neće smeti to da uradi. (He won’t dare/be allowed to do it.), Ko sme, taj pobeđuje. (The one who dares wins.); Ne mogu da okrečim celu sobu danas, ali mogu okrečiti ovaj zid. (I can't paint the whole room today, but I can paint this wall.); Marijana neće moći doći na posao. (Mariana won’t be able to come to work.)
"Nisam smeo" and "nisam mogao" are in the perfect tense of the verbs "smeti" and "moći". Examples: Nisam smeo to da uradim. (I wasn't allowed to it./I didn't dare to do it.); Nisam mogao to da uradim. (I couldn't/wasn't able to do it.)
Differences between modal verbs can be subtle when it comes to their translation from Serbian to English, but these seem to be the most appropriate in my opinion.