It took me 4 hours. - 4시간이나 걸렸어요.
it took me 5 minutes - 5분 걸렸어요.
Maybe I should explain why you are confusing.
'시간' means 'time' but it doesn't mean the number of tries (twice, thrice, 4 times... like that)
'시간' doesn't mean 'the exact time' either (what time shall we make it?')
Sometimes '시간' means 'hours (taken)'.
Someimtes '시간' means 'minutes(taken)'.
Let me introduce some examples. maybe it will clarifying the meaning.
What time shall we make it? (What time shall we see again?) - 언제(몇 시에) 만날래요?
How many times have you visited Seoul? - 서울에 몇 번이나 왔었어요?
Do you have the time? - 몇 시예요? (시계있어?) Maybe you should use "What time is it?" in Korea.
How much will it take to the bus stop? - 버스 정류장까지 얼마나 걸려요?