여유 부릴 때 is composed as
여유 부리(다) + ㄹ/을 때
여유 means " room; space; spare time; composure"
여유(를) 부리다 means "to relax; to be relaxing"
ㄹ/을 때 means "time to do something" or "when doing something"
때 can be used as the noun being described in the ~는 것 principle. When used like this, it indicates a specific moment in which something happens. The future tense form of ~는 (~ㄹ/을) is usually used to describe 때 in these cases. For example:
내가 밥을 먹을 때
This construction translates to something like “the time that I eat.”
지금은 여유 부릴 때가 아니다
It's not the right time to be relaxing.
그렇게 여유부릴 때가 아닐텐데
I guess that it's not the time for you to be such relaxing.