It depends. If you want to keep him in the friend zone, you could say "아직 서로 잘 모르니 사양할게요. " If you could see your self on a date with him in the future, then you can put him in his place by saying "좀 더 알아가는게 좋겠어요" :)