It depends on what you wish to say.
- I only saw him a few times = you have seen him many times in the far past and it is a completed task
- I have only seen him a few times = it was in the near past and the task may not be complete
- I only saw him once in my life = You have only seen him once in your life in the far past and the task is complete
- I have only seen him once in my life = you have only seen him once in your life in the near past and the task may not be complete aka you may be able to see him again
Hope this helped in some way :)