やってみようかな=I think I will (try to) do it, I think I will give it a shot.
やる=to do, みよう=to try. But you don't necessarily need to translate all of the two into English.
かな means "I think/wonder".
These English combinations would depend on the situations in terms of the nuances.
A: You may want to/Why don't you learn a new language.
B: I think I will. やってみようかな. I will (no "I think" part)=やってみます.
A: Why don't you install the software?
B: (It seems difficult but) やってみようかな. I think I will give it a shot/I think I will try.
A: I recommend this book.
B: I think I will read it. よんでみようかな. Specifying the verb in this case sounds natural.
~かな usage for the "I wonder" version.
A: I wonder if it will rain tomorrow. あしたは雨がふるのかな.
I hope this helps.