It means "to think hard, searching for an answer to a troublesome, difficult problem."
Literally, it means to find the answer to a numeric or arithmetic problem. The terms "number," "numeral," "digit," and "figure" are roughly synonymous. A person thinking about buying a car might look at the price of the car, their income, and their savings, and do some arithmetic. He might say "I'm trying to figure out if I can afford to buy that car."
By extension, it means to think hard about every possibility, and try to find a solution to a complicated problem.