You said it best by saying, "after taking everything into consideration."
You can also use "All in all," or "Everything considered," or "When all is said and done,".
"Putting two and two together" is used when certain actions begin to arouse suspicions:
"Without those glasses, Clark Kent looks a lot like Superman, and he is never there when Superman is around. Why can't Lois Lane put two and two together?"