Usually, when you approve something, you are making a formal decision, like approving a project. To approve of something is stating your opinion as to the suitability of a situation or of someone else's choice. For example, approving of someone's choice for a spouse. In either case, you can be said to "give your approval".

Approving something means you're directly giving permission for something. Approving of something is more passive, and has more to do with your opinion regarding some situation. So if you approve of the terms in a budget, then you're also likely to approve the implementation of that budget.