Being fond of something or somebody, means that you like it or them. It is less than love. If you say, "I am fond of playing football", it is very similar to saying, "I like playing football." Fondness sometimes suggests a stronger feeling than like, but not as strong as love.
Your wife would not be happy if you said, "I am fond of you." She wants to hear, "I love you." By inference, saying, "I am fond of you", means "I don't love you, but I like you a lot."