No, and I've never been called "bunny," either. However, terms of endearment can be regional, or (in the United States) associated with an ethnic group, or a family tradition... or just one person's own style or manner.
There's a great book, a classic for reading to little children, entitled "The Runaway Bunny," by Margaret Wise Brown. A little bunny tells her mother various ways he is going to run away--"I will become a fish in a trout stream and swim away from you," and the mother replies "If you become a fish in a trout stream, I will become a fisherman and I will catch you." I can easily imagine that in a family where the kids liked that book, "you are my little bunny" could become a term of endearment within the family.