Not negative!
It's used here as a playful name, a term of endearment. This person considers you to be his/her friend, so he/she is calling you "silly" as a little joke - saying you are silly to think that you don't talk with him/her very often.
Another example: "I'm not sure if you want me to join you" "Don't be silly! I want you to come with us! "
kind of like 哪裡的話!