It is appropriate to say to anyone that has a close relationship with the deceased, though it is most commonly said to relatives. You can also say: "I'm sorry for your loss."