If you feel really bad and unfortunate toward someone, that means that you feel bad for them. If you feel bad and unfortunate "with" someone, then that means that you are sharing in their bad feelings or poor situation.
Usually we say, that we feel bad or unfortunate "for" someone.
I hope that helps!