It usually means, "This makes me nervous or uncertain." Or, "This might be wrong/immoral." Examples:
He told me to meet him here, but there is no one in sight--this doesn't feel right.
We shouldn't be talking about their marriage--it doesn't feel right.