A is correct. “A rueful smile”, meaning a smile that shows sorrow, pity, shame, or embarrassment, is a set phrase in English. “A decent smile” is not only not a set phrase, it is unclear what it could mean. A smile that isn’t sexually suggestive? Why not say “a demure smile”? A smile that is as enthusiastic as necessary? Why not say “a convincing/winning smile”?

Jane looked at us with a decent smile and left the room Thanks Gary, "rueful smile" is better. But a decent smile is not forbidden to say as well.

Neither is strictly incorrect, but Irene has explained well how b is ambiguous and confusing, so therefore not a useful thing to say.

a is correct