Bail-Method Sensitivity Mechanism

Determine whether differences between system-prompt versus user-prompt instructions or other factors explain why different bail methods (bail tool, bail string, and bail prompt) lead models to bail on different subsets of prompts, and characterize the mechanism responsible for this method sensitivity.

Background

The paper observes that the subset of prompts triggering bail varies substantially across bail methods. Prompt ablations modified overall rates but did not reconcile the discrepancy in which categories elicit bails.

The authors hypothesized a system-prompt versus user-prompt explanation and attempted ablations (e.g., pre-announcing bail prompt in system or moving it into system), but these did not resolve the discrepancy and even created a new behavior category, leaving the cause of sensitivity open.