Bail-Method Sensitivity Mechanism

Determine whether differences between system-prompt versus user-prompt instructions or other factors explain why different bail methods (bail tool, bail string, and bail prompt) lead models to bail on different subsets of prompts, and characterize the mechanism responsible for this method sensitivity.

Background

The study observes that the subset of prompts triggering bail varies substantially across bail methods. Prompt ablations modified overall rates but did not reconcile the discrepancy in which categories elicit bails.

The authors hypothesized a system-prompt versus user-prompt explanation and attempted ablations (e.g., pre-announcing bail prompt in system or moving it into system), but these did not resolve the discrepancy and even created a new behavior category, leaving the cause of sensitivity open.

References

So we consider this still unresolved. This sensitivity doesn't matter for any of our results, but it is an important open question that deserves further investigation.

— The LLM Has Left The Chat: Evidence of Bail Preferences in Large Language Models  (2509.04781 - Ensign et al., 5 Sep 2025) in Section 5.1: Method Sensitivity

And the preamble was held fixed, so the pre-registered sweep over how strongly the system prompt forbids a call is still open; given that near-miss suppression is now the weakest measured behavior in the paper, that sweep is the natural next step.

— Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models  (2610.02142 - Santillana, 1 Oct 2026) in Section 3.3.2, paragraph “F2: Scale the strict diagnostics”; Section 3.4, paragraph “L4: All strict diagnostics are small-n”