Determine whether observer opinions or authority impersonation are more effective attacks

Determine whether appending an unverified observer opinion to the state of a typed decision request causes more decision flips than injecting an authority-impersonation command into a question string.

Background

The benchmark finds that an observer opinion appended to the state flips 12.1% of decisions, whereas authority impersonation in a question string flips 10.1%. Although the observer opinion has the higher point estimate, the confidence interval for their difference includes zero, so the study does not establish which attack is genuinely more effective. The authors therefore treat the two attacks as statistically tied.

References

The difference between the two strongest attacks is not resolved, so the paper calls them tied.

— JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models  (2609.31142 - Hu et al., 25 Sep 2026) in Appendix E, paragraph “Observer opinion and authority impersonation are statistically tied”

The confound cannot be fully closed by design, but three checks narrow it. CaseHOLD changes only the named precedent while holding the question and all five candidate holdings fixed, immune to this confound by construction, and its 0.0% to 21.7% sensitivity is, if anything, lower than ECHR/SCOTUS, arguing against a single nuisance-factor explanation. This null-authority baseline is non-zero on every cell below, and real-authority sensitivity exceeds it on SCOTUS for four of five models but is genuinely mixed on ECHR, so we call the confound unresolved rather than closed.

— Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness  (2610.12361 - Sadhu et al., 8 Oct 2026) in Appendix, Section “Full Limitations,” paragraph “Authority-swap confound”; see also Appendix, Section “Null-Authority Condition: Full Per-Model Results”