Create and explore imaginative 3D worlds with our new product Dreamfield (opens in a new tab)

Evaluate safety routing on genuinely complementary agentic vulnerabilities

Determine whether pre-execution or request-conditional safety routing provides meaningful benefits in agentic environments containing several models with genuinely complementary vulnerabilities, rather than environments dominated by a single near-perfect configuration.

Background

The AgentDojo analysis is based on a saturated pool in which one configuration is already close to perfect, so nested routing largely selects that configuration or the honest fixed baseline. The authors explicitly state that this design cannot address the setting for which routing is intended: a pool in which different models fail on different scenarios and no single model dominates.

An artificially de-saturated analysis produces apparent routing gains, but the pool was constructed by dropping configurations according to their measured harm on the same data. Because that procedure introduces hindsight selection, the result cannot resolve whether routing works in a naturally complementary agentic model pool.

References

No corpus we have exhibits that naturally, and our attempt on a live skill-injection surface returned AUROC \num{0.5000} under complete-case filtering and is void (Appendix~\ref{sec:experiments}). This is the principal gap in the evidence, and we do not treat the agentic result as closing the question.

— False Floors: LLM Safety Routing Evaluations Break Under Distribution Shift  (2610.01535 - Bhatti et al., 1 Oct 2026) in Section 5, “The Agentic Setting,” paragraph beginning “The corpus cannot settle the case”; Section 5.1, “An artificially de-saturated pool”

Their binary task outcome flips far less often, in \num{5} of \num{358} swap branches and none of \num{359} controls, but that rate is the same order as the entire pre-execution headroom, so a replay approximation would carry an error the size of the quantity it estimates. Whether that flip rate transfers to a harm label is itself unestablished.

— False Floors: LLM Safety Routing Evaluations Break Under Distribution Shift  (2610.01535 - Bhatti et al., 1 Oct 2026) in Appendix, Section “Cascade Diagnostics,” subsection “Why mid-run switching is a separate estimand”