Rethinking Out-of-Distribution Detection for Reinforcement Learning
This presentation examines a critical challenge in deploying reinforcement learning agents: detecting when the environment differs from training conditions. The authors argue that existing terminology conflates two distinct types of anomalies—observation-level corruption and dynamics changes—and propose a clearer framework. They introduce three controlled environments that generate temporally correlated perturbations, enabling systematic evaluation of detection methods for both sensor anomalies and state-level perturbations that affect environment dynamics.Script
Autonomous vehicles and robotic systems face a hidden risk: their sensors might fail, or the physical world might behave differently than expected during training. The authors of this paper noticed that the reinforcement learning community has been using inconsistent language to describe these failures, making it harder to build reliable detectors.
The key insight is distinguishing between two fundamentally different problems. Observation anomalies corrupt what the agent senses, like a dirty camera lens. Dynamics anomalies change how the environment actually behaves, like altered friction or gravity. Existing definitions conflated these, but real deployments need to catch both.
The researchers created three controlled environments to test detectors systematically. ARTS generates pure time-series noise. ARNO adds autoregressive noise to observations after the environment computes the next state, simulating sensor corruption. ARNS injects noise before the environment transition, so the perturbation actually influences what happens next.
They implemented scenarios with light, medium, and strong noise levels for CartPole and Reacher environments. The autoregressive structure creates temporally correlated perturbations that mirror real-world sensor drift or gradual physical changes, not just random spikes.
For evaluation, they adapted two sequential change-point detectors from prior work and measured performance using two-sided AUROC, since anomalies might produce either higher or lower scores than normal observations. One unexpected limitation emerged: in Acrobot, state-level noise actually increased rewards rather than degrading them, making that scenario uninterpretable.
By separating observation corruption from dynamics shifts and providing reproducible benchmarks, this work gives researchers the tools to build detectors that can protect deployed agents from both kinds of failure. If you want to dive deeper into how these environments work or create your own video explainer, visit EmergentMind.com to explore the full paper.
