Don't Fool Me Twice: Adapting to Adversity in the Wild with Experience-Driven Reasoning

Abstract: In robotics, dangers and adversity modes are often embodiment-specific and relative to each agent. A frontier of autonomous mobile robotics is to enable agents to operate effectively in the wild in unseen unstructured environments. A significant challenge in unseen unstructured environments is that it may not be possible to predict all the dangers to the specific robot. Although recent work has used large foundation vision-LLMs (VLMs) to preemptively predict an exhaustive list of common-sense dangers, it remains difficult to capture possible interaction and embodiment-dependent adversities. We propose a continual learning framework for a mobile embodied agent to learn online from disturbances and attribute anomalous behaviours to causes through semantics, enabling better prediction and planning of the world in the future. Our framework, "Don't Fool Me Twice", first observes disturbances and describes their effects on the robot; this description is augmented with visual context to query a VLM to predict possible causes; the local disturbance is characterized using kernel regression, which allows for efficient, few-shot modeling of transient anomalies. We leverage semantic voxel-centric modeling to estimate epistemic uncertainty, enabling richer downstream recovery by treating interaction-driven disturbances as learnable spatial behaviors. We present four hypotheses and validate them in simulation and on hardware across embodiments and adversity modes.

• The paper introduces DFM2, a novel framework that uses event-driven multimodal anomaly detection fused with semantic reasoning to build an adaptive hazard library.
• It employs a dual-loop architecture to rapidly detect anomalies and subsequently refine spatial disturbance maps using kernel regression and Bayesian inference.
• Through simulation and hardware tests, DFM2 yields an 81.8% survival rate and significantly reduces trajectory disturbances compared to traditional methods.

Experience-Driven Reasoning for Embodied Robots: An Analysis of "Don't Fool Me Twice"

Introduction

"Don't Fool Me Twice: Adapting to Adversity in the Wild with Experience-Driven Reasoning" (2605.31119) addresses adaptation in autonomous mobile robots operating in unstructured and unfamiliar environments. Instead of relying solely on exhaustive pre-trained semantic hazard lists, the framework (DFM2) proposes continual online learning of hazards based on actual robot experience. Through an event-driven combination of multimodal anomaly detection, vision-LLM (VLM) reasoning, semantic mapping, and kernel-based disturbance modeling, DFM2 enables robots to dynamically construct and refine an adversity library, guiding future planning and robustifying operation against unknown or embodiment-specific hazards.

Framework Overview

DFM2 organizes experience-driven reasoning into two interconnected loops: a fast loop for online perception, risk detection, and avoidance, and a slow loop for attribution, semantic grounding, and disturbance field modeling. As the robot traverses its environment, anomalous deviations in operational signals (e.g., trajectory tracking error) trigger event-driven queries to VLMs, identifying the semantic cause of disturbance and updating the internal danger library. Critical to this approach is the explicit modeling of spatial disturbance fields using voxel-centric kernel regression, allowing the agent to move beyond binary hazard representations to sample-efficient, geometry-aware hazard anticipation.

Figure 1: DFM2 pipeline interleaving fast event-driven anomaly detection with slow post-hoc semantic reasoning and adaptive hazard library update.

Event-Driven Anomaly Detection and Causal Attribution

DFM2 introduces an operational monitoring signal $\nu(t)$, measuring task performance indicators (e.g., position tracking error, pose estimation uncertainty). An anomaly is registered when $\nu(t)$ exceeds a calibrated context-dependent threshold, at which point relevant sensor and state data are buffered and a multi-modal narrative is constructed.

Figure 2: Workflow of event-driven multimodal narration and structured VLM prompt generation for targeted causal attribution.

In contrast with proactive approaches that preemptively query VLMs on every scene or at high frequency, DFM2 triggers VLM causal reasoning only at event onset, focusing attribution on temporally and spatially localized disruptions. Dense visual-semantic embeddings from multi-encoder vision foundation models (NARadio, DINOv2, SAM, SigLIP) provide detailed spatial semantics, which are used to ground identified hazards at the voxel level.

Semantic Voxel-Centric Disturbance Modeling

Rather than representing hazards as binary forbidden regions, DFM2 employs anisotropic RBF kernel regression over local semantic voxel embeddings to characterize the spatial distribution and magnitude of effect for each detected hazard. Within the slow loop, this regression is performed per semantic cause, coupling learned spatial disturbance templates (shape parameters) with amplitude and bias inference via Bayesian Linear Regression to capture epistemic uncertainty due to limited data.

Figure 3: Hazard anticipation using semantic voxel characterization—showcasing comparison between proactive, fixed avoidance, and DFM2's adaptive, sample-efficient spatial field learning.

The resulting models enable the robot to predict both the effect and the uncertainty associated with re-encountered hazards, which is instrumental in proactively planning safe trajectories without excessive conservatism.

Evaluation in Simulation and Hardware

The experimental validation covers both simulated and real-world physical platforms. In simulation (Isaac Sim with aerial robots), DFM2 demonstrates the ability to (i) substantially increase survival rates, (ii) reduce cumulative trajectory disturbance, and (iii) generate shorter, more efficient paths compared to both geometric and proactive semantic baselines.

Quantitative highlights:

• Survival rate: DFM2 achieves 81.8% survival, a significant improvement over proactive reasoning (59.1%) and geometric-only planning (40.9%).
• Cumulative disturbance: DFM2 reduces normalized cumulative disturbance by over 3× relative to the next best approach.
• Arrival time and path length: DFM2 plans more direct paths by avoiding unnecessary detours around innocuous objects.

  Figure 4: IsaacSim warehouse testbed with randomized placements of disturbance sources across navigable aisles.

In hardware, the methodology generalizes to wheeled robots subject to perception-induced localization failure (e.g., large textureless surfaces or active IR overexposure). DFM2, when integrated into the navigation stack, leads to significantly:

• Enhanced inlier correspondences in odometry,
• Lower pose covariance traces,
• Shorter durations spent in degraded localization states.

  Figure 5: Wheeled robot demonstration platform with RGBD, LiDAR, and IMU for testing perception-induced adversity.

  

  Figure 6: Detection, mapping, and characterization of adversity from feature-sparse (black) surfaces—DFM2 marks regions of perceptual hazard unobservable to na\"ive geometric planners.

  

  Figure 7: Recognition and spatial mapping of overexposure-induced localization failures during active robot operation.

  

  Figure 8: Mapped office space with detected adversity zones (red) and traversable regions (green) under DFM2 planning.

Theoretical and Practical Implications

DFM2's architecture exposes several strengths:

• Embodiment-specific adaptation: By tying hazard library formation to operational experience, DFM2 learns what is hazardous for a given robot, effectively filtering internet-scale hazard priors to only those with empirical impact.
• Sample efficiency: Kernel regression over semantic voxels delivers effective disturbance field modeling from few interaction samples, circumventing the data efficiency limitations of pure GP regression in low-sample regimes.
• Reduced over-conservatism: Event-driven querying and experience-driven library expansion alleviate the over-conservatism of proactive semantic methods—robots avoid what is actually hazardous, not merely what looks hazardous to generic perception models.

The approach is, however, limited by reliance on the visual detectability of hazard causes and current VLM attribution reliability. Generalization is currently limited when hazards cannot be semantically grounded in the observed modality or when scene clutter hinders attribution.

Future Directions

Open research directions include:

• Enhancement of multimodal grounding for adversity attribution, leveraging audio, touch, or low-level physical signals.
• End-to-end differentiable integration reducing brittleness at modular boundaries.
• Continual learning with evidence accumulation to improve robustness in cluttered and ambiguous scenes.
• Extension of semantic disturbance characterization to other domains (e.g., legged locomotion, manipulation-induced hazards).
• Joint optimization of hazard field learning and planning under epistemic uncertainty for high-assurance autonomy.

Conclusion

DFM2 presents a technically rigorous, empirically validated framework for continual, embodiment-specific hazard learning and adaptation in mobile robots. By reframing adversities as learnable, semantically grounded spatial fields, and leveraging event-driven narrative reasoning, DFM2 achieves superior robustness, efficiency, and generality across both environmental and perception-induced adversity domains. The experience-driven reasoning paradigm outlined here has considerable implications for deploying autonomous agents in open-world settings where predict-and-avoid approaches are fundamentally insufficient.