Belief Dynamics for Detecting Behavioral Shifts in Safe Collaborative Manipulation

Abstract: Robots operating in shared workspaces must maintain safe coordination with other agents whose behavior may change during task execution. When a collaborating agent switches strategy mid-episode, continuing under outdated assumptions can lead to unsafe actions and increased collision risk. Reliable detection of such behavioral regime changes is therefore critical. We study regime-switch detection under controlled non-stationarity in ManiSkill shared-workspace manipulation tasks. Across ten detection methods and five random seeds, enabling detection reduces post-switch collisions by 52%. However, average performance hides significant reliability differences: under a realistic tolerance of +-3 steps, detection ranges from 86% to 30%, while under +-5 steps all methods achieve 100%. We introduce UA-TOM, a lightweight belief-tracking module that augments frozen vision-language-action (VLA) control backbones using selective state-space dynamics, causal attention, and prediction-error signals. Across five seeds and 1200 episodes, UA-TOM achieves the highest detection rate among unassisted methods (85.7% at +-3) and the lowest close-range time (4.8 steps), outperforming an Oracle (5.3 steps). Analysis shows hidden-state update magnitude increases by 17x at regime switches and decays over roughly 10 timesteps, while the discretization step converges to a near-constant value (Delta_t approx 0.78), indicating sensitivity driven by learned dynamics rather than input-dependent gating. Cross-domain experiments in Overcooked show complementary roles of causal attention and prediction-error signals. UA-TOM introduces 7.4 ms inference overhead (14.8% of a 50 ms control budget), enabling reliable regime-switch detection without modifying the base policy.

• The paper introduces UA-ToM, a modular belief-tracking detector that integrates selective state-space dynamics, causal attention, prediction-error signals, and memory-based prototyping for regime-switch detection.
• The paper demonstrates that UA-ToM reduces post-switch collision rates by 52% and achieves smoother belief revisions compared to baseline methods in both continuous and discrete coordination tasks.
• The paper highlights that hybrid architectural design with adaptive evidence fusion directly improves safety outcomes by balancing rapid detection with smooth state transitions.

Belief Dynamics for Detecting Behavioral Shifts in Safe Collaborative Manipulation: An Expert Analysis

Problem Setting and Motivation

Reliable detection of dynamic behavioral regime shifts is essential for robotic manipulators operating collaboratively in shared workspaces where co-agents—be they human or robotic—may adapt strategies during ongoing interaction. Failing to rapidly update beliefs about an agent whose behavior switches mid-episode can substantially elevate collision risk due to outdated interaction assumptions. The paper addresses this problem by framing regime-switch detection as a closed-loop, low-latency inference challenge under strict operational tolerances (±3 timesteps, 150 ms) and demonstrates that naive random seed averaging of mean performance metrics can conceal significant, deployment-relevant reliability failures.

UA-ToM: Design and Architecture

The centerpiece of the work is UA-ToM, a modular, belief-tracking detector designed to augment frozen large-scale vision-language-action (VLA) backbones without direct policy modification. UA-ToM integrates four pathways:

1. Selective State-Space Model (SSM) Dynamics: Maintains persistent latent beliefs using learnable dynamics, with a discretization parameter $\Delta_t$ that stabilizes update cadence (converging empirically to $\sim$0.78).
2. Causal Attention: Anchors beliefs to temporally recent observations for resilience to long-term drift, critical in discrete coordination tasks.
3. Prediction-Error Signal: Monitors action mismatches as explicit indicators of behavioral change, contributing direct evidence of regime switches—in particular, spikes in prediction error act as perturbation triggers for belief revision.
4. Contrastive Memory-Based Prototyping: Maintains type-specific running prototypes, facilitating categorical discrimination of partner regimes.

These components are joined via a gated evidence fusion layer producing type estimates, switch probabilities, and co-agent action predictions.

Figure 1: UA-ToM computation flow, highlighting SSM-based belief dynamics, causal attention, prediction error integration, and prototype-augmented outputs.

Experimental Evaluation: Protocols, Metrics, and Baselines

Evaluation is conducted in the ManiSkill shared-workspace manipulation platform (two 7-DOF Franka Panda arms, four distinct co-agent regimes: Helper, Competitor, Blocker, Passive), and the Overcooked coordination benchmark for cross-domain validation. The central metrics are:

• Hard Detection Rate within strict tolerance windows (±3, ±5 steps)
• Detection Latency (delay from regime switch to detection)
• Post-Switch Collisions (per-episode collision rate following switches)
• Close-Range Time (CRT): Timesteps spent dangerously close (<15 cm) to the other agent after regime shift
• Reliability Ratio (R): Minimum/maximum seed-based detection rate
• Coefficient of Variation (CV) across seeds
• Representation Separability (Bhattacharyya distance, silhouette score)
• Inference Overhead

Baselines include classical changepoint detectors (BOCPD), sequence models (GRU, Transformer, Mamba), agent-modeling methods (LIAM, BToM), and privileged oracles (ground-truth signal, type labels).

Figure 2: Enabling regime-switch detection reduces post-switch collision rate by 52% across all methods, establishing the operational relevance of rapid behavioral change detection.

Results

Collision Risk Reduction and Detection Reliability

Across 1200 episodes and five random seeds, regime-switch detection reduces mean post-switch collision rate by 52% (from 2.34 to roughly 1.1 per episode). However, standard evaluation at ±5 steps masks considerable disparities: all methods report 100% detection, while a stricter ±3 window reveals a three-tier structure (fast: >82%, intermediate: 58–62%, slow: <34%).

Figure 3: Detection reliability heatmap under operational tolerance highlights performance tiers and inter-transition variability, unobservable at conventional evaluation thresholds.

UA-ToM leads unassisted detectors with an 85.7% mean hard detection rate and the highest reliability ratio ($R = 0.93$). Notably, privileged baselines ('Context-Conditioned') achieve marginally better rates but rely on explicit type labels.

Adaptation Quality and Safety Beyond Detection

Stronger claims emerge in extended, longer-horizon episodes. All detectors achieve high switch-detection rates (>95%), but qualitative differences surface in CRT, which reflects sustained post-switch safety. UA-ToM (ManiSkill variant) yields the lowest CRT (4.8 steps), outperforming the Oracle (5.3 steps) and privileged baselines. This counterintuitive result is attributed to UA-ToM's smooth belief revision via SSM dynamics: immediate, hard switches (Oracle) induce trajectory discontinuities, momentarily increasing risk.

Figure 4: UA-ToM (MS) achieves the lowest close-range time (CRT), even outperforming perfect Oracle switch detection due to smoother trajectory adaptation.

Figure 5: Post-switch collision rate over time shows UA-ToM delivers a monotonic risk decline, contrasting with sharp transients under Oracle and persistently elevated risk for LIAM due to delayed or discrete adaptation.

Notably, detection rate alone is not predictive of safety outcomes: LIAM matches UA-ToM on detection rate (97.3%) but accumulates nearly 2.8× more CRT, confirming that belief revision mechanism and adaptation smoothness dominate post-adaptation safety.

Mechanistic Insights: Dynamics and Representational Analysis

Analysis of internal dynamics reveals that SSM discretization, $\Delta_t$, remains stable, while regime switches trigger a 17× spike in hidden-state update magnitude, decaying over ten steps. The principal source of this spike is a 2.3× increase in prediction error, validating the architectural emphasis on the prediction error pathway.

Figure 6: SSM hidden-state updates spike sharply at the switch, with near-constant discretization, corroborating that detection sensitivity is primarily carried by model dynamics as opposed to input-dependent gating.

Representation analysis further shows that while classical detectors (BOCPD) display strong type separation in embedding space, architectural factors preclude effective closed-loop regime localization, yielding 0% switch F1. UA-ToM's contrastive memory achieves the highest silhouette score (0.52), confirming robust categorical evidence accumulation.

Cross-Domain Robustness and Ablation

UA-ToM consistently delivers the highest detection floor and lowest variability on Overcooked; the architectural component most critical to performance is domain-dependent: hierarchical prediction error dominates in continuous manipulation (ManiSkill), whereas causal attention is paramount for discrete multi-agent coordination (Overcooked).

Ablation underscores that modular detector design enables resilience; removing key components causes drastic (up to -78%) performance degradation depending on domain, supporting the case for adaptable, hybridized mechanism selection.

Practical and Theoretical Implications

Practically, integrating UA-ToM as a wrapper around legacy or frozen VLA policy backbones enables rapid, robust deployment of safety-enhancing behavior change detection, incurring minimal inference overhead (7.4 ms per step, ~14.8% of a 50 ms budget). Reliable operation is attainable without requiring access to co-agent internals or ground-truth labels.

Theoretically, the results argue against conflating detection rate with practical safety—adaptive behavior transition is a function of both event detection latency and the smoothness of downstream belief updates. The explicit role of model dynamics, rather than heuristic gating, highlights the need for structured designs that admit temporal evidence integration. Cross-domain results motivate further research into adaptive evidence-weighting frameworks for generalized regime-switch detection.

Conclusion

This work rigorously demonstrates that in collaborative manipulation, reliable in-loop regime-switch detection critically reduces collision risk, and that current VLA control models require explicit belief-tracking augmentation for operational reliability. UA-ToM sets a new standard for unassisted switch detection, achieving both high detection accuracy and, more notably, the lowest post-switch danger time through smooth state evolution. The mechanistic findings endorse selective SSM dynamics and hybrid evidence pathways as key design drivers. Future research should extend modular belief trackers to continuous, human-in-the-loop settings and vision-based state estimation, moving towards universally robust multi-agent adaptation and safety.