Assistron: Bayesian Shared Autonomy with Off-the-shelf Vision-Language-Action Models

Abstract: We propose Assistron, a shared autonomy model that leverages Vision-Language-Action (VLA) models to assist the user in daily activities. Our approach is grounded in two core principles: (1)~minimizing human cognitive and physical effort by leveraging VLA-driven autonomy for macro-movements, and (2)~prioritizing human intervention specifically at critical failure points. Driven by the user's verbal language commands, Assistron utilizes the VLA to autonomously execute macro-reaching trajectories, saving users' effort. In contact-rich interactions where VLAs tend to fail, Assistron employs a phase-aware interaction detection mechanism and solicits the user to intervene, in turn adjusting the VLA's action generation via flow matching guidance. Critically, our formulation eliminates the need for VLA fine-tuning, protecting its broad behavioral priors from catastrophic forgetting and ensuring the model does not become a narrow specialist. We validate our approach on a comprehensive multi-task scene recovery benchmark encompassing diverse daily manipulation skills. Empirical results demonstrate that Assistron significantly improves task success rates over pure autonomous baselines while significantly reducing human cognitive and physical workload compared to traditional teleoperation, offering a scalable, smooth, and effortless paradigm for assistive manipulation. The code is available in https://github.com/mousecpn/Assistron.git.

• The paper introduces a Bayesian shared autonomy framework that leverages off-the-shelf vision-language-action models without the need for retraining.
• It employs a ResNet-based interaction detector and posterior-guided policy blending to seamlessly switch between autonomous and shared control during critical contact phases.
• Experimental results show over 40% reduction in user effort and improved task completion compared to pure teleoperation and verbal VLA policies.

Assistron: Bayesian Shared Autonomy with Off-the-Shelf Vision-Language-Action Models

Introduction and Motivation

Assistron introduces a Bayesian shared autonomy framework that exploits frozen Vision-Language-Action (VLA) models as open-world semantic priors for assistive manipulation without requiring any model re-training or fine-tuning. The key observation underpinning the approach is that VLA-driven manipulation policies, while strong at high-level goal understanding and macro-movement synthesis, routinely underperform in contact-rich phases due to spatial imprecision and unreliable zero-shot generalization. Rather than re-engineering these models or risking catastrophic forgetting by fine-tuning, Assistron proposes a principled arbitration: VLA autonomy for semantic macro-trajectories and sparse, phase-aware human intervention only during critical failure points.

Figure 1: The Assistron architecture, featuring autonomous VLA policy, phase-aware intervention triggering, and shared autonomy via flow-matching guidance during contact-rich events.

Methodology

Arbitration via Interaction Detection

Assistron utilizes a ResNet-18-based, task-agnostic visual interaction detector to anticipate imminent contact phases by analyzing wrist-camera inputs for cues of spatial proximity and correlating these with the VLA’s intended gripper actuation. Automatic switching to shared control is triggered only when both visual and action-based intent align. The dual-verification logic yields high isolation of critical bottlenecks, minimizing unnecessary manual interventions.

Figure 2: Temporal profile of interaction probabilities and corresponding transitions between autonomous and shared/autonomous states for multi-step tasks.

Probabilistic Policy Blending: Posterior Inference over Latent Actions

During user intervention, Assistron fuses low-bandwidth joystick commands into the latent action manifold of the VLA using a policy blending framework grounded in posterior inference. The VLA policy acts as a strong semantic prior over the action trajectory $p(\bm{a}_1)$, and the user’s corrective command is incorporated as a Gaussian likelihood. Conditional action synthesis is accomplished using flow-matching guidance, which analytically injects user-correction gradients into the generative process, yielding a blending trajectory consistent with the VLA’s multimodal action distribution.

This approach, unlike standard linear or “diffusion assistance” blending, ensures semantic alignment and avoids incoherent or abrupt motions that emerge when latent dimension dependencies are broken.

Experimental Evaluation

Scene Recovery Benchmark and User Study

Assistron is evaluated on a multi-stage long-horizon scene recovery benchmark, encompassing tasks such as drawer opening, object insertion, pick-and-place, and articulated manipulation, reflecting the diversity of unstructured household environments. The baseline comparisons include:

• Direct Joystick Teleoperation (pure human control)
• Verbal VLA Policy (full autonomous execution based on verbal commands)

Assistron consistently matches or exceeds teleoperation performance in task completion while reducing cognitive and physical effort by over 40%, as measured by active control time and NASA-TLX workload indices. The VLA-only baseline, by contrast, achieves sub-15% partial task completion due to failures in contact precision.

Figure 3: Scene recovery results: (a) objective performance metrics—completion time and partial success rates, (b) user satisfaction, (c) NASA-TLX workload analysis.

Users benefit most from Assistron when their teleoperation proficiency is low, indicating the system's ability to act as an adaptive effort amplifier for less-experienced users.

Qualitative Demonstration and Control Mode Visualization

Assistron’s dynamic policy switching is illustrated across long-horizon tasks. The VLA autonomously executes macro-actions, while user interventions occur precisely at points of semantic ambiguity or contact uncertainty. Natural-language intent serves as both a high-level goal descriptor and as an override signal for misaligned VLA transitions.

Figure 4: Scene execution trace: timeline of control-state transitions, intervention keyframes, and natural language prompts aligned to task sub-goals.

Ablation: Flow-Matching vs. Linear Blending

A targeted ablation on flow-matching guidance demonstrates significant reductions in completion time and path length versus linear blending and pure teleoperation. The posterior approach resolves the multi-modality of latent VLA policies and maintains trajectory smoothness across all robot DoF, eliminating erratic motion spikes.

Structural Analysis: Limits of “Diffusion Assistance”

Direct blending of user input with latent action tokens (diffusion assistance) corrupts the structural coherence of VLA-generated action chunks, resulting in high-frequency velocity/acceleration jitter. In contrast, Assistron's guidance approach yields smoother, hardware-safe trajectories and robust transitions even with high-dimension action embeddings.

Performance of Interaction Detector

The ResNet-based detector, trained on automatically annotated gripper-contact events, achieves 81.2% accuracy and 84.5% AP, reliably flagging critical moments for user intervention across diverse visual contexts.

Limitations

Assistron’s framework is fundamentally limited by the expressiveness and behavioral coverage of the underlying frozen VLA. Severe failures in semantic grounding or high-level intent misalignment cannot be rectified by local correction. Furthermore, the posterior-based control blending implicitly assumes user corrections are in-distribution with VLA priors; out-of-distribution detection and fallback to pure teleoperation is an open area for robustness.

Implications and Outlook

Assistron defines a new paradigm for scalable shared autonomy in assistive manipulation via model-based arbitration and analytically optimal policy blending. It points toward future shared autonomy systems that integrate foundation policies maintained in a frozen state (to avoid overfitting) with adaptive, Bayesian user arbitration informed by phase-aware interaction detection. Extensions could incorporate skill-level user modeling, finer out-of-distribution detection, and generalized arbitration for multi-agent collaborative manipulation.

Conclusion

Assistron demonstrates that phase-aware, posterior-guided shared autonomy leveraging frozen VLA models enables assistive robotic manipulation at high success rates and minimal user effort, without retraining or sacrificing broad generalization. The system’s modularity and reliance on strong semantic priors indicate a scalable path forward for deploying large-scale, user-adaptive assistive robots in open-world settings (2606.23147).