AC-VLA: Robust Out-of-Distribution Action Execution via Compositional Learning

Abstract: Vision-Language-Action (VLA) models excel at end-to-end robotic manipulation but struggle with out-of-distribution (OOD) generalization when familiar sub-tasks are recombined in unseen configurations. We identify two mutually reinforcing failure modes: \emph{trajectory overfitting}, where models overfit to holistic trajectory patterns rather than compositional sub-skill semantics; and \emph{perceptual shortcut}, where action tokens over-rely on wrist-view textures at the expense of global spatial grounding. To address both, we introduce \textbf{AC-VLA}, a plug-and-play Action Compositional learning framework comprising two architecture-agnostic components: \textbf{(i)} a compositional learning module that uses an LLM-driven instruction decomposer and a proprioceptive trajectory aligner to generate dense sub-task supervision, followed by mixed training on complete demonstrations and decomposed data to endow the model with compositional generalization; and \textbf{(ii)} a state-conditioned asymmetric masking strategy that suppresses wrist-view inputs during closed-gripper phases, enforcing global semantic grounding. All components are architectural modification-free and directly integrable into any VLA backbone. Instantiated on $π_{0.5}$ and evaluated on LIBERO and LIBERO-OOD benchmarks, AC-VLA achieves a ~28% absolute improvement on compositional OOD tasks while maintaining near-perfect in-distribution performance.

• The paper introduces AC-VLA, which combines offline LLM–proprioception task decomposition, mixed-granularity training, and state-conditioned wrist-view masking to improve compositional OOD execution without architectural changes.
• AC-VLA achieves 64.2% Spatial OOD and 73.3% Goal OOD success on LIBERO with π0.5, compared with 35.5% and 46.6% for the baseline, while retaining 96.7% in-distribution performance.
• The method improves real-world OOD success on a 6-DoF robot from 35.0% to 82.5% by encouraging global spatial grounding during placement, though it depends on reliable decomposition and alignment.

AC-VLA addresses a specific and well-documented weakness of Vision-Language-Action (VLA) models: the inability to execute instructions that recombine familiar sub-skills in novel configurations. The authors diagnose two mutually reinforcing failure modes—trajectory overfitting, where the policy memorizes holistic motion sequences tied to seen task combinations, and perceptual shortcut, where action tokens over-rely on wrist-view textures at the expense of global spatial grounding. Their remedy is a training-time-only framework, instantiated without architectural modification on flow-matching VLA backbones, that yields large out-of-distribution (OOD) gains while preserving near-perfect in-distribution performance.

Diagnosis of compositional failure

The paper's motivating observation is that compositional OOD tasks create token-level conflicts between modalities. In training data where "grasp object A" always co-occurs with placement at target A, a novel instruction pairing object A with target B produces conflicting evidence: visual cues indicate object A while language specifies target B. Models trained on holistic demonstrations resolve this conflict by replaying memorized trajectory patterns rather than composing primitives.

The perceptual-shortcut claim is supported by an ablation on camera views using $\pi_{0.5}$ on LIBERO: removing the wrist camera collapses success rates to 1.9/1.5/2.9/0.1% across the Spatial/Goal/Object/Long suites, whereas removing the third-person view degrades performance less severely (52.2/77.7/68.9/22.2%). Attention visualizations from the DiT action head reinforce this: on OOD tasks, attention during the "place" phase remains anchored to the original target rather than the newly instructed destination, even when the "pick" phase is semantically correct. This is a strong empirical claim—that wrist-view textures function as an indispensable but spurious cue—and it directly motivates the masking design.

Compositional learning module

The first component converts raw demonstrations into dense sub-task supervision through two stages:

• Collaborative decomposition: An LLM (Qwen-3.5-Flash) segments each instruction into an ordered sequence of sub-task descriptions whose count $N$ is determined dynamically by semantic complexity. A proprioceptive aligner then synchronizes these descriptions with trajectory segments using naturally occurring physical cues: gripper-state transitions ($g_t \neq g_{t-1}$), which mark grasp/transport boundaries, and chunk-wise average end-effector displacement falling below a clustering-derived threshold $\epsilon$, which identifies approach-to-contact or placement events. Segmentation operates chunk-wise so that any chunk containing at least one boundary point yields a usable sub-task. This alignment is fully offline and requires no manual annotation, distinguishing it from fixed-length temporal partitioning used in prior stage-wise methods.
• Mixed training: Mini-batches are sampled from the full-demonstration set $\mathcal{D}_{\text{full}}$ and the decomposed set $\mathcal{D}_{\text{sub}}$ at a fixed ratio. Training only on decomposed data improves OOD performance but causes catastrophic in-distribution collapse; mixed training resolves this stability-plasticity tension by preserving long-horizon coherence alongside primitive-level supervision.

The framework is instantiated on $\pi_{0.5}$'s flow-matching objective: a VLM produces conditioning features $\bm{\phi}_t$, and a DiT action head $\mathbf{V}_\theta$ regresses the flow velocity over noised action chunks, with inference via $K$-step Euler integration.

State-conditioned asymmetric masking

The second component suppresses wrist-view tokens via attention masking during closed-gripper ("place") phases at training time, while leaving wrist inputs intact during open-gripper approach and grasping phases where fine-grained egocentric feedback is genuinely useful. Masking is applied online during training and requires no changes to architecture or inference. The asymmetry is deliberate: it severs the spurious correlation between wrist textures and task configuration precisely where that correlation misleads placement behavior, forcing reliance on third-view observations and language for spatial grounding.

Experimental results

On LIBERO and LIBERO-OOD (each OOD suite containing 20 novel compositional tasks, evaluated over 150 episodes per task), AC-VLA($N$0) achieves the following headline numbers:

Model	In-D avg	Spatial OOD	Goal OOD	Overall avg
OpenVLA-OFT	~97	0.0	1.0	64.9
UniVLA	~95	11.0	32.0	70.7
GR00T-N1	~96	17.9	24.1	71.1
$N$1	~97	35.5	46.6	78.6
Spatial Forcing-$N$2	~98	48.3	57.8	83.3
AC-VLA ($N$3)	96.7	64.2 (+28.7)	73.3 (+26.7)	87.3
AC-VLA (GR00T-N1)	~95.9	36.4 (+18.5)	44.0 (+19.9)	77.3

The ablation isolates each component's contribution: sub-task-only training raises OOD to 54.8/68.6 but drops in-distribution to 61.4; masking alone lifts OOD to 47.3/67.0 with negligible in-distribution cost; combined, they reach 64.2/73.3 at 96.7 in-distribution. The super-additive combination indicates the two failure modes are complementary rather than redundant. Qualitatively, masked models show more than a 100% increase in per-token third-person-view attention during placement, confirming the intended shift toward global grounding.

Real-world evaluation on a 6-DoF PIPER arm with ORBBEC DaBai DC1 wrist and RealSense D455 third-person cameras shows AC-VLA improving OOD success from 35.0% to 82.5% (+47.5 absolute) while in-distribution performance declines modestly from 93.7% to 88.7%, yielding an overall average gain of +21.2 points.

One caveat the authors note: $N$4-TLI reports higher OOD scores (85/81) by combining two in-distribution inference passes, but requires manually tuned interpolation hyperparameters and double inference; AC-VLA's comparison to it is therefore not strictly like-for-like.

Limitations

The paper concedes two limitations. First, decomposition quality depends jointly on LLM semantic parsing and proprioceptive alignment fidelity; ambiguous or domain-specific instructions can produce inaccurate sub-task descriptions that propagate errors into training. Second, the benefit of explicit compositional learning may diminish on extremely large and diverse datasets that already cover most object–target combinations, rendering the OOD gap negligible—an assumption about data coverage that bounds the method's practical relevance at scale.

Conclusion

AC-VLA demonstrates that compositional OOD generalization in VLA models can be substantially improved through supervision restructuring alone—offline LLM-plus-proprioception decomposition, mixed-granularity training, and state-conditioned view masking—without architectural change. The consistent gains across two backbones and both simulation and real-world settings support the paper's central claim that trajectory overfitting and perceptual shortcuts are distinct, addressable failure modes. Open questions include how the proprioceptive aligner scales beyond tabletop pick-and-place semantics and whether the masking heuristic generalizes to manipulation phases not cleanly indexed by gripper state.