Drop-Then-Recovery: How Redundant Are Vision-Language-Action Models?

Abstract: Vision-Language-Action (VLA) models enable instruction-driven robotic manipulation, but they inherit oversized language backbones from pretrained VLMs whose capacity far exceeds what is needed for short robotic instructions. This raises a basic question: how much of a VLA model is actually necessary for closed-loop control? In this work, we study architectural redundancy in VLA models by using transformer block removal as a controlled intervention. We introduce \textbf{Drop-Then-Recovery (DTR)}, an analysis protocol that removes selected blocks from a pretrained VLA model and then fine-tunes the resulting model to measure whether the removed capacity was necessary for downstream control. To make this intervention reliable, we propose \textbf{GateProbe}, a one-shot virtual-gate sensitivity metric that ranks blocks by their contribution to the downstream action loss. Across multiple VLA architectures, manipulation benchmarks and even real-robot industrial scenarios, we find a strong asymmetry in post-removal recoverability: \ul{\textit{language backbones are highly redundant for standard robotic manipulation tasks, whereas vision and action pathways are substantially less tolerant to removal}}. On LIBERO, removing half of the LLM blocks even improves OpenVLA-OFT from 95.0% to 98.3% under the same downstream fine-tuning budget, and retaining only two language blocks still recovers baseline-level performance. These results suggest that current VLA benchmarks may exert limited pressure on deep language grounding and compositional instruction understanding, and that future VLA architectures should allocate capacity more deliberately across language, vision, and action components. The code is available at https://github.com/s1ghhh/VLADrop.

• The paper demonstrates that aggressively dropping redundant language transformer blocks via the DTR protocol can recover or even improve baseline manipulation performance.
• It introduces GateProbe, a gradient-based sensitivity metric that effectively identifies non-critical blocks for targeted pruning in VLA models.
• The study shows that reducing language capacity enhances throughput and efficiency, though excessive drops may compromise robustness under challenging conditions.

Architectural Redundancy in Vision-Language-Action Models: An Analysis via Drop-Then-Recovery

Introduction and Motivation

Vision-Language-Action (VLA) models have emerged as the dominant paradigm for instruction-driven robotic manipulation, combining pretrained vision-language foundations with specialized action modules to predict robotic control signals from multimodal inputs. These architectures typically rely on inherited large-scale language and vision backbones, originally optimized for web-scale understanding tasks, but their actual necessity for closed-loop robotic control—where instructions are often templated and linguistically simple—remains insufficiently scrutinized. The core question is: to what extent is the model capacity, especially in the language backbone, redundant for standard manipulation tasks?

Drop-Then-Recovery (DTR): Methodological Overview

To systematically interrogate VLA redundancy, the paper introduces the Drop-Then-Recovery (DTR) protocol, which consists of a two-stage process. In the "drop" phase, selected transformer blocks are physically excised from a pretrained VLA model; in the "recovery" phase, the resulting pruned model is fine-tuned exclusively on downstream control, evaluating whether excised capacity is essential for task success.

Block selection is a critical step. The authors propose GateProbe, a one-shot, gradient-based virtual-gate sensitivity metric operating in activation space. GateProbe quantifies a block’s recoverability by estimating the gradient sensitivity of the task loss to an explicit gating of each residual branch, thereby prioritizing removal of blocks with minimal expected impact on closed-loop performance.

Figure 1: Overview of DTR. A pretrained VLA model's transformer blocks are ranked by importance, the least important are physically removed, and the smaller model is recovery fine-tuned.

Empirical Insights on Redundancy

Asymmetry Across Modalities

A central empirical finding is the striking asymmetry in recoverability between model components. Language backbones, when inherited from large vision-LLMs, are highly redundant for standard manipulation datasets; vision and action components are not. On the LIBERO benchmark, removing half of the language transformer blocks from OpenVLA-OFT actually increases performance from 95.0% to 98.3% after recovery fine-tuning, while equivalent interventions on vision and action pathways result in sharp degradation. In extremis, retaining only 2 language blocks recovers baseline performance (OpenVLA-OFT 95.1%).

Controlled Drop Granularity

Whole-block dropping consistently proves superior to dropping only sublayers (MHA/MLP), providing maximal compression while preserving or exceeding task success rates. Metrics that rely solely on activation similarity (cosine) or magnitude underperform. GateProbe outperforms static and other gradient-based metrics, particularly under aggressive compression, where it identifies recoverable minimal block sets.

Real-World Deployment and Robustness

VLA redundancy, as revealed through DTR, directly transfers to real-world warehouse picking with minimal loss in success rate provided deployment conditions closely match training distributions. Dropped models (with >50% language blocks excised) perform on par with full models when tested on standard environments.

Figure 2: Real-world experimental setup for warehouse sorting and main results, showing Drop-9 (50% dropped) equals or surpasses full model performance.

However, robustness analyses under visual (e.g., lighting changes) and physical distribution perturbations show significant performance drops for highly compressed models. For instance, under green lighting, the most aggressively dropped model achieved only 35% success versus 50% for the full model. Thus excess language capacity, while redundant for in-distribution success, does contribute marginally to robustness under heavy OOD shift.

Figure 3: Robustness under distribution shift; (a) performance drop under lighting perturbations, (b) under physical perturbations.

Cross-Benchmark and Cross-Architecture Generalization

Evaluation on LIBERO-Plus (with increased perturbations and linguistic diversity) and RoboTwin 2.0 (dual-arm, highly randomized tasks) confirm the universality—but not the absoluteness—of language-block redundancy. The degradation grows with increased physical and linguistic variation, especially on 'Hard' task variants or with significant scene perturbations.

Figure 4: Per-task DTR results on RoboTwin 2.0 for $\pi_{0.5}$ show that while Easy variants are robust to language block removal, Hard variants incur marked performance losses.

The DTR protocol was validated on several high-profile VLA model families (OpenVLA-OFT, $\pi_{0.5}$, Lingbot-VLA, GigaBrain-0), consistently yielding comparable or better performance after severe language backbone reductions. This underscores that redundancy is not an artifact of any specific architecture or scale but stems from the design practice of uncritically inheriting high-capacity LLMs into VLA pipelines.

Practical Implications and Acceleration

Removing redundant language blocks provides direct and material efficiency benefits: accelerated throughput in fine-tuning (per step and per compute budget), significant wall-clock speedup at inference due to reduced model depth, and substantial memory savings—all achieved without reliance on hardware-specific quantized or sparse kernels.

The pruned models remain standard dense networks, ensuring hardware-agnostic acceleration. Unlike quantization or sparse pruning—both of which require runtime support and may lack efficiency on robotic edge hardware—DTR results in reduced latency on any device, including Jetson-class edge platforms.

Theoretical and Benchmarking Implications

The results highlight a profound capacity-task mismatch between inherited language foundation models and the requirements of closed-loop robotic control. Current benchmarks lack the linguistic complexity and compositional depth required to leverage the full representational range of LLMs. As such, VLA paradigm development should transition towards:

• More capacity-balanced architectural designs, allocating compute and parameter budgets more judiciously across vision, language, and action.
• Development of benchmarks with longer-horizon, more linguistically compositional, and physically diverse task specifications to exert genuine pressure on language grounding and generalization.

Conclusion

Drop-Then-Recovery, coupled with GateProbe block ranking, provides a powerful, methodology-agnostic framework for probing and exploiting redundancy in VLA models. The demonstrated overprovisioning of language backbones in the context of standard robotic manipulation suggests an urgent need for both architectural recalibration and more challenging benchmarks. For practitioners and theorists alike, these findings motivate a reassessment of scale and modularity in VLA design, with an emphasis on matching model capacity to task demands and real-world deployment constraints.