Vision-Action Adapter (VA-Adapter) Overview

• Vision-Action Adapters are modular mechanisms that connect perceptual representations to actionable outputs without redesigning the backbone.
• They encompass diverse architectures including system-level, residual action-head, and semantic-to-dynamics modules used in robotics, video processing, and medical imaging.
• This pattern enhances efficiency and robustness, enabling parameter-efficient adaptation and effective real-time control in complex multimodal systems.

Searching arXiv for papers on Vision-Action Adapters and closely related adapter-based VLA methods. A Vision-Action Adapter (VA-Adapter) denotes a modular mechanism that connects perceptual or multimodal representations to executable actions without requiring wholesale redesign of the underlying backbone. Across recent arXiv literature, the term spans several related but distinct constructions: a system-level adaptor inserted between a Vision-Language-Action (VLA) policy and a low-level controller, as in CompliantVLA-adaptor (Zhang et al., 21 Jan 2026); a lightweight delay-aware residual branch inside an action head, as in Action ControlNet (Guo et al., 24 Jun 2026); a semantic-to-control token compiler between a frozen vision-LLM and a fast controller, as in the Pons Adapter of SaiVLA-0 (Shi et al., 9 Mar 2026); and a parameter-efficient sequence module inserted into a frozen medical image encoder for probe guidance, explicitly named “VA-Adapter” (Wang et al., 8 Oct 2025). In all of these cases, the common function is to adapt how perceptual information becomes action, rather than to replace the backbone that produces perceptual or semantic features.

1. Conceptual definition and scope

The most general formulation treats a VA-Adapter as the action-side interface in a model of the form

$$\mathbf{a}_t = H\big(F(\mathbf{x}\,|\,\theta)\big),$$

where $F$ is a vision-language or multimodal backbone and $H$ converts its latent representation into trajectories, controls, or action tokens (Hu et al., 18 Dec 2025). In this sense, the adapter is the bridge between representation space and control space.

Recent work uses the term in at least three technically distinct senses. First, there is the system-level adapter, exemplified by CompliantVLA-adaptor, which leaves the VLA policy unchanged and instead modulates execution via variable impedance control conditioned on visual, linguistic, and force feedback (Zhang et al., 21 Jan 2026). Second, there is the policy-head or action-head adapter, exemplified by Action ControlNet, which augments a mostly frozen action generator with a residual branch conditioned on executed motion suffixes so that asynchronous chunk handoffs remain smooth under inference delay (Guo et al., 24 Jun 2026). Third, there is the representation adapter, exemplified by the Pons Adapter in SaiVLA-0, which compresses frozen multi-layer VLM states into a small set of context tokens usable by a fast downstream controller (Shi et al., 9 Mar 2026).

The term also appears outside robot manipulation. In echocardiography probe guidance, VA-Adapter is a lightweight module inserted into the deep layers of a frozen ultrasound foundation model so that the encoder can represent vision-action sequences and predict 6D probe adjustments toward standard cardiac planes (Wang et al., 8 Oct 2025). In few-shot video action recognition, D$^2$ST-Adapter and related video adapters do not output robot controls, but they instantiate the same adapter principle: a frozen image backbone is augmented with lightweight spatio-temporal modules that introduce task-specific inductive biases while preserving the backbone (Pei et al., 2023). This suggests that “VA-Adapter” has become a broader architectural pattern rather than a single method.

A recurring distinction in the literature is between VA-Adapters as external or residual control interfaces and adapters in the classical parameter-efficient fine-tuning sense. CompliantVLA-adaptor explicitly notes that its adaptor is not a small learned layer inside the VLA, but a system-level bridge from semantic reasoning to classical control (Zhang et al., 21 Jan 2026). By contrast, medical VA-Adapter, IB-Adapter, and CLARE all instantiate trainable modules inserted inside or adjacent to frozen backbones (Wang et al., 8 Oct 2025, Fu et al., 18 May 2026, Römer et al., 14 Jan 2026).

2. Architectural patterns

A first architectural pattern is execution-side modulation without modifying the high-level policy. CompliantVLA-adaptor treats baseline VLAs such as RDT, Pi0, and OpenVLA-oft as position-only policies

$$\pi_\text{VLA} : \mathcal{S} \times \mathcal{T} \rightarrow \mathcal{A},$$

and inserts an adaptor that maps visual, linguistic, and force context to translational stiffness and damping

$$\text{CompliantVLA}: \text{VLM}(\mathcal{S} \times \mathcal{T} \times \mathcal{F}) \rightarrow (\mathcal{K}, \mathcal{D}).$$

The adaptor therefore changes execution gains and safety behavior rather than the nominal motion $\mathbf{x}_d$ produced by the VLA (Zhang et al., 21 Jan 2026).

A second pattern is residual conditioning inside the action head. Action ControlNet decomposes a chunked VLA policy as

$$\mathcal{M}_\theta(o_t,l) = \mathcal{A}_\psi\big(\mathcal{B}_\omega(o_t,l)\big),$$

freezes the backbone $\mathcal{B}_\omega$ and most of the action expert $\mathcal{A}_\psi$, and adds a delay-action encoder $F$0 plus projection layers $F$1. The resulting adapter injects a residual into selected hidden states: $F$2 where $F$3 summarizes the executed motion suffix during inference delay (Guo et al., 24 Jun 2026). Architecturally, this is close to ControlNet-style conditioning, but applied to action generation rather than image synthesis.

A third pattern is semantic compression into control-ready tokens. SaiVLA-0 separates a frozen Cerebrum, a trainable Pons Adapter, and a trainable Cerebellum. The Pons receives multi-layer hidden states $F$4, projects them to a common dimension, concatenates them,

$F$5

and pools them into a small set of context tokens

$F$6

These tokens are then concatenated with real-time vision, language, proprioception, and action queries for downstream control (Shi et al., 9 Mar 2026). The adapter thereby functions as a semantic-to-dynamics compiler.

A fourth pattern is sequence-aware encoder insertion. In echocardiography, VA-Adapter is inserted into the latter half of a CNN encoder or twice inside each deep Transformer block of a Transformer encoder. At depth $F$7, it projects visual features and relative probe motions into a bottleneck space, interleaves image and action tokens, processes them with a Transformer interaction module $F$8, and fuses the updated visual token back through an up-projection: $F$9 This design makes the frozen encoder sensitive to how ultrasound appearance evolves under probe motion (Wang et al., 8 Oct 2025).

A fifth pattern is lightweight vision-to-policy projection with robustness control. StableVLA replaces a simple projector with a Fused IB-Adapter,

$H$0

where the IB-Adapter computes channel covariance, applies sigmoid-based gating, and reconstructs a filtered latent before the VLA policy consumes it (Fu et al., 18 May 2026). This suggests a family of VA-Adapters whose primary role is not semantic enrichment or dynamics integration, but robustness filtering at the interface between visual features and action generation.

3. Conditioning signals and representational interfaces

The conditioning variables used by VA-Adapters differ substantially across domains, but several recurring interface patterns are evident. One is layerwise access to multimodal latents rather than reliance on the final layer only. VLA-Adapter explicitly distinguishes Raw latent $H$1 and ActionQuery latent $H$2 from every backbone layer, and reports that all-layer Raw yields 90.6% success rate on LIBERO-Long, all-layer ActionQuery yields 92.6%, and combining all-layer Raw + all-layer AQ yields 95.0% (Wang et al., 11 Sep 2025). The same paper further reports that last-layer Raw only gives 85.8%, intermediate Raw only gives 88.4%, and last-layer ActionQuery only gives 90.2% (Wang et al., 11 Sep 2025). This establishes a concrete empirical case for layerwise adapter conditioning.

Another interface pattern is compact task conditioning. CT-VAM formalizes a two-stage separation between language grounding and grounded visuomotor execution. A compact task condition $H$3, implemented in experiments as a one-hot task identifier $H$4, is embedded as

$H$5

and used as a single-token task stream within TARS, the stream-separated conditional attention decoder (Li et al., 8 Jun 2026). The theoretical condition

$H$6

is used to motivate the idea that once a control-sufficient intent has been formed, raw language need not remain in the high-frequency loop (Li et al., 8 Jun 2026). This suggests a design in which a VA-Adapter consumes compact intent variables from an upstream VLM while handling local visuomotor execution on edge hardware.

A third pattern is stateful or history-dependent conditioning. Action ControlNet conditions on an executed suffix

$H$7

pads it with learnable tokens, encodes it with a lightweight transformer, and injects the resulting motion-suffix embedding into the action head (Guo et al., 24 Jun 2026). In tactile-force alignment, the TaF-Adapter uses a window of $H$8 tactile observations and a causal Transformer to produce a history-dependent latent $H$9 aligned with force codes, then plugs this latent into a VLA policy (Huang et al., 28 Jan 2026). Although TaF-Adapter is not named a VA-Adapter, the paper explicitly argues that the same pattern can be used for vision or combined vision+tactile inputs (Huang et al., 28 Jan 2026).

A fourth pattern is explicit stream separation. CT-VAM’s TARS defines separate streams for current action tokens, learnable action queries, dense visual patches plus proprio history, and a single task token: $^2$0 Per-stream softmax normalization and gated aggregation are used to prevent the dense visual stream from overwhelming compact task-relevant conditions (Li et al., 8 Jun 2026). This is a direct architectural answer to the token-count imbalance that would arise under a shared softmax over all streams.

A fifth representational pattern is token or feature compression for modular reuse. SaiVLA-0 compresses frozen VLM states into $^2$1 context tokens $^2$2, while CLARE inserts residual bottleneck adapters into selected feedforward or projection layers and routes among them using discriminator reconstruction error (Shi et al., 9 Mar 2026, Römer et al., 14 Jan 2026). In both cases, the adapter acts as a structured interface between a large shared representation and a smaller, task-conditioned action pathway.

4. Learning regimes and optimization strategies

VA-Adapters are trained under several different regimes, ranging from pure prompting to fully supervised imitation. CompliantVLA-adaptor is notable because its VLM-based mapping is not trained at all. The VLM is frozen, prompts encode contact-phase recognition and impedance heuristics, and the adapter relies on a force-regularized safety layer plus classical variable impedance control. The paper nevertheless logs and releases a “force-torque-impedance-scenario context” dataset, explicitly suggesting that a future learned mapping

$^2$3

could later be trained from those logs (Zhang et al., 21 Jan 2026).

A dominant regime is supervised behavior cloning or action prediction with the backbone largely frozen. In echocardiography, only VA-Adapter, the GRU sequence encoder, and ten plane-specific heads are trainable; the vision encoder remains frozen. Ground-truth supervision is the 6D relative motion

$^2$4

and each plane-specific head is optimized with Smooth L1 loss on translation and rotation components (Wang et al., 8 Oct 2025). The default training setup uses sequence length $^2$5, bottleneck dimension $^2$6, Adam, batch size 256, initial learning rate $^2$7, cosine decay to $^2$8, and 5 epochs on 4 $^2$9 NVIDIA A100 GPUs (Wang et al., 8 Oct 2025).

Another common regime is generative action modeling with diffusion or flow matching. Action ControlNet trains only about 20% of total parameters on Kinetix, leaving the backbone fully preserved and the main action expert mostly frozen, while optimizing a delayed-control flow-matching loss conditioned on the encoded executed suffix (Guo et al., 24 Jun 2026). CT-VAM uses rectified flow over flattened action chunks $$\pi_\text{VLA} : \mathcal{S} \times \mathcal{T} \rightarrow \mathcal{A},$$0, constructs

$$\pi_\text{VLA} : \mathcal{S} \times \mathcal{T} \rightarrow \mathcal{A},$$1

uses target velocity $$\pi_\text{VLA} : \mathcal{S} \times \mathcal{T} \rightarrow \mathcal{A},$$2, and minimizes an MSE loss between predicted and target velocity fields (Li et al., 8 Jun 2026). TaF-VLA similarly plugs a frozen TaF-Adapter into a flow-matching policy and optimizes

$$\pi_\text{VLA} : \mathcal{S} \times \mathcal{T} \rightarrow \mathcal{A},$$3

for contact-rich manipulation (Huang et al., 28 Jan 2026).

A further regime is parameter-efficient adaptation and routing under continual learning. CLARE freezes the base VLA and previously added modules, inserts residual bottleneck adapters of the form

$$\pi_\text{VLA} : \mathcal{S} \times \mathcal{T} \rightarrow \mathcal{A},$$4

and routes among them with autoencoder discriminators using reconstruction error

$$\pi_\text{VLA} : \mathcal{S} \times \mathcal{T} \rightarrow \mathcal{A},$$5

Expansion is triggered only when layerwise feature novelty exceeds a threshold $$\pi_\text{VLA} : \mathcal{S} \times \mathcal{T} \rightarrow \mathcal{A},$$6, enabling exemplar-free continual learning without task identifiers at inference (Römer et al., 14 Jan 2026).

StableVLA introduces a distinct optimization philosophy: rather than adding an explicit information bottleneck regularizer to the loss, it bakes the bottleneck into the architecture. The IB-Adapter splits channels into heads, computes a Gram matrix $$\pi_\text{VLA} : \mathcal{S} \times \mathcal{T} \rightarrow \mathcal{A},$$7, applies sigmoid gating $$\pi_\text{VLA} : \mathcal{S} \times \mathcal{T} \rightarrow \mathcal{A},$$8, and reconstructs a filtered latent $$\pi_\text{VLA} : \mathcal{S} \times \mathcal{T} \rightarrow \mathcal{A},$$9 (Fu et al., 18 May 2026). The architecture is derived from variational information bottleneck arguments, but the policy is still trained with the standard robotic fine-tuning objective rather than an explicit mutual-information term.

5. Empirical evidence across domains

The empirical case for VA-Adapters is strongest in settings where the backbone is small, frozen, stale, delayed, or otherwise mismatched to deployment. In robot manipulation, VLA-Adapter reports LIBERO success rates of 97.8% on Spatial, 99.2% on Object, 97.2% on Goal, and 95.0% on Long, for an average of 97.3%, using a 0.5B Prismatic VLM based on Qwen2.5-0.5B without robotic pre-training (Wang et al., 11 Sep 2025). On LIBERO-Long, replacing OFT-style bridging with VLA-Adapter bridging raises success from 85.8% to 95.0% for the 0.5B Qwen2.5 backbone, from 87.5% to 95.2% for a 7B LLaMA2 backbone, and from 94.5% to 95.4% for OpenVLA-7B (Wang et al., 11 Sep 2025). The same paper also reports throughput of 219.2 Hz and latency of 0.0365 s on LIBERO-Long, compared with 71.4 Hz and 0.1120 s for full OpenVLA-OFT, and 4.2 Hz with 0.2396 s latency for original OpenVLA (Wang et al., 11 Sep 2025).

Delay-aware adapters show a different empirical profile. Action ControlNet improves average delayed success on Kinetix from 0.61 for direct stitching to 0.79, close to 0.80 for full Training-RTC, while retraining only about 20% of total parameters (Guo et al., 24 Jun 2026). On Meta-World MT50 it reaches average success 0.74 with latency 91 ms and 11.0 Hz, matching Training-RTC’s 0.74 but at lower latency than 134 ms and higher frequency than 7.46 Hz (Guo et al., 24 Jun 2026). On the real-world SO-ARM101 platform, direct stitching achieves 17/20 success, whereas ACNet reaches 20/20, with visibly smoother contact during cleaning (Guo et al., 24 Jun 2026).

Execution-side safety adapters show gains in physically interactive manipulation. CompliantVLA-adaptor evaluates eight contact-rich tasks from LIBERO and ManiSkill under a 30 N safety threshold with three consecutive violations terminating a trial. The average success rate across all tasks rises from $$\text{CompliantVLA}: \text{VLM}(\mathcal{S} \times \mathcal{T} \times \mathcal{F}) \rightarrow (\mathcal{K}, \mathcal{D}).$$0 for the baselines to $$\text{CompliantVLA}: \text{VLM}(\mathcal{S} \times \mathcal{T} \times \mathcal{F}) \rightarrow (\mathcal{K}, \mathcal{D}).$$1 with the adaptor, with maximum success increasing from 54% to 76%, and consistent improvement on 7/8 tasks (Zhang et al., 21 Jan 2026). The paper further reports that even when the VLA misplans in real hardware, the adaptor still reduces unsafe forces, functioning as a safety layer (Zhang et al., 21 Jan 2026).

Compact execution policies conditioned by lightweight task tokens provide another empirical line. CT-VAM, with 68M parameters including a frozen DINOv3-S+ backbone, achieves 82.1% average success on LIBERO, compared with 72.4% for Diffusion Policy, 76.1% for MDT, and 86.0% for $$\text{CompliantVLA}: \text{VLM}(\mathcal{S} \times \mathcal{T} \times \mathcal{F}) \rightarrow (\mathcal{K}, \mathcal{D}).$$2 at 3.3B parameters (Li et al., 8 Jun 2026). On the Ball Pouring real-world task, CT-VAM with FCI reaches 95% success on RTX 4080 in 6.41 s with 56 ms inference, and 90% success on Jetson Orin NX in 7.23 s with 200.6 ms inference, showing how chunked execution plus overlap can hide most of the latency (Li et al., 8 Jun 2026).

In robustness-oriented projection adapters, StableVLA reports that under LIBERO corruptions the 0.5B model with IB-Adapter substantially narrows or closes the gap to much larger models. For example, on LIBERO-Object at severity 5, VLA-Adapter reaches 29.3 while StableVLA reaches 70.2; on LIBERO-Long at severity 5, VLA-Adapter reaches 26.2 while StableVLA reaches 45.3 (Fu et al., 18 May 2026). The paper states that IB-Adapter consistently improves over the baseline by an average of 30% while adding fewer than 10M parameters, and that even without Open X-Embodiment pre-training the 0.5B model achieves robustness competitive with 7B-scale VLAs (Fu et al., 18 May 2026).

Beyond robotics, the medical VA-Adapter reports average translation MAE reductions from 8.21 mm for single-frame EchoCLIP to 5.40 mm for EchoCLIP+VA, from 8.26 mm for single-frame USFM to 5.35 mm for USFM+VA, and average rotation MAE reductions from 8.52° to 6.74° for EchoCLIP and from 8.62° to 6.71° for USFM (Wang et al., 8 Oct 2025). It does so while training only 2.61M parameters for EchoCLIP+VA and 3.97M or 3.94M parameters for USFM+VA and BiomedCLIP+VA, corresponding to 95.4–97.0% fewer trainable parameters than full tuning (Wang et al., 8 Oct 2025). These results show that the VA-Adapter abstraction is not confined to language-conditioned robot control.

6. Relation to adjacent adapter paradigms, limitations, and open questions

VA-Adapters sit at the intersection of several adapter traditions. ViT-Adapter shows how a plain ViT can be retrofitted for dense prediction by adding a pre-training-free adapter that injects spatial priors and produces a multi-scale feature pyramid, without changing the backbone architecture (Chen et al., 2022). D$$\text{CompliantVLA}: \text{VLM}(\mathcal{S} \times \mathcal{T} \times \mathcal{F}) \rightarrow (\mathcal{K}, \mathcal{D}).$$3ST-Adapter similarly adapts frozen image models to video by inserting disentangled spatial and temporal pathways with deformable spatio-temporal attention (Pei et al., 2023). These works are not action policies, but they establish the broader design principle that a frozen general backbone can be repurposed for downstream structure by adding lightweight modules rather than retraining the trunk.

Within VLA proper, several misconceptions are addressed by the literature. One is that the backbone alone determines performance. VLA-Adapter explicitly shows that bridging design can be as consequential as backbone scale: a 0.5B model with a stronger bridge can outperform OFT-style bridging on 7B backbones (Wang et al., 11 Sep 2025). Another is that robustness must come from more data or larger pre-training. StableVLA argues for an architectural bottleneck at the vision-to-policy interface and reports competitive robustness without extra data or Open X-Embodiment pre-training (Fu et al., 18 May 2026). A third misconception is that language must be processed at high frequency during execution. CT-VAM formalizes the opposite hypothesis: once a compact control-sufficient intent has been derived, low-level control can proceed from vision, proprioception, and that compact condition (Li et al., 8 Jun 2026).

The limitations are equally consistent across papers. CompliantVLA-adaptor notes that VLMs are too slow for high-frequency control, API access is expensive, on-board models are resource-heavy, and VLM-generated impedance suggestions are not guaranteed to be optimal or to generalize to unseen objects and environments (Zhang et al., 21 Jan 2026). Action ControlNet identifies robustness primarily within the delay range sampled during training and treats asynchronous errors as local boundary-conditioning problems rather than global replanning problems (Guo et al., 24 Jun 2026). CLARE adds routing and discriminator overhead, and its layerwise routing remains independent across layers rather than globally coordinated (Römer et al., 14 Jan 2026). StableVLA highlights a trade-off between robustness and high-frequency detail, with IB-only adapters degrading precision on some long-horizon tasks and requiring a fused path to recover fidelity (Fu et al., 18 May 2026). The medical VA-Adapter is trained on a single device family and ten standard views, so broader clinical and hardware generalization remains unverified (Wang et al., 8 Oct 2025).

Taken together, these works suggest a technically coherent but still unsettled research area. A plausible synthesis is that future VA-Adapters will become modular action interfaces that combine several of the current strands: compact intent tokens from high-level VLMs, layerwise multimodal conditioning, motion-history encoding, robustness bottlenecks, and latency-aware chunk execution. The existing literature already provides the constituent mechanisms—token compilers (Shi et al., 9 Mar 2026), residual action-head adapters (Guo et al., 24 Jun 2026), safety and impedance adaptors (Zhang et al., 21 Jan 2026), robustness filters (Fu et al., 18 May 2026), continual-routing modules (Römer et al., 14 Jan 2026), and sequence-aware encoder insertions (Wang et al., 8 Oct 2025)—but a unified theory of when each adapter form is preferable has not yet been established.