IVRA: Training-Free Vision-Token Refinement

• IVRA is a technique that restores 2D spatial relations in visual-token sequences by leveraging frozen vision encoder affinity hints during inference.
• It computes a cosine similarity-based affinity matrix and blends pooled token features with original embeddings via a mixing coefficient for effective spatial realignment.
• Experimental results demonstrate consistent improvements in simulation and real-robot benchmarks, with gains up to 30% in manipulation task performance.

IVRA (Improving Visual-Token Relations for Robot Action Policy with Training-Free Hint-Based Guidance) is a lightweight, training-free technique that enhances the spatial structure of vision-language-action (VLA) models by injecting affinity hints derived from a model's own frozen vision encoder. IVRA addresses the loss of 2D spatial locality caused by the tokenization and flattening of image patches in standard VLA pipelines. IVRA operates as an inference-time intervention, requiring no architectural modification nor retraining, and yields consistent improvements in robot manipulation policies across both simulation and real-robot settings (Park et al., 22 Jan 2026).

1. Motivation and Problem Context

Modern VLA models—exemplified by architectures such as LLaRA, OpenVLA, and FLOWER—typically process input images by partitioning them into an $H \times W$ grid of $N$ patches, which are then flattened into a one-dimensional token sequence. This transformation destroys local spatial relationships, causing instance-level visual boundaries (such as object edges or color transitions) to blur after repeated mixing in the model's transformer layers. As VLA models interleave these visual tokens with text tokens within LLM-based self-attention blocks, critical geometric structure is lost, hampering the model's ability to localize and manipulate distinct object features.

Pretrained vision encoders (e.g., CLIP-ViT or DINO) still encode strong 2D spatial priors within their intermediate representations. By extracting pairwise patch correlations—termed affinity hints—from these features, one can recover a mapping of which patches are likely to belong to the same object or region. Leveraging such affinity hints enables the restoration of spatial structure without introducing additional training or parameters.

2. Mathematical Formulation

Given a set of $N$ image patches $P = \{p_1, \ldots, p_N\}$, a frozen vision encoder produces patch-level features $f_i \in \mathbb{R}^d$. The pairwise cosine similarity-based affinity matrix $A \in \mathbb{R}^{N \times N}$ is defined as:

$$A_{ij} = \frac{\langle f_i,\,f_j\rangle}{\|f_i\|\,\|f_j\|}$$

At a selected layer $\ell$ of the LLM, the corresponding sequence of visual tokens is represented as $$H = [v_1, \ldots, v_N]^T \in \mathbb{R}^{N \times d}$$. The affinity matrix is used to pool information across visual patches via nonnegative, row-normalized weights:

$$\alpha_{ij} = \frac{\max(A_{ij},0)}{\sum_{k=1}^N \max(A_{ik},0)}$$

The affinity-pooled token is then computed as:

$N$0

A mixture of the original and affinity-pooled tokens is produced:

$N$1

where $N$2 is a mixing coefficient. In matrix notation, with $N$3 and $N$4:

$N$5

This operation effectively realigns and sharpens object-level neighborhoods within the flattened visual embedding sequence.

3. Inference-Time Intervention and Integration

IVRA is deployed at inference by registering forward pre-hooks on selected transformer layers. No model weights are modified, and all processing is stateless:

1. Patch & Encode: The input image is divided into $N$6 patches. Feature vectors $N$7 are extracted from a specific intermediate vision encoder layer and used to compute the affinity matrix $N$8.
2. Token Construction: Visual patches are associated with their embeddings $N$9. The complete input token sequence combines language, vision, and delimiter tokens.
3. Model Forward Pass: For each designated transformer layer $N$0, the visual-token block $N$1 is extracted, affinity-weights $N$2 are computed, pooled features are calculated via matrix multiplication, and the blended features are substituted back into the model. All other layers proceed normally.
4. Policy Decoding: The updated sequence is passed on for action decoding.

In practical implementations (e.g., with PyTorch), this is achieved by registering a forward_pre_hook at the chosen block to intercept and modify the requisite tensor slice before self-attention. IVRA integration for LLaRA and OpenVLA hooks into layers 19–23 (with layer 20 preferred), and for FLOWER requires only minimal forward-hook registration.

Typical values are $N$3 for LLaRA/OpenVLA and $N$4 for FLOWER, and the method targets one or a small range of mid-to-late LLM layers per empirical ablation.

4. Experimental Evaluation and Results

IVRA was evaluated across both simulated and real-robot manipulation benchmarks.

Benchmarks

• 2D VIMA (Jiang et al., ICML 2023): Four tasks covering generalization on novel tasks, novel objects, object combinations, and object placement, using only 12% ($N$5k) of the original demonstrations.
• 3D LIBERO (Kim et al., 2024): Goal, Object, Spatial, and Long task suites, plus LIBERO-90 for manipulation generalization in 3D.
• Real-Robot Zero-Shot: Four tabletop tasks (T1–T4) involving object selection based on diverse attributes. All policies trained in simulation only.

Performance Gains

Benchmark	Baseline Model	Baseline (%)	+IVRA (%)	Absolute Gain
VIMA (12% data)	LLaRA	53.9	58.1	+4.2
LIBERO	OpenVLA	76.5	77.6	+1.1
LIBERO	FLOWER	96.3	97.1	+0.8
Real-Robot T1	inBC-8k policy	50	60	+10
Real-Robot T2	inBC-8k policy	30	60	+30
Real-Robot T3	inBC-8k policy	45	75	+30
Real-Robot T4	inBC-8k policy	50	70	+20

Per-task improvements (e.g., Novel Task +5.0%, Novel Object +4.2%) indicate IVRA's broad impact across generalization suites (Park et al., 22 Jan 2026).

5. Ablation Studies and Visualization

Single-layer placement at layer 20 in a 32-layer transformer was optimal, outperforming both other single layers and multi-layer placements. The location of pooling just before the self-attention sublayer provided the best result. The mixing coefficient $N$6 yielded the best balance between restoration of spatial structure and oversmoothing; higher values degraded performance.

Visualizations of affinity maps in token space reveal that baseline models generate diffuse maps, causing object-background bleeding. IVRA produces sharply localized affinity maps, tightly confined to object regions, demonstrably improving object boundary preservation and attribute discrimination for features such as color, shape, and relative height.

6. Significance, Limitations, and Future Directions

IVRA illustrates that 2D spatial priors intrinsic to frozen vision backbones can be injected into downstream VLA models at inference-time, providing substantial and generalizable boosts in spatial grounding and action success without retraining or parameter updates.

Current implementations utilize fixed values for $N$7 and layer indices; adaptive schemes or online tuning are potential areas for exploration. Extending IVRA to operate over dynamic inputs, such as continuous video (spatiotemporal affinities) or 3D voxel representations, may enhance its utility for temporal or high-dimensional spatial reasoning. Additionally, hybridization with other zero-shot signal sources (depth, motion cues) is a plausible direction for achieving finer-grained semantic manipulation.

In summary, IVRA is a plug-and-play, training-free method for restoring 2D geometric structure in VLA models. It achieves consistent gains across diverse VLA architectures and tasks by selectively leveraging vision-encoder-derived affinities to realign and sharpen visual-token interactions deep within LLM pipelines (Park et al., 22 Jan 2026).