Point-VLA: Pixel-Level Visual Grounding for VLA

• Point-VLA is a plug-and-play module that augments standard VLA models with explicit pixel-level visual cues, enhancing precision in target identification.
• It overlays bounding boxes on reference images to resolve ambiguity in text descriptions, enabling fine-grained robotic manipulation in cluttered and OOD scenarios.
• Empirical evaluations show Point-VLA achieves up to 92.5% success rates, significantly outperforming text-only methods in various manipulation tasks.

Point-VLA refers to a plug-and-play policy module for Vision-Language-Action (VLA) models that enables pixel-level visual grounding of object references via explicit visual cues (e.g., bounding boxes) overlaid on a reference image. Designed for robotic embodied control tasks involving fine-grained object manipulation, Point-VLA augments language instructions with user- or model-indicated localization in the visual input, resolving the ambiguity and limitations associated with text-only referring policies—especially in cluttered, OOD (out-of-distribution), and precise spatial manipulation scenarios (Yu et al., 22 Dec 2025).

1. Motivation and Problem Setting

Conventional VLA policies synthesize language inputs $l_t$ and multi-view camera images $I_t$ to produce continuous action commands $a_t$, training purely by imitation of expert demonstrations: $\hat a_t = \pi_\theta(l_t, I_t)$. However, text-based reference alone suffers from key limitations:

• Inexpressible references: Language is often incapable of uniquely specifying amorphous, ambiguous, or spatially dense targets (e.g., “that lump of clay”, “the 2nd egg from the left in row 3”).
• Generalization: Text descriptions for previously unseen objects or complex referential tasks often result in policy failures in OOD placement or cluttered scenes.

Point-VLA is explicitly designed to address these deficits. It introduces an explicit visual marker—a bounding box or mask—on the initial overhead camera frame, $\tilde{I}_{g,0}$, indicating the region of interest directly. The modified policy thus unifies two operation modes:

• Text-only: $\hat a_t = \pi_\theta(l_t, I_t)$
• Visually grounded: $\hat a_t = \pi_\theta(l_t, I_t, \tilde{I}_{g,0})$

By supplying $(\tilde{I}_{g,0}, g)$, Point-VLA enables unambiguous, pixel-level object specification even for otherwise linguistically intractable tasks (Yu et al., 22 Dec 2025).

2. Model Architecture and Input Modalities

The Point-VLA policy operates as an augmentation (“plug-in”) atop any modern VLA transformer backbone (e.g., π₀.₅):

• Inputs:
  • Multi-view camera images: Live RGB input from one or more cameras, typically one overhead and multiple wrist cameras.
  • Textual intent: Compact, imperative language instruction (e.g., “pick up”, “place here”).
  • Grounding frame: $\tilde{I}_{g,0}$, the first overhead frame with the target region $g$, usually rendered as a bounding box.
• Modality Encoders:
  • Language: Tokenized and embedded with learned positional encodings, yielding $X^l \in \mathbb{R}^{N_l \times d}$.
  • Vision: Each frame (including $\tilde{I}_{g,0}$) is embedded by a shared visual backbone, resulting in $X^v \in \mathbb{R}^{N_v \times d}$ per image.
• Fusion Mechanism:
  • Modal encodings are stacked: $$X_{\mathrm{all}} = [X^l; X^v(I_t^{(1)}); ... ; X^v(\tilde{I}_{g,0})]$$.
  • All tokens are processed by a multimodal transformer encoder, producing contextualized features $H = \mathrm{Transformer}(X_{\mathrm{all}})$.
• Policy Head:
  • The final token embedding (e.g., [CLS]) is fed into an MLP, outputting a probability distribution over action candidates: $p(a_t|\cdot) = \mathrm{Softmax}(Wh^* + b)$.

This architecture supports joint training and inference across both text-only and visually grounded modes without separate networks (Yu et al., 22 Dec 2025).

3. Training Regime and Data Annotation

Co-training on mixed samples is critical for Point-VLA:

• Supervised Datasets:
  • $$\mathcal{D}_{\mathrm{text}} = \{(l_t, I_t) \to a^*_t\}$$, standard text-to-action pairs.
  • $$\mathcal{D}_{\mathrm{visual}} = \{(l_t, I_t, \tilde{I}_{g,0}) \to a^*_t\}$$, samples with explicit visual grounding.

Batch sampling is balanced at 1:1 between text-only and visually grounded examples during training. The policy loss is the cross-entropy over action predictions:

$$L_{\mathrm{policy}}(\theta) = - \mathbb{E}_{(l, I, \tilde{I}_g) \sim \mathcal{D}} \sum_t \log \pi_\theta(a_t^*|l, I_t, \tilde{I}_{g,0})$$

Automatic Annotation Pipeline: Large-scale supervised visual grounding data are generated via a multi-stage pipeline using robust MLLM annotation (Gemini ER-1.5):

1. Sample 20 frames per episode from overhead and wrist cameras.
2. MLLM identifies task type, controlling arm, and the frame of grasp/release.
3. MLLM produces a tight 2D bounding box (normalized coordinates) for the manipulation target.
4. Box is projected to the first overhead frame, forming $(\tilde{I}_{g,0}, g)$.
5. The annotated sample is augmented through random translations (±10%) and CutMix (localized patch up to 10% box area).

Manual audit indicates 92% annotation fidelity; errors (≈8%) typically arise with occlusion or ambiguous target localization (Yu et al., 22 Dec 2025).

4. Core Algorithmic Procedures

Key operational components include data preparation and inference:

Annotation Pipeline (pseudocode):

for each demonstration episode D: sample T frames from overhead + wrist cameras prompt MLLM with (frames, text desc) → JSON { task_type, arm_used, key_frame, bounding_box } project bounding_box to first-frame overhead → (Ĩ_{g,0}, g) augment (Ĩ_{g,0}, g) with random shift and localized CutMix add to 𝒟_visual: (l, Iₜ, Ĩ_{g,0}, g) → aₜ* end

Inference Procedure—two supported modes:

• Manual: User draws box $g$ on $\tilde{I}_{g,0}$ and inputs $l$; the policy predicts $\hat{a}_t$.
• Automatic: User points on overhead image; MLLM infers $g$; policy processes $(\tilde{I}_{g,0}, g)$ as above.

5. Empirical Evaluation and Results

Point-VLA was benchmarked on six robot manipulation tasks, across 12 real-world scenes (30 trials/scene), including irregular-shape picking, OOD object retrieval, dense clutter, and slot-specific placement tasks. Models compared:

• “Text VLA”: standard $\pi_{0.5}$ trained on text only
• “Interleave-VLA”: images interleaved as text tokens, no explicit cue
• “Point-VLA”: $\pi_{0.5}$ with joint text and pixel cue training

Task Success Rates:

Method	Irreg.	OOD	Clutter	Egg-pick	Plain	Egg-place	Avg
Text VLA	30.0	57.5	43.3	10.0	30.0	23.3	32.4
Interleave-VLA	60.0	86.7	33.3	13.3	26.7	20.0	40.0
Point-VLA	96.7	92.5	94.3	86.7	95.0	90.0	92.5

Text-only compatibility was maintained: co-training on mixed-mode data did not degrade text-only performance and in some cases improved it. Plug-and-play operation was demonstrated even when fine-tuned on lighter ($\pi_0$) architectures or on alternate robotic embodiments, with similar (>40 point) average gains over baselines (Yu et al., 22 Dec 2025).

6. Analysis: Generalization, Limitations, and Scaling

Generalization: Point-VLA achieved robust performance gains in clutter, OOD, and precise grid tasks, and maintained generalization under variations in viewpoint, scene, and embodiment. Notably, model scaling studies show continued improvement with added object diversity and data, whereas baseline text-only policies saturate.

Ablation Studies:

• Overlaying the visual box as an explicit cue outperformed box text-coordinate input and mask-only approaches ($\sim$80–86% vs. 10–43% success in egg-picking OOD).
• Removing random shift augmentation rendered egg tasks brittle to tray perturbations; removing CutMix crippled OOD object picking.

Failure Modes and Limitations:

• The approach is dependent on a fixed first-frame overhead view; camera miscalibration or drift can misalign referents.
• Annotation inaccuracies (~8%) are concentrated in scenes with occlusion or ambiguous containment. Multi-view temporal consistency or online target tracking is identified as a direction for mitigation (Yu et al., 22 Dec 2025).

7. Implementation Details and Reproducibility Considerations

• Backbone: π₀.₅ VLA transformer (and π₀ for validation).
• Training: 20,000 fine-tuning steps per task; Adam optimizer with backbone defaults; batches alternate text-only and visually grounded samples.
• Dataset: ~2 hours of demonstration per scene, 12 scenes (~24 hours total); all demonstrations paired with curated human-written text instructions.
• Data Augmentation: Random shifts (±10%), localized CutMix patches (10% of box area).
• MLLM Annotation: Gemini ER-1.5, four-stage structured prompt covering 20 frames × 3 views.
• Evaluation: 30 trials/scene, up to 2 retries/trial, success determined by manipulation lift or ≤10 cm placement tolerance.

In summary, Point-VLA constitutes a direct and practical extension to existing VLA transformer policies. It delivers large, reliable gains in referential accuracy in manipulation tasks through the simple device of supplementing language with pixel-level spatial cues, realized via a scalable and high-fidelity annotation pipeline, while retaining and in some cases enhancing native text-only capabilities (Yu et al., 22 Dec 2025).