---
title: 'SPEAR-VLM: 3D Robotics & Space Perception'
url: https://www.emergentmind.com/topics/spear-vlm
type: topic
---

# SPEAR-VLM: 3D Robotics & Space Perception

Searching arXiv for the relevant SPEAR-VLM papers to ground the article in current arXiv usage.
In current arXiv usage, **SPEAR-VLM** denotes two distinct vision-language systems with different problem settings and objectives. In robotics, SPEAR-VLM is a **3D-aware VLM** introduced as the perceptual backbone of **SPEAR-1**, where it extends PaliGemma with a monocular depth encoder, 3D token embeddings, and VQA-style 3D supervision so that the backbone can infer object coordinates in 3D space from a single 2D image [2511.17411]. In space-domain perception, SPEAR-VLM denotes an **annotation-free detection and segmentation pipeline** that uses a pre-trained VLM to generate pseudo-labels for spacecraft imagery, refines those labels with test-time augmentation and Weighted Boxes Fusion, and distills them into lightweight student models for inference [2602.04699]. The shared designation reflects a common reliance on VLMs, but the two systems differ fundamentally in architecture, supervision, downstream tasks, and deployment assumptions.

## 1. Terminological scope and research context

The robotics usage emerges from the argument that most vision-language models are pretrained on **2D web image-language data**, making them strong at semantics but weak at the **3D spatial reasoning** required for embodied control [2511.17411]. Within that framing, SPEAR-VLM is not itself the final robot policy; rather, it is the 3D-aware perceptual substrate used inside SPEAR-1. The stated motivation is to “replace” some robot data with **non-robotic images enriched with 3D annotations**, thereby reducing dependence on expensive robot demonstrations.

The space-domain usage addresses a different bottleneck: **manual annotation is expensive and difficult** for spacecraft imagery because targets are often small, low-visibility, partially occluded, and embedded in cluttered planetary or laboratory backgrounds [2602.04699]. Here, SPEAR-VLM is not a 3D-aware backbone for control, but an **annotation-free** pseudo-labeling, refinement, and distillation pipeline for spacecraft detection and segmentation.

This naming overlap can lead to a common misconception that SPEAR-VLM designates a single unified model family. The available evidence instead indicates two independent usages. A plausible implication is that the shared acronym should be interpreted locally, in relation to each paper’s domain and objective, rather than as a stable cross-domain benchmark family.

## 2. SPEAR-VLM in robotics: architectural definition

In the robotics formulation, SPEAR-VLM extends **PaliGemma** into a 3D-aware VLM [2511.17411]. PaliGemma is described as having three components: a **SigLIP** image encoder, a linear projector into language space, and a **Gemma** language model. SPEAR-VLM adds a **monocular depth encoder**, namely **MoGe**, extra **3D token embeddings** in the tokenizer, and training tasks that require explicit 3D reasoning.

The fusion mechanism is specified in implementation detail. The model keeps SigLIP and MoGe as separate encoders, extracts SigLIP last-layer visual tokens, extracts MoGe features from the **last 4 layers** of the MoGe ViT encoder, concatenates the MoGe features along the channel dimension, projects both SigLIP and MoGe features into the LLM embedding space, and **averages the outputs** of the SigLIP and MoGe projectors before feeding them to the language model [2511.17411]. The tokenizer is extended with **\(N=1024\) 3D tokens**, representing quantized distance values.

MoGe is selected because it is **monocular depth estimation** based and **affine-invariant**, which the paper associates with better generalization across cameras with different intrinsics. This is significant in robotics because camera variation is pervasive across embodiments and environments. The architecture therefore combines semantic visual tokens from SigLIP with dense geometric priors from MoGe, while preserving the autoregressive language-model interface of the base VLM.

A central claim in the paper is that object-centric 3D pretraining teaches the backbone **camera-to-object geometry**, **spatial relations between objects**, **3D localization**, and **viewpoint robustness** [2511.17411]. This suggests that the architectural additions are intended not as generic multimodal scaling, but as a targeted intervention on the geometric deficiencies of 2D-only VLM pretraining.

## 3. Robotics pretraining pipeline and 3D objectives

Because open datasets with explicit 3D object annotations are scarce, the robotics paper constructs a semi-automatic annotation pipeline using only **2D images** plus off-the-shelf models [2511.17411]. For each image, **Gemini** detects 2D bounding boxes and semantic labels, **SAM2** is prompted with these boxes to generate **instance masks**, and **MoGe** predicts a dense **3D point cloud**. The point cloud is then filtered with the instance mask, an **oriented 3D bounding box** is computed around the resulting object points, and VQA-style question-answer pairs are created from that result.

The pretraining data consists of **~200k images** from the **cooking** and **bike repair** subsets of **EgoExo4D** and **~30k frames** from **Bridge-V2** robot demonstrations, for a total of **~230k images** [2511.17411]. The paper emphasizes that this is a relatively small amount of 3D-annotated image data.

SPEAR-VLM is trained as a **VQA-style autoregressive model** that predicts 3D-related textual outputs. The listed task families are **3D keypoints prediction**, **3D bounding box prediction**, **object-to-object distance prediction**, **object-to-object bounding box distance**, **backprojection tasks**, and **chain-of-thought comparison tasks** about which object is closer to the camera [2511.17411]. Example prompts include “Output the vertices of the 3D bounding box of object X” and “Output the \(xyz\) components of the distance between object X and object Y.”

The 3D coordinates are represented using **1024 quantized tokens**, each corresponding to a distance bin in a range defined by the **1st and 99th percentiles** of the 3D point-cloud coordinate distribution [2511.17411]. The training objective remains next-token prediction, but the loss for 3D tokens is scaled by \(\lambda = 2\). The model is explicitly not described as directly regressing continuous coordinates with a separate head; instead, it **generates 3D token sequences** corresponding to quantized coordinates.

Training follows **two stages**, similar to LLaVA-style VLM training. In Stage 1, the model initializes from **pretrained PaliGemma** and **pretrained MoGe**, randomly initializes the MoGe projector, the new 3D token embeddings, and the SigLIP projector, trains only the randomly initialized weights and SigLIP projector, and keeps the rest frozen. In Stage 2, only the **SigLIP and MoGe encoders** are frozen, training continues for the remaining components, and the 3D-token loss weight is increased with \(\lambda = 2\) [2511.17411].

## 4. Integration into SPEAR-1 and embodied-control performance

SPEAR-1 is described as a **VLA / flow-matching action policy** built on top of SPEAR-VLM [2511.17411]. Its high-level structure follows the general \(\pi_0\)-style design: the **VLM** processes image and language inputs, an **action expert** predicts robot actions, and that action expert attends to the VLM’s intermediate key-value representations.

The observation at time \(t\) is
\[
\mathbf{o}_t = [\mathbf{I}_t^1, \dots, \mathbf{I}_t^n, \mathbf{p}_t, \mathbf{l}_t]
\]
where \(\mathbf{I}_t^i\) are camera images, \(\mathbf{p}_t\) is proprioception or robot state, and \(\mathbf{l}_t\) is the language instruction [2511.17411]. The policy outputs a horizon-\(H\) action sequence
\[
\mathbf{A}_t = [\mathbf{a}_t, \mathbf{a}_{t+1}, \dots, \mathbf{a}_{t+H-1}]
\]
with each action decomposed as
\[
\mathbf{a}_t = [\mathbf{x}_t, \mathbf{q}_t, \mathbf{g}_t]
\]
for translation, rotation, and gripper state.

The action expert is reported to have about **300M parameters**, the same as \(\pi_0\), and to use **shared attention** with the VLM transformer [2511.17411]. Tokens are organized into blocks \([\mathbf{I}_t, \mathbf{l}_t]\), \([\mathbf{p}_t]\), and \([\mathbf{a}_{t+1}, \dots, \mathbf{a}_{t+H-1}]\) with block-wise causal attention. The paper’s key claim is that SPEAR-VLM provides a stronger geometric backbone, reducing the need for the action model to infer 3D structure from scratch.

SPEAR-1 is trained on **24 Open X-Embodiment datasets** comprising about **~45M frames** total [2511.17411]. The most heavily weighted datasets are listed as **droid: 35.0**, **bridge: 18.0**, and **fractal20220817_data: 12.0**. Actions are resampled to a common **5 Hz** control frequency.

The reported headline comparison is that SPEAR-1 **outperforms \(\pi_0\)-FAST** and **matches \(\pi_{0.5}\)** while using **20× fewer robot demonstrations** [2511.17411]. On challenging unseen Franka environments in a DROID-like setup, it beats \(\pi_0\)-FAST, matches \(\pi_{0.5}\), and does so **without fine-tuning on the target environment**. The paper further states that SPEAR-1 can reach about **5× higher performance than \(\pi_0\)-FAST** in the zero-shot Franka setting. On WidowX, it achieves **~10% higher average task progress** than OpenVLA. On SIMPLER WidowX tasks, reported average success is **1.0%** for OpenVLA, **42.7%** for SpatialVLA, and **57.3%** for SPEAR-1 [2511.17411].

Ablations attribute these gains specifically to object-level 3D reasoning. In a Bridge V2 subset / SIMPLER setup, **baseline PaliGemma** and **SPEAR-VLM without object-level 3D tasks** both obtain **20.8% avg success**, whereas **SPEAR-VLM with object-level 3D tasks** reaches **35.4%** [2511.17411]. On Franka tasks, \(\pi_0\)-PaliGemma (DROID) achieves **34% avg task progress**, while \(\pi_0\)-SPEAR-VLM (DROID) reaches **46% avg task progress**. The paper interprets this as evidence that the benefit comes from **task-relevant 3D supervision**, not merely from adding another encoder.

## 5. SPEAR-VLM in spacecraft perception: annotation-free pipeline

In the spacecraft-perception formulation, SPEAR-VLM denotes an **annotation-free detection and segmentation pipeline for space targets using VLMs** [2602.04699]. The pipeline has four stages: **pseudo-labeling stage**, **label refinement stage**, **label distillation stage**, and **inference stage**.

For an image \(I\) and text prompt \(T\), the VLM \(\mathcal{V}\) produces segmentation and box predictions
\[
P_S, P_B = \mathcal{V}(I, T)
\]
where \(P_S\) denotes instance segmentation predictions and \(P_B\) denotes bounding box predictions [2602.04699]. Because the spacecraft datasets used in the paper mostly contain **single-instance images**, the method keeps only the **top detection and/or segmentation prediction** for each image, yielding a single pseudo box and pseudo mask. The fixed prompt is **“spacecraft.”**

The paper evaluates three open-source VLMs as teachers: **SEEM**, **OpenSEED**, and **GroundedSAM-2** [2602.04699]. Their zero-shot predictions are first evaluated directly; later, only **GroundedSAM-2** is carried forward, because it performs best.

Label refinement combines **test-time augmentation (TTA)**, **Weighted Boxes Fusion (WBF)**, and confidence filtering. With augmentations \(\mathcal{A} = \{ a_1, a_2, \dots, a_K \}\), each augmented image \(I_k = a_k(I)\) is processed by the VLM, box predictions are mapped back through the inverse augmentation, and all predictions are collected into a common set before fusion [2602.04699]. WBF groups boxes whose IoU exceeds a threshold \(\tau\), and the paper sets the overlap threshold to **\(\tau = 0.55\)**. For a cluster
\[
C = \{ (b_j, s_j) \}_{j=1}^M,
\]
the fused box is
\[
b^\ast = \frac{\sum_{j=1}^M s_j \, b_j}{\sum_{j=1}^M s_j},
\qquad
s^\ast = \frac{1}{M} \sum_{j=1}^M s_j.
\]
The fused class label is chosen by **majority voting or weighted voting based on scores** [2602.04699].

Confidence filtering then retains only fused boxes whose score exceeds a dataset-specific threshold \(\theta\). The reported thresholds are **\(\theta = 0.5\)** for **SPARK-2024**, **\(\theta = 0.5\)** for **TANGO**, and **\(\theta = 0.6\)** for **SPEED+** [2602.04699]. This step is presented as necessary because fused confidence may remain low when predictions across augmentations disagree strongly.

The distillation stage treats the refined pseudo-labels as teacher outputs and trains a **shallow student model**. Given the refined pseudo boxes \(\mathcal{P}_B^\ast_\theta\), the student \(S_\phi\) predicts
\[
\mathcal{Q} = S_\phi(I) = \{(b_j, p_j)\}_{j=1}^{N_S}.
\]
The supervised loss is written as
\[
\mathcal{L}_{\text{distill}} =
\alpha \, \mathcal{L}_{\text{cls}}(p, \hat{y}) +
\beta \, \mathcal{L}_{\text{reg}}(b, \hat{b}),
\]
where \(\hat{b}, \hat{y}\) are teacher box and class predictions, and \(p, b\) are student outputs [2602.04699]. The method also uses a single round of **iterative distillation**: train a student on pseudo-labels, use that student to relabel the training images, and train a new student from scratch.

At inference time, only the **student model** is used. The stated rationale is that the student is lightweight enough for real-time or near-real-time deployment onboard spacecraft [2602.04699].

## 6. Spacecraft datasets, models, and empirical results

The spacecraft paper evaluates on **SPARK-2024**, **SPEED+**, and **TANGO** [2602.04699]. SPARK-2024 is used for **detection only**, because it contains **bounding box annotations** but **no segmentation annotations**. The paper uses only test sequences, with **500 images** randomly selected as a training split and the remaining **1600** for evaluation. For SPEED+, the authors create custom splits of **500 images for training**, with **6200** Lightbox images and **2200** Sunlamp images for testing. For TANGO, the paper uses the test split as-is together with **500 randomly selected training images**.

Evaluation uses COCO-style **Average Precision (AP)**, averaged over IoU thresholds from \(0.50\) to \(0.95\) with step size \(0.05\), together with **AP\(_{50}\)** and **AP\(_{75}\)** [2602.04699]. The distilled students are **Efficient-Det** for object detection and **YOLOv11** for object detection and segmentation. Both are described as around **7M parameters**, running in real time with **FPS > 60**, trained with **official implementations** for **300 epochs**, and using images resized to **640×640**.

The zero-shot VLM baseline results identify **GroundedSAM-2** as the strongest teacher. On **SPARK-2024 detection**, SEEM obtains **AP 17.7**, **AP\(_{50}\) 49.8**, **AP\(_{75}\) 2.6**; OpenSEED obtains **AP 8.5**, **AP\(_{50}\) 28.0**, **AP\(_{75}\) 0.8**; and GroundedSAM-2 obtains **AP 53.3**, **AP\(_{50}\) 97.9**, **AP\(_{75}\) 61.7** [2602.04699]. For **SPEED+ Sunlamp**, GroundedSAM-2 reaches **detection AP 73.9**, **AP\(_{50}\) 97.9**, **AP\(_{75}\) 92.1**, and **segmentation AP 72.5**, **AP\(_{50}\) 94.7**, **AP\(_{75}\) 85.0**. For **SPEED+ Lightbox**, it achieves **detection AP 65.8**, **AP\(_{50}\) 91.4**, **AP\(_{75}\) 74.0**, and **segmentation AP 66.2**, **AP\(_{50}\) 83.9**, **AP\(_{75}\) 73.5**. For **TANGO**, it obtains **detection AP 58.2**, **AP\(_{50}\) 91.8**, **AP\(_{75}\) 75.7**, and **segmentation AP 58.2**, **AP\(_{50}\) 85.9**, **AP\(_{75}\) 69.0**.

The paper reports that TTA and WBF improve zero-shot predictions modestly, with the **best result obtained using only vertical flip**; adding more augmentations tended to **decrease** performance [2602.04699]. After TTA and WBF, example results include **SPARK AP 54.4** with **AP\(_{75}\) 64.0**, **Sunlamp detection AP 74.6** with **AP\(_{75}\) 93.7** and **segmentation AP 73.2**, **Lightbox detection AP 66.9** with **AP\(_{75}\) 75.6** and **segmentation AP 66.8**, and **TANGO detection AP 58.5** with **AP\(_{75}\) 76.3** and **segmentation AP 58.7**.

The principal contribution is the improvement after distillation. On **SPARK-2024**, the paper reports **Efficient-Det AP 65.0** and **YOLOv11 AP 68.9** under TTA+WBF, **YOLOv11 AP 70.8** after confidence filtering, and **YOLOv11 AP 76.4** with **AP\(_{75}\) 98.2** after iterative relabeling [2602.04699]. On **SPEED+ Sunlamp**, the distilled **TTA+WBF+CF+TR** result is **detection AP 75.0** and **segmentation AP 74.0**. On **SPEED+ Lightbox**, the corresponding final result is **detection AP 71.0** and **segmentation AP 63.5**. On **TANGO**, the distilled pipeline reaches **detection AP 86.5** and **segmentation AP 70.0**, and the authors explicitly state that, compared with the initial VLM predictions, the method improves **more than 10 points on TANGO segmentation AP**.

The paper’s explanation is that the student acts as a **denoising regularizer**: it can ignore some teacher noise through limited capacity, generalize better with less overfitting, and learn a task-specific decision boundary adapted to the space domain [2602.04699]. This is a domain-adaptation argument rather than a claim about the student exceeding the teacher in general open-vocabulary capability.

## 7. Comparative significance, limitations, and recurrent themes

Despite their different domains, the two SPEAR-VLM usages share a methodological pattern: each treats a VLM not only as a semantic recognizer, but as a platform that can be specialized by injecting task-relevant structure. In robotics, that structure is **explicit 3D reasoning** grounded in quantized geometric tokens and object-centric VQA supervision [2511.17411]. In spacecraft perception, it is **annotation-free pseudo-supervision** refined through TTA, WBF, confidence filtering, and teacher–student distillation [2602.04699].

The most important distinction is where the additional structure enters the learning system. In the robotics paper, SPEAR-VLM modifies the **backbone representation itself** by augmenting PaliGemma with MoGe, 3D token embeddings, and 3D pretraining tasks. In the spacecraft paper, SPEAR-VLM primarily modifies the **training pipeline** around an existing teacher VLM, then transfers its predictions into compact students. This suggests two broad interpretations of VLM specialization: one centered on **backbone geometric enrichment**, the other on **pseudo-label-mediated downstream adaptation**.

Both formulations also expose clear limitations. The robotics paper notes that gains depend on the quality of object-level 3D supervision and on model-training choices such as whether MoGe is frozen during VLA training; it also states that most RFMs remain limited in their ability to generalize across new environments, tasks, and embodiments [2511.17411]. The spacecraft paper states that performance still depends on the quality of the initial VLM predictions, that gains are smaller on some easier or cleaner datasets, that iterative relabeling can occasionally slightly hurt performance due to overfitting, and that validation is restricted to datasets with mostly single spacecraft instances and similar target categories [2602.04699].

A common misconception would be to read either system as evidence that generic 2D web-scale VLM pretraining alone is sufficient for specialized perception. Both papers argue the opposite in domain-specific ways. The robotics work contends that 2D-only pretraining lacks the spatial priors required for manipulation, while the spacecraft work shows that direct zero-shot VLM inference can be substantially improved by refinement and distillation. A plausible implication is that VLMs, in these settings, function most effectively when coupled with domain-aligned inductive bias: 3D grounding for embodied control, and label-refinement plus compact adaptation for space-domain detection and segmentation.

Source: https://www.emergentmind.com/topics/spear-vlm