RoboBrain 2.0: Embodied Vision-Language Model

• RoboBrain 2.0 is an embodied vision–language foundation model that unifies perception, reasoning, and planning for complex physical tasks.
• It integrates a lightweight vision encoder with a decoder-only large language model using adaptive attention and multi-dimensional RoPE for robust spatial and temporal performance.
• The framework employs multi-stage training on diverse multi-modal data and demonstrates significant improvements over baselines in real-world robotic applications.

RoboBrain 2.0 is an embodied vision–language foundation model designed to unify perception, reasoning, and planning for complex tasks in physical environments. It is built as a heterogeneous encoder–decoder model with a lightweight vision encoder and a decoder-only LLM inheriting from Qwen2.5-VL. Two configurations are offered: a lightweight 7B model and a full-scale 32B model. RoboBrain 2.0 achieves strong results on both spatial and temporal embodied AI benchmarks and is evaluated against both open-source and proprietary models. It is released as a practical framework for embodied foundation models, addressing spatial understanding (such as affordance prediction and trajectory forecasting), temporal decision-making (including closed-loop interaction and multi-agent planning), and real-world physical robotic tasks (Team et al., 2 Jul 2025).

1. Model Architecture

RoboBrain 2.0 comprises a lightweight vision encoder (∼689M parameters) and a decoder-only LLM (either 7B or 32B parameters). The vision encoder is a ViT-style transformer with adaptive (windowed) attention and multi-dimensional rotary positional encoding (RoPE), optimized for high-resolution images, multi-view inputs, and video frames. Vision representations are projected into the LLM token space by an MLP projector: $$\mathbf{v}_i = E_{\rm vision}(I_i)\quad\in\R^{d_v}, \qquad \mathbf{e}_i = W_p\;\mathbf{v}_i+b_p \quad\in\R^{d_{\rm LLM}}$$ where $W_p, b_p$ are the learned projection parameters.

The decoder LLM performs unified reasoning and planning. It supports three output modalities: free-form text, spatial coordinates or bounding boxes, and chain-of-thought traces. Multi-modal input sequences, including visual embeddings, tokenized language, and optionally scene-graph JSONs, are concatenated and processed. Cross-attention integrates visual keys/values and text queries: $$\mathrm{CrossAttn}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$ The 7B and 32B variants differ in depth, hidden dimensionality, and attention modules (see table).

Variant	Decoder layers	Hidden dim	Attention heads	Novel Blocks	Total params
RoboBrain 7B	32	4,096	32	windowed attention in encoder	~7B
RoboBrain 32B	64	8,192	64	multi-dim RoPE, adaptive vision blocks	~32B

A composite loss is used during stage 3 RL fine-tuning: $$\mathcal{L}(\theta) = \lambda_{\rm CE}\mathcal{L}_{\rm CE} + \lambda_{\rm RL}\mathcal{L}_{\rm RL} + \lambda_{\rm reg}\|\theta\|_2^2$$ with cross-entropy, group relative policy optimization, and weight decay respectively (Team et al., 2 Jul 2025).

2. Data Construction and Preprocessing

RoboBrain 2.0 is trained on approximately 5 million multi-modal samples, combining general VQA, spatial, and temporal datasets:

• General multimodal VQA (873K): LLaVA-665K, LRV-400K (synthetically generated via GPT-4)
• Spatial perception (∼2.2M):
  • Visual grounding: 152K images, 86K conversation sequences
  • Object pointing: 190K QA pairs
  • Affordance: 561K QA pairs
  • Spatial understanding: 826K samples (31 concepts)
  • Spatial referring: 802K samples
• Temporal modeling (∼1.1M):
  • Ego-view planning: 50K
  • ShareRobot planning: 1M QA
  • AgiBot planning: 9K QA
  • Multi-robot planning: 44K (DeepSeek-V3)
  • Closed-loop interaction: ~100K over-the-air trajectories

Annotation types include 2D points, bounding boxes, segmentation masks, 3D boxes/point clouds, keypoint trajectories, scene graphs, instructions, and chain-of-thought traces.

The preprocessing pipeline uses JSON serialization with WebDataset for data streaming, dynamic augmentation (random cropping, color jitter for images, template-based paraphrasing for language), category balancing, and maximum-length padding to avoid sequence fragmentation.

3. Multi-Stage Training Strategies

Training is conducted in three progressive stages:

Stage	Data samples	Objective	BS (per GPU)	LR (vision/LLM)	Epochs	Optimizer	Precision
Foundational SFT	4.8M	Supervised finetuning (VQA & video)	2	1e-4 / 1e-4	1	AdamW	FP16
Embodied SFT	224K	SFT on spatial + temporal data	2	1e-5 / 1e-5	1	AdamW	FP16
CoT-SFT	195K	SFT w/ GPT-4 rationales	4	1e-5	1	AdamW	FP16
RFT (RLVR)	45K	RL fine-tuning	1	1e-6	3	GRPO	FP16

Pseudocode for stage 3 RL fine-tuning (summarized):

1. Initialize policy from stage-2 checkpoint
2. For each epoch and batch (trajectory, reward), compute the expected verifiable reward
3. Update parameters by gradient descent on

$$\mathcal{L}_{\rm RL} + \lambda_{\rm CE}\mathcal{L}_{\rm CE}$$

This sequence enables the model to acquire both general and task-specific embodied behaviors.

4. Benchmarking and Performance

RoboBrain 2.0 demonstrates state-of-the-art or competitive performance on spatial and temporal embodied AI benchmarks, including affordance, spatial referring, trajectory, and multi-robot planning tasks.

Key spatial reasoning results:

Task	Metric	RoboBrain 7B	RoboBrain 32B	Best Baseline
Affordance (ShareRobot)	Accuracy ↑	28.05%	35.28%	Gemini 2.5 Pro: 23.8%
Spatial referring	Loc/Pla ↑	36.0/29.0	54.0/54.0	Gemini 2.5 Pro: 44.6/31.7
Trajectory (ShareRobot)	DFD ↓ (lower=better)	0.5512	0.2368	Qwen2.5-VL 72B: 0.5034

Key temporal reasoning results:

Task	Metric	RoboBrain 7B	RoboBrain 32B	Best Baseline
Multi-Robot Planning	Mean acc ↑	81.50	80.33	GPT-4o: 74.5
Ego-Plan2	Mean acc ↑	33.23	57.23	Qwen2.5-VL 32B: 56.25
RoboBench Planning	Plan acc ↑	72.16	68.33	GPT-4o: 68.6

No explicit p-values are reported, but improvements of +10–20% over prior baselines significantly exceed typical confidence margins for these evaluation regimes. Ablations indicate a 2–3% gain in long-horizon tasks arising from RL fine-tuning after chain-of-thought SFT (Team et al., 2 Jul 2025).

5. Real-World Applications, Infrastructure, and Limitations

RoboBrain 2.0 is applied to physical robots and simulators in the following domains:

• Embodied task execution: Object affordance detection and pointing on physical mobile manipulators.
• Closed-loop control: Cup-pouring and pick-and-place on dual-arm real robots and AI2-Thor simulation.
• Multi-robot planning: Coordinated navigation and tasking in restaurant/hospital scenarios using RoboOS.
• Deployment infrastructure: Training uses FlagScale distributed across hybrid DP/PP/TP setup, with memory optimizations and ViT-layer recompute. Inference employs mixed-precision quantization (vision in FP32; LLM in 8-bit/FP16) for ∼30% latency reduction with <0.5% accuracy loss. Supports on-edge GPU execution (e.g., NVIDIA Jetson AGX) and serverless deployment paradigms.
• Resource footprint: The 32B model trains on 64 GPUs across 8 nodes in 3 days; online inference is 4 req/sec at 1 sec latency on a 16GB GPU. The 7B model fits 4×A100 40GB GPUs, serving 10 req/sec at 0.3 sec latency.

Limitations are primarily in the scope of real-world robotic testing (currently restricted to tabletop tasks), with language–action integration and open-world navigation cited as future directions. Planned work includes further integration into robotics middleware via an “intelligence app store” for modular perception, reasoning, and control components (Team et al., 2 Jul 2025).

6. Context and Significance

RoboBrain 2.0 combines architectural innovations (heterogeneous encoder–decoder design with adaptive attention, multi-dimensional RoPE), a composite loss encompassing both cross-entropy and reinforcement learning, and a curated large-scale multi-modal dataset with diverse annotation, to push the boundaries in unified spatial and temporal reasoning for embodied agents. The system sets new performance baselines on shared benchmarks, and its infrastructure facilitates both practical deployment and extensibility for research applications. A plausible implication is that RoboBrain 2.0 provides a robust foundation for the design and scaling of future generalist embodied agents (Team et al., 2 Jul 2025).