VLA Foundry: Unified VLA Research

• VLA Foundry is a unified research framework standardizing Vision-Language-Action model development with modular components and end-to-end pipelines.
• It streamlines the integration of LLM, VLM, and VLA stages using a three-phase pretraining and fine-tuning process to ensure reproducibility.
• The infrastructure supports backbone swaps, extensible action heads, and robust benchmarking tools to accelerate progress in embodied AI.

VLA Foundry refers to both a conceptual and concrete infrastructure supporting unified, reproducible, and extensible research in Vision–Language–Action (VLA) modeling for embodied agents. It emerged in response to fragmentation across VLA research—particularly disparate pretraining pipelines, incompatible code bases, and inconsistent evaluation protocols. VLA Foundry frameworks standardize model components, training regimens, and benchmarking, facilitating transparent attribution of empirical gains and accelerating progress toward robust, generalist robotic policies (Mercat et al., 21 Apr 2026).

1. Conceptual Foundations and Motivation

The VLA Foundry paradigm addresses the ecosystem's demand for minimal, modular, and extensible VLA research baselines. Historically, VLA model development splintered along lines of action-head design, vision–language backbone selection, and data engineering. These differences complicated benchmarking and attribution of improvements (Wu et al., 20 Feb 2026). With the introduction of SimVLA and VLANeXt, the notion of a "foundry" baseline—i.e., a minimal, reproducible starting point—gained prominence. VLA Foundry frameworks push this approach further by unifying all learning stages (language, vision–language, and vision–language–action) within a single stack. By collapsing the separation between LLM, VLM, and VLA model development and supporting both from-scratch and pretrained backbone workflows, the VLA Foundry infrastructure offers an unprecedented degree of reproducibility and modularity (Mercat et al., 21 Apr 2026).

2. Unified LLM→VLM→VLA Training Pipeline

VLA Foundry enables a three-stage pretraining and fine-tuning workflow:

• Stage 1: LLM Pretraining
  • Standard transformer decoder trained on raw text corpora with cross-entropy loss.
  • Example: 24-layer, 2048-dimensional decoder; vocab size 49,280.
  • Output: Foundry-LLM checkpoint.
• Stage 2: Vision–LLM (VLM) Pretraining
  • Encodes images (e.g., 224×224 ViT input) and captions into the LLM embedding space using CLIP-style ViT encoders and pixel-shuffle pooling.
  • Output: Foundry-VLM checkpoint.
• Stage 3: VLA Policy Fine-Tuning
  • Inputs: Synchronized multi-camera images, prompt text, proprioception, action windows.
  • Model: VLM features injected with an observation token; action head predicts denoising vectors for future SE(3) actions using flow-matching losses.
  • Output: Foundry-VLA checkpoint.

The pipeline enforces a homogeneous data-handling, optimizer, and orchestration regime across all stages. Stochastic multimodal dataset mixing, distributed gradient accumulation, and Draccus-configured hyperparameter registries ensure end-to-end traceability and modularity (Mercat et al., 21 Apr 2026).

3. Model Architecture and Module Specifications

Within the VLA Foundry codebase, models are constructed from the following modular components:

Module	Parameters	Typical Architectures
LLM	~1.4B	24-layer transformer decoder, 2048-dim
VLM	~86M–2B	CLIP-style ViT, patch 16×16, Qwen3-VL
Action Head	~325M (flow transformer)	12–24 transformer layers, 2048-dim
Registry	Configurable (YAML)	Any HuggingFace VLM/LLM, plug-in head

Input sequences are composed as follows: [pooled image tokens] + [prompt tokens] + [obs token] + optional past proprioception + (for diffusion models) noised action window. The flow-matching head outputs denoising vectors for each step in the future action trajectory. Downstream modules can be replaced or frozen independently, enabling easy experimentation with new architectures and objectives (Mercat et al., 21 Apr 2026).

4. Training Objectives, Loss Functions, and Hyperparameters

VLA Foundry standardizes objectives at each stage:

• LLM Pretraining Loss: Standard cross-entropy over next-token prediction.

$$\mathcal{L}_{\text{lang}} = -\frac{1}{N\,L}\sum_{i=1}^N\sum_{t=1}^L \log p_\theta(w_{t}^{(i)}\mid w^{(i)}_{<t})$$

• Vision–Language Captioning Loss:

$$\mathcal{L}_{\text{vision}} = -\frac{1}{N\,L_c}\sum_{i=1}^N\sum_{t=1}^{L_c} \log p_\theta(c_{t}^{(i)}\mid I^{(i)},c^{(i)}_{<t})$$

• VLA Action Loss (Flow Matching):

$$\mathcal{L}_{\text{action}} = \mathbb{E}_{a_0,\,t,\,\epsilon} \Bigl\| s_\theta(a_t,t\mid\mathrm{obs}) - \epsilon \Bigr\|^2$$

where $a_t = \alpha(t)a_0 + \sigma(t)\epsilon$, with $\epsilon\sim\mathcal{N}(0,I)$, leverages a continuous regression target for denoising vector prediction.

• Protocol: Uses AdamW (β₁=0.9, β₂=0.95, decay=1e-2), batch sizes 512–1024, cosine learning rate decay plus linear warmup. Precision settings are amp_bfloat16 with efficient distributed training via DDP or FSDP2 orchestration. Each stage has concrete YAML-configured schedules (Mercat et al., 21 Apr 2026).

5. Benchmarking, Evaluation, and Codebase Infrastructure

Evaluation centers on open, closed-loop simulation via the LBM Eval platform:

• Simulator: LBM Eval (open-source, Drake-based, dockerized).
• Tasks: 49 tabletop bimanual manipulation tasks (16 seen, 3 held out for zero-shot); input modalities: four RGB cameras and proprioception.
• Metrics: Task-level success rate (on at least 200 rollouts per policy), aggregated via Bayesian posterior estimates and the STEP toolkit for family-wise error control and unbiased multi-task comparison.
• Reproducibility Infrastructure:
  • Codebase (https://github.com/TRI-ML/vla_foundry) delivers a single training loop supporting all pipeline stages and backbone swaps.
  • Preprocessed WebDataset shards, configuration presets, and manifest files facilitate reproducibility.
  • Model checkpoints released on Hugging Face and results with videos available through project website.

Empirical benchmarks show that the VLA Foundry from-scratch model (1.7B) matches prior state-of-the-art multi-task policies, while Qwen3-VL backboned variants achieve substantial (23 percentage point) success improvements over established baselines (Mercat et al., 21 Apr 2026).

6. Extensions, Best Practices, and Impact

VLA Foundry is explicitly designed for extensibility and robust deployment:

• Backbone Swaps: Any transformer or VLM backbone supported via a plug-in registry. E.g., Qwen3-VL, ViT-CLIP, PaLI-Gemma, DINOv2.
• Action Heads: Easily extensible to diffusion or autoregressive discretized policies.
• New Modalities: Depth and audio can be incorporated by extending preprocessing and batch collation modules.
• Simulator Improvements: Dockerization ensures consistent evaluation; real-time dashboards offer experiment management, streaming, and diagnostics.
• STEP Analysis Tools: Facilitate statistically rigorous, unbiased policy comparisons through incremental analysis and balanced aggregation.

Through transparent modularization, standardized recipes, and comprehensive benchmarking, the VLA Foundry platform becomes a reproducible “foundry” resource for future VLA research and deployment. Released weights, code, and documentation support full-cycle reproduction from data preprocessing to closed-loop evaluation, addressing long-standing barriers in embodied AI methodology (Mercat et al., 21 Apr 2026).