VLA Foundry: A Unified Framework for Training Vision-Language-Action Models

Abstract: We present VLA Foundry, an open-source framework that unifies LLM, VLM, and VLA training in a single codebase. Most open-source VLA efforts specialize on the action training stage, often stitching together incompatible pretraining pipelines. VLA Foundry instead provides a shared training stack with end-to-end control, from language pretraining to action-expert fine-tuning. VLA Foundry supports both from-scratch training and pretrained backbones from Hugging Face. To demonstrate the utility of our framework, we train and release two types of models: the first trained fully from scratch through our LLM-->VLM-->VLA pipeline and the second built on the pretrained Qwen3-VL backbone. We evaluate closed-loop policy performance of both models on LBM Eval, an open-data, open-source simulator. We also contribute usability improvements to the simulator and the STEP analysis tools for easier public use. In the nominal evaluation setting, our fully-open from-scratch model is on par with our prior closed-source work and substituting in the Qwen3-VL backbone leads to a strong multi-task table top manipulation policy outperforming our baseline by a wide margin. The VLA Foundry codebase is available at https://github.com/TRI-ML/vla_foundry and all multi-task model weights are released on https://huggingface.co/collections/TRI-ML/vla-foundry. Additional qualitative videos are available on the project website https://tri-ml.github.io/vla_foundry.

• The paper introduces a unified, configuration-driven framework that consolidates LLM, VLM, and VLA training pipelines for embodied AI.
• The methodology employs modular YAML configurations, pluggable registries, and scalable distributed training to achieve high throughput and reproducibility.
• Benchmark evaluations reveal that backbone quality and multi-task finetuning are critical for enhanced robotics manipulation and simulation performance.

VLA Foundry: A Unified Framework for LLM-VLM-VLA Training

Motivation and Context

Vision-language-action (VLA) models represent a critical frontier in robotics foundation modeling, bridging perception, language, and control for embodied intelligence. Prevailing open-source VLA efforts have been bottlenecked by fragmented, stage-specific pipelines and lack of holistic infrastructure for full-stack multimodal training. The VLA Foundry framework directly addresses these limitations by establishing a unified, configuration-driven codebase for seamless LLM, VLM, and VLA training. This architectural consolidation enables end-to-end control, composable experimentation, and reproducible results for researchers tackling data heterogeneity, backbone pretraining, and cross-modal fusion challenges.

Figure 1: VLA Foundry provides a unified pipeline for LLM, VLM, and VLA model training with flexible model composition and backbone swapping, natively supporting pretrained models from Hugging Face.

Architectural Principles and Framework Design

VLA Foundry is constructed around a modular YAML-based configuration system, pluggable model and data registries, modality-aware preprocessing, and a training loop agnostic to model stage (LLM, VLM, VLA). This design is informed by four principles:

• Modularity & Composability: Experiments are conducted via composable presets, enabling rapid swapping of model architectures, vision encoders, and data pipelines through simple configuration changes.
• Hackability & Interoperability: Minimal reliance on heavy frameworks (e.g., PyTorch Lightning, Hugging Face Trainer), with explicit exposure of parallelism primitives for extensibility.
• Performance: Training throughput is benchmarked to scale across up to 128 GPUs, supporting both DDP and FSDP parallelization modes.
• Reproducibility: Deterministic RNG seeding, dataloader state checkpointing, and frozen dataclasses ensure exact restart and prevent runtime config drift.

Scalable distributed training incorporates standard levers: FSDP (with optional CPU offloading), mixed precision, gradient checkpointing, torch.compile, and gradient accumulation. Dataset mixing and probabilistic dataloader sampling are readily achieved through weighted dataset lists in configuration. Preprocessing leverages ray for parallelization, with output in WebDataset tar shards facilitating extensibility across text, caption, and robotics modalities.

Figure 2: Training throughput scaling for LLM, VLM, and VLA stages across DDP and FSDP parallelization on multi-node H100 setups.

Robotics Data Handling and VLA Architecture

Robotics data preprocessing in VLA Foundry is distinguished by robust normalization (global, per-timestep, percentile-based), absolute and relative pose representations, and action chunking across configurable past/future windows. Merged statistics across datasets employ t-digest for percentiles, and proprioceptive observations are causally restricted to history. Actions use 6D continuous rotations and SE(3) composition for relational pose encoding.

The VLA architecture extends a vision-language backbone by appending an observation token; the hidden states corresponding to this token over the last $N$ layers condition a flow-matching transformer action head (325M parameters). The action sequence is denoised via flow matching, concatenating vision, text, proprioception, and noised actions, paralleling recent advances in diffusion policies.

Figure 3: Foundry-VLA-1.7B architecture—eight images are encoded via ViT and pooled using pixel-shuffle, then integrated with task description and observation token. Flow transformer predicts robot actions.

Model Training and Released Architectures

Two major model series demonstrate framework capabilities:

• Foundry-VLA-1.7B: Full LLM$\rightarrow$VLM$\rightarrow$VLA pipeline trained from scratch, providing complete controllability.
• Foundry-Qwen3VLA-2.1B-MT: VLA model leveraging the pretrained Qwen3-VL backbone, showcasing plug-and-play backbone swapping and leveraging strong vision-language representations.

Intermediate Foundry-LLM-1.2B and Foundry-VLM-1.3B checkpoints are made available to facilitate pipeline reproducibility and modification.

LLM pretraining is conducted on DCLM (500M samples, 1T tokens), achieving moderate accuracy on commonsense and reasoning benchmarks, though lacking instruction tuning. VLM training employs a randomly initialized ViT (86M params) and the pre-trained LLM, trained on DataCompDR-1B. Captioning evaluation on COCO shows competitive performance for end-to-end trained backbones.

Figure 4: Qualitative image caption predictions from the VLM model, illustrating competence on normalized images including out-of-domain samples.

Benchmarking, Evaluation, and Statistical Analysis

Evaluation is performed on the LBM simulation benchmark (lbm), encompassing 49 tabletop bimanual manipulation tasks with high-fidelity Drake physics. Closed-loop policy performance is statistically analyzed using STEP, combining Bayesian success rate estimation and Compact Letter Display (CLD) for significance at 5% FWER.

Aggregate comparisons demonstrate:

• Foundry-Qwen3VLA-2.1B-MT consistently outperforms Foundry-VLA-1.7B-MT-sim and legacy closed-source LBM-MT by more than 20 percentage points in average task success.
• Foundry-VLA-1.7B-MT-sim maintains parity with closed-source LBM performance.
• Stronger backbones and multi-task training with finetuning maximize aggregate success, while real-only training shows near zero transfer to simulation due to distributional mismatch.

  Figure 5: Aggregate multi-task policy comparison—Foundry-Qwen3VLA-2.1B-MT achieves dominant performance over baseline and from-scratch models.

  

  Figure 6: Simulation results across variant multi-task models trained on different data splits, reinforcing the primacy of simulation data for simulated tasks.

Qualitative Simulation and Generalization

Qualitative rollouts confirm robust manipulation capabilities on diverse tasks, spanning pick-and-place, non-prehensile manipulation, and bimanual coordination. Models exhibit moderate zero-shot generalization, with finetuned multi-task models showing enhanced success across held-out tasks.

Figure 7: Representative successful simulation rollouts in tabletop manipulation tasks, evidencing multimodal policy integration.

Figure 8: Foundry-VLA-1.7B-MT-sim results on unseen tasks—demonstration of zero-shot generalization.

Practical and Theoretical Implications

VLA Foundry's unified pipeline radically lowers experimentation friction for robotics foundation model research. Practically, the framework supports rapid backbone swapping, flexible fusion architectures, efficient distributed training, and integrated evaluation—all interoperable with Hugging Face models and open benchmarks. Theoretically, the system underpins probing of data scaling biases, multi-modal data recipes, architectural fusions, and embodied transfer learning.

Strong numerical results validate the hypothesis that backbone quality directly cascades to action policy performance, while modular abstractions enable future integration of alternative action heads (diffusion, autoregressive, flow-matching), simulation platforms, and real-hardware deployment. Framework limitations in current scope (evaluation restricted to LBM simulation, single action head, incomplete data recipe exploration) and open directions are acknowledged: optimal pretraining regimens, safety/alignment in embodied agents, and scaling to broad embodiments remain fertile research terrain.

Conclusion

VLA Foundry operationalizes rigorous, fully unified LLM-VLM-VLA training for embodied AI, combining modularity, performance, and reproducibility in an open-source setting. Released model checkpoints, shared evaluation dashboards, and statistical comparison tools empower the community to dissect, benchmark, and advance the multimodal model design space. The demonstrable gains from backbone strength and pipeline integration signal further progress as model, data, and architecture scales continue to advance.