Foundry-VLA-1.7B Transformer Architecture

• Foundry-VLA-1.7B is a multi-modal transformer-based architecture integrating vision, language, and action domains through a novel fusion mechanism.
• Its architecture comprises approximately 1.85 billion parameters spread across a CLIP-style vision encoder, a 24-layer LLM, and a dedicated action head for policy learning.
• Designed for end-to-end training, it employs innovations like pixel-shuffle pooling and proprioceptive normalization to optimize robotics and embodied AI tasks.

Foundry-VLA-1.7B is a multi-modal transformer-based architecture described in VLA Foundry, an open-source framework for training Vision-Language-Action (VLA) models in robotics and embodied AI. Designed for end-to-end learning across language, visual input, and action domains, Foundry-VLA-1.7B is constructed from modular Transformer components—including a CLIP-style vision backbone, an LLM-style multimodal fusion stack, and a dedicated action (“flow”) Transformer—yielding a unified and flexible structure driven by recent advances in scalable learning and policy optimization (Mercat et al., 21 Apr 2026).

1. Architectural Composition and Parameterization

Foundry-VLA-1.7B is subdivided into three principal transformers—vision, language, and action—integrated through a novel multimodal fusion mechanism. The total architecture comprises approximately 1.85 billion parameters (with around 1.7B excluding embeddings), distributed as follows:

Module	Depth (Blocks)	Hidden Dim	Heads	MLP Dim	Param Count (B)
Vision Encoder (ViT-B/16 style)	12	768	12	3,072	≈0.09
Language Backbone (LLM)	24	2,048	16	8,192	≈1.23
Action Head (“Flow Transformer”)	7–8*	2,048	16	8,192	≈0.33
Embeddings Total (all domains)	(—)	(—)	(—)	(—)	≈0.20

** Depth adjusted so the flow head accounts for ≃325M parameters—suggests 7–8 transformer layers of 2,048 hidden size.

Key equations defining internal operations include the multi-head self-attention (MHA), two-layer MLP (FFN), and pre-norm LayerNorm:

• $$\mathrm{MHA}(X) = \mathrm{Concat}(h_1,\dots,h_H)\,W^O, \quad h_i = \mathrm{softmax}\left(\frac{XW^Q_i (XW^K_i)^T}{\sqrt{d_k}}\right)\,XW^V_i$$
• $$\mathrm{FFN}(x) = W_2\,\mathrm{GELU}(W_1\,x+b_1)+b_2$$
• $$\mathrm{LN}(x) = \gamma \frac{x-\mu}{\sigma} + \beta$$

2. Vision Backbone and Visual Tokenization

The vision encoder is a 12-layer, “CLIP-style” ViT-B/16 transformer operating on 224×224 RGB images:

• The input is patch-embedded with 16×16 pixel patches, yielding 196 image tokens per view.
• A pixel-shuffle pooling operation (“patch patch unshuffle”) reduces these to 64 tokens per image, each linearly projected to 768 dimensions.
• These tokens traverse 12 transformer encoder blocks comprised of MHSA (12 heads, 768/12=64 per head) and feed-forward sublayers with 3,072-dimensional expansion.

This reduction enables downstream multimodal processing by limiting sequence length without significant spatial information loss.

3. Multimodal Fusion and Language Processing

Multimodal integration is performed by pooling visual tokens from all camera or temporal perspectives (e.g., $N = 8$, resulting in $8 \times 64 = 512$ image tokens) and appending them to a textual prompt. The prompt is tokenized and augmented by the special <image> token at the beginning and <obs> at the end.

All image and text tokens (plus <obs>) are projected to a 2,048-dimensional embedding and processed together via a single, 24-layer Transformer with causal self-attention and MLP blocks (16 heads, 8,192 expansion). Notably, there are no explicit cross-attention layers; fusion is achieved within each LLM layer by self-attention over the concatenated multimodal sequence. This direct interleaving allows vision-language relationship modeling in every block, with the entire multimodal context available to the language backbone at all depths.

4. Action Head and Temporal Flow Transformer

The “flow Transformer” serves as the model’s action head, generating action trajectories for robotic policy execution:

• The input sequence consists of:
  • The last-layer hidden state of the <obs> token from each of the final four layers of the 24-block multimodal LLM (yielding $4 \times 2,048$ tokens).
  • Optionally, past proprioceptive tokens (windowed and percentile-normalized).
  • A chunked, noisy action sequence, linearly embedded to match internal dimensions.
• This sequence is processed by a standalone transformer (7–8 layers, 2,048 hidden size, 16 heads), outputting a denoising vector per action timestep.
• The model is trained with the standard flow-matching objective:

$$\mathcal{L}(\theta) = \mathbb{E}_{t, x_0, \epsilon} \left\| \epsilon - \epsilon_\theta(x_t, t) \right\|^2$$

A diffusion sampler then reconstructs the entire action trajectory from the Transformer’s denoised predictions.

5. Modality-Specific Innovations and Preprocessing

Several architectural and workflow adaptations target VLA-specific challenges:

• Pixel-shuffle pooling is applied pre-transformer to compress vision tokens, reducing sequence length and computational cost while preserving local detail.
• The <obs> token is used as a global latched context embedding, whose representations from the last four LLM layers form the “state” input for decision making.
• Robotics-oriented preprocessing includes percentile-based proprioceptive normalizers, SE(3) 6D rotation encoding, and windows for both past and future action sequences. These operations act before the data reaches the transformer stack; there are no additional architectural adapters or gating mechanisms within core transformer blocks.

6. Pipeline Integration and Training Flow

Foundry-VLA-1.7B is designed for streamlined end-to-end training that unifies language, vision, and action learning in one codebase. The model can be trained:

• Entirely from scratch via a staged LLM→VLM→VLA pipeline.
• By incorporating pretrained backbones (e.g., Qwen3-VL) at any point, allowing flexible initialization.
• With transparent control over data preprocessing, backbone selection, and fine-tuning, suitable for both open research and reproducibility in robotics and multimodal learning.

All components are open-source (https://github.com/TRI-ML/vla_foundry), and multi-task model weights are available (https://huggingface.co/collections/TRI-ML/vla-foundry).

7. Summary Table: Layerwise Model Structure

Component	Input	Blocks	Output
Vision Stem	224×224 RGB image	12×[MHSA(768,12)→MLP(3072)]	64 tokens @768 (per image)
Multimodal LLM	[N×64] vision tokens + text + <obs>	24×[MHSA(2,048,16)→MLP(8,192)]	Hidden-states for tokens, use final 4 <obs> for action
Flow Transformer	4 <obs> + proprio + noisy actions	7–8×[MHSA(2,048,16)→MLP(8,192)]	Denoising vector per action token

All architectural components and workflow details directly correspond to the framework and model specification given in (Mercat et al., 21 Apr 2026). This design enables research into closed-loop vision-language-action policy learning, multimodal representation, and robotics application evaluation in open-access simulation environments.