--- title: Foundry-VLA-1.7B Transformer Architecture url: https://www.emergentmind.com/topics/foundry-vla-1-7b-architecture type: topic --- # Foundry-VLA-1.7B Transformer Architecture Foundry-VLA-1.7B is a multi-modal transformer-based architecture described in VLA Foundry, an open-source framework for training Vision-Language-Action (VLA) models in robotics and embodied AI. Designed for end-to-end learning across language, visual input, and action domains, Foundry-VLA-1.7B is constructed from modular Transformer components—including a CLIP-style vision backbone, an LLM-style multimodal fusion stack, and a dedicated action (“flow”) Transformer—yielding a unified and flexible structure driven by recent advances in scalable learning and policy optimization [2604.19728]. ## 1. Architectural Composition and Parameterization Foundry-VLA-1.7B is subdivided into three principal transformers—vision, language, and action—integrated through a novel multimodal fusion mechanism. The total architecture comprises approximately 1.85 billion parameters (with around 1.7B excluding embeddings), distributed as follows: | Module | Depth (Blocks) | Hidden Dim | Heads | MLP Dim | Param Count (B) | |------------------------------- |-------------- |----------- |------ |-------- |-----------------| | Vision Encoder (ViT-B/16 style)| 12 | 768 | 12 | 3,072 | ≈0.09 | | Language Backbone (LLM) | 24 | 2,048 | 16 | 8,192 | ≈1.23 | | Action Head (“Flow Transformer”)| 7–8* | 2,048 | 16 | 8,192 | ≈0.33 | | Embeddings Total (all domains) | (—) | (—) | (—) | (—) | ≈0.20 | *\*** Depth adjusted so the flow head accounts for ≃325M parameters—suggests 7–8 transformer layers of 2,048 hidden size. Key equations defining internal operations include the multi-head self-attention (MHA), two-layer MLP (FFN), and pre-norm LayerNorm: - $$ \mathrm{MHA}(X) = \mathrm{Concat}(h_1,\dots,h_H)\,W^O, \quad h_i = \mathrm{softmax}\left(\frac{XW^Q_i (XW^K_i)^T}{\sqrt{d_k}}\right)\,XW^V_i $$ - $$ \mathrm{FFN}(x) = W_2\,\mathrm{GELU}(W_1\,x+b_1)+b_2 $$ - $$ \mathrm{LN}(x) = \gamma \frac{x-\mu}{\sigma} + \beta $$ ## 2. Vision Backbone and Visual Tokenization The vision encoder is a 12-layer, “CLIP-style” ViT-B/16 transformer operating on 224×224 RGB images: - The input is patch-embedded with 16×16 pixel patches, yielding 196 image tokens per view. - A pixel-shuffle pooling operation (“patch patch unshuffle”) reduces these to 64 tokens per image, each linearly projected to 768 dimensions. - These tokens traverse 12 transformer encoder blocks comprised of MHSA (12 heads, 768/12=64 per head) and feed-forward sublayers with 3,072-dimensional expansion. This reduction enables downstream multimodal processing by limiting sequence length without significant spatial information loss. ## 3. Multimodal Fusion and Language Processing Multimodal integration is performed by pooling visual tokens from all camera or temporal perspectives (e.g., $N = 8$, resulting in $8 \times 64 = 512$ image tokens) and appending them to a textual prompt. The prompt is tokenized and augmented by the special `` token at the beginning and `` at the end. All image and text tokens (plus ``) are projected to a 2,048-dimensional embedding and processed together via a single, 24-layer Transformer with causal self-attention and MLP blocks (16 heads, 8,192 expansion). Notably, there are no explicit cross-attention layers; fusion is achieved within each LLM layer by self-attention over the concatenated multimodal sequence. This direct interleaving allows vision-language relationship modeling in every block, with the entire multimodal context available to the language backbone at all depths. ## 4. Action Head and Temporal Flow Transformer The “flow Transformer” serves as the model’s action head, generating action trajectories for robotic policy execution: - The input sequence consists of: - The last-layer hidden state of the `` token from each of the final four layers of the 24-block multimodal LLM (yielding $4 \times 2,048$ tokens). - Optionally, past proprioceptive tokens (windowed and percentile-normalized). - A chunked, noisy action sequence, linearly embedded to match internal dimensions. - This sequence is processed by a standalone transformer (7–8 layers, 2,048 hidden size, 16 heads), outputting a denoising vector per action timestep. - The model is trained with the standard flow-matching objective: $$ \mathcal{L}(\theta) = \mathbb{E}_{t, x_0, \epsilon} \left\| \epsilon - \epsilon_\theta(x_t, t) \right\|^2 $$ A diffusion sampler then reconstructs the entire action trajectory from the Transformer’s denoised predictions. ## 5. Modality-Specific Innovations and Preprocessing Several architectural and workflow adaptations target VLA-specific challenges: - Pixel-shuffle pooling is applied pre-transformer to compress vision tokens, reducing sequence length and computational cost while preserving local detail. - The `` token is used as a global latched context embedding, whose representations from the last four LLM layers form the “state” input for decision making. - Robotics-oriented preprocessing includes percentile-based proprioceptive normalizers, SE(3) 6D rotation encoding, and windows for both past and future action sequences. These operations act before the data reaches the transformer stack; there are no additional architectural adapters or gating mechanisms within core transformer blocks. ## 6. Pipeline Integration and Training Flow Foundry-VLA-1.7B is designed for streamlined end-to-end training that unifies language, vision, and action learning in one codebase. The model can be trained: - Entirely from scratch via a staged LLM→VLM→VLA pipeline. - By incorporating pretrained backbones (e.g., Qwen3-VL) at any point, allowing flexible initialization. - With transparent control over data preprocessing, backbone selection, and fine-tuning, suitable for both open research and reproducibility in robotics and multimodal learning. All components are open-source (https://github.com/TRI-ML/vla_foundry), and multi-task model weights are available (https://huggingface.co/collections/TRI-ML/vla-foundry). ## 7. Summary Table: Layerwise Model Structure | Component | Input | Blocks | Output | |------------------|---------------------|------------------------------------|---------------------------------------------------| | Vision Stem | 224×224 RGB image | 12×[MHSA(768,12)→MLP(3072)] | 64 tokens @768 (per image) | | Multimodal LLM | [N×64] vision tokens + text + `` | 24×[MHSA(2,048,16)→MLP(8,192)] | Hidden-states for tokens, use final 4 `` for action | | Flow Transformer | 4 `` + proprio + noisy actions | 7–8×[MHSA(2,048,16)→MLP(8,192)] | Denoising vector per action token | All architectural components and workflow details directly correspond to the framework and model specification given in [2604.19728]. This design enables research into closed-loop vision-language-action policy learning, multimodal representation, and robotics application evaluation in open-access simulation environments. Source: https://www.emergentmind.com/topics/foundry-vla-1-7b-architecture