---
title: 'TinyGiantVLM: Compact RGB-D Spatial Reasoning'
url: https://www.emergentmind.com/topics/tinygiantvlm
type: topic
---

# TinyGiantVLM: Compact RGB-D Spatial Reasoning

TinyGiantVLM is a lightweight vision-language model architecture for physical spatial reasoning in warehouse-scale environments. It is presented as a modular two-stage framework that targets fine-grained spatial relationships in industrial logistics scenes, rather than traditional geographic reasoning. The model encodes both global and region-level features from RGB and depth modalities using pretrained visual backbones, integrates them with a T5-small language backbone, and is trained in two phases: first with free-form answers to enhance spatial reasoning ability, then with normalized answers for evaluation. On Track 3 of the AI City Challenge 2025, the 64M-parameter base model achieved 5th place on the leaderboard with a score of 66.8861; the paper also reports an 80M-parameter variant with expanded Mixture-of-Experts capacity and improved spatial reasoning performance [2508.17595].

## 1. Problem setting and intended scope

TinyGiantVLM is defined around warehouse spatial intelligence rather than around broad, open-domain multimodal generation. The paper motivates the model by noting that existing vision-language models often struggle to comprehend 3D layouts, object arrangements, and multimodal cues in real-world industrial settings. In this formulation, the target is not merely object recognition or captioning, but reasoning over physical relations among pallets, shelves, and other scene elements under resource constraints [2508.17595].

The benchmark setting is Track 3 of the AI City Challenge 2025, described as involving four spatial reasoning tasks: distance estimation, object counting, multiple-choice spatial grounding, and left/right spatial relation queries. This task design is significant because it constrains the model toward physically grounded answers rather than toward unconstrained visual description. A common misconception is to treat TinyGiantVLM as a generic multimodal chatbot; the reported system is instead specialized for structured warehouse reasoning workloads, with explicit support for region-level queries such as identifying which marked object is furthest right [2508.17595].

The paper also distinguishes this problem from traditional geographic reasoning. In the latter, an LLM may answer symbolic questions such as whether one city is north of another. TinyGiantVLM instead grounds language in visual perception and explicitly leverages RGB and depth cues to infer object relations inside a scene. This distinction suggests that the work belongs to the subarea of embodied or physically grounded multimodal reasoning rather than to text-only knowledge-based spatial inference [2508.17595].

## 2. Architectural organization

At the architectural level, TinyGiantVLM is a multimodal encoder-decoder system built on T5-small. Its visual front end consists of two pretrained vision transformers: CLIP-ViT for RGB images and DPT for depth images. The design is dual-branch and modular, extracting both global and region-level features from both modalities before fusing them for downstream reasoning [2508.17595].

The RGB backbone is specified as CLIP-ViT Large-Patch14, while the depth backbone is Intel DPT-Hybrid-MiDaS. The model uses the final layer’s $[\mathrm{CLS}]$ token from each modality as a global embedding, projects the RGB and depth features into a shared space, and concatenates them. The global fusion therefore preserves modality separation up to the projection stage and only merges the channels after backbone-specific encoding. This is consistent with the model’s emphasis on handling heterogeneous inputs without collapsing RGB and depth information too early [2508.17595].

The architecture also includes region-aware processing. Region-level features are extracted from patch embeddings, fused across RGB and depth, inserted into the language token stream at placeholder positions, contextualized by the T5 encoder, and then enriched through cross-attention with global scene features. An optional Mixture-of-Experts fusion module is placed after this contextualization stage. In the 80M variant, the MoE expands task specialization capacity; in the 64M base model, the architecture remains lighter and was the version used for the competition submission [2508.17595].

This modular decomposition is central to the system’s stated design objective: physical spatial reasoning under resource constraints. The paper describes the model as designed to run efficiently on a single P100 GPU with only 64–80M parameters. That parameter range is unusually small for a contemporary VLM and locates TinyGiantVLM within the compact-model regime rather than among large multimodal assistants [2508.17595].

## 3. RGB-D feature extraction and language integration

The global feature pathway uses the final layer’s class token from each visual backbone:
$$
\mathbf{f}^{\mathrm{RGB}} = \mathbf{e}^{\mathrm{RGB}}_{[\mathrm{CLS}]}, \qquad
\mathbf{f}^{\mathrm{Depth}} = \mathbf{e}^{\mathrm{Depth}}_{[\mathrm{CLS}]}
$$
These features are linearly projected and concatenated:
$$
\tilde{\mathbf{f}}^{\mathrm{RGB}} = W^{\mathrm{RGB}} \mathbf{f}^{\mathrm{RGB}} + \mathbf{b}^{\mathrm{RGB}}, \qquad
\tilde{\mathbf{f}}^{\mathrm{Depth}} = W^{\mathrm{Depth}} \mathbf{f}^{\mathrm{Depth}} + \mathbf{b}^{\mathrm{Depth}}
$$
$$
\mathbf{g} = [\tilde{\mathbf{f}}^{\mathrm{RGB}} \,\|\, \tilde{\mathbf{f}}^{\mathrm{Depth}}]
$$
The notation makes explicit that TinyGiantVLM does not rely on a single fused image representation from the outset; instead, it computes modality-specific summaries and combines them only after projection [2508.17595].

Region-level extraction is more granular. CLIP-ViT processes RGB images at $224 \times 224$ resolution and produces a $16 \times 16$ patch grid, corresponding to 256 regions. DPT processes depth at $384 \times 384$ and produces a $24 \times 24$ patch grid, corresponding to 576 regions. Region annotations are provided as segmentation masks in RLE format and are downsampled to the relevant patch resolution. For a given region, the feature is obtained by mean pooling the encoded patch embeddings inside the corresponding mask:
$$
\mathbf{f}_r^{\mathrm{RGB}} = \frac{1}{|P_r|} \sum_{i \in P_r} \mathbf{e}_i, \qquad
\mathbf{f}_r^{\mathrm{Depth}} = \frac{1}{|P'_r|} \sum_{i \in P'_r} \mathbf{e}'_i
$$
After projection and concatenation, a feed-forward network with ReLU produces the fused region representation:
$$
\mathbf{h}_j = [\tilde{\mathbf{f}}_j^{\mathrm{RGB}} \,\|\, \tilde{\mathbf{f}}_j^{\mathrm{Depth}}]
$$
$$
\mathbf{r}_j = \sigma\!\left(W_2\, \sigma(W_1\, \mathbf{h}_j + \mathbf{b}_1) + \mathbf{b}_2\right)
$$
All extracted features are precomputed and cached, a detail that directly supports the model’s efficiency objective [2508.17595].

Language integration is performed through explicit region injection. Questions contain special `<mask>` tokens for regions, such as “Which `<mask>` is furthest right?”. Each mask is replaced by a unique placeholder such as `<extra_id_0>` in the T5 tokenizer, and the embedding at that position is substituted with the corresponding region feature:
$$
\mathbf{e}_{p_j} \leftarrow \mathbf{r}_j
$$
The full sequence is then passed through the T5 encoder, allowing the model to contextualize language tokens together with region-specific visual embeddings. Cross-attention subsequently enriches the contextualized region embeddings using the fused global visual feature:
$$
\mathbf{Q} = \mathbf{R}^{\mathrm{ctx}} W^Q, \qquad
\mathbf{K} = \mathbf{g} W^K, \qquad
\mathbf{V} = \mathbf{g} W^V
$$
$$
\mathbf{C} = \operatorname{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d}}\right)\mathbf{V}
$$
This mechanism gives the region tokens access to scene-level context while preserving their localized grounding [2508.17595].

## 4. Mixture-of-Experts fusion and two-phase optimization

The MoE module is introduced to handle high-modality inputs and diverse question types. Its purpose is task-specific reasoning through dynamic expert selection, with sparse gating for efficiency. The paper describes the routing as Top-$K$ selection over $S$ experts using Laplace-based gating:
$$
h_\ell(\mathbf{c}_j) = \mathrm{Top\ K}(-\|W - \mathbf{c}_j\|_2)
$$
For each selected expert $i \in \mathcal{S}_j$, the gate weight is
$$
G(\mathbf{c}_j)_i =
\frac{\exp(-\|\mathbf{c}_j - W_i\|_2)}
{\sum_{l \in \mathcal{S}_j} \exp(-\|\mathbf{c}_j - W_l\|_2)}
$$
and the final output is
$$
\mathbf{z}_j = \sum_{i \in \mathcal{S}_j} G(\mathbf{c}_j)_i \cdot E_i(\mathbf{c}_j)
$$
The paper states that, in practice, $S=4$ with one expert per task and $k=2$ for sparsity. It further motivates Laplace, or distance-based, gating as avoiding inner-product norm bias and as being more robust to scale and input heterogeneity [2508.17595].

After MoE fusion, the final region representations are re-inserted at the appropriate encoder positions for decoding. The decoder then generates either free-form or normalized answers, depending on the training stage. The two-phase curriculum is central to the reported method. Phase 1 trains the model to produce open-ended, natural-language answers to spatial reasoning questions, with the stated goal of learning richer spatial reasoning priors and flexible output distributions. Phase 2 fine-tunes the model to produce canonical structured answers, such as numbers for counting or “left” and “right” for relational questions. The sequence loss is given as
$$
\mathcal{L} = - \sum_{t=1}^T \log p(y_t \mid y_{<t}, \mathbf{x})
$$
where $y_{1:T}$ is the answer sequence and $\mathbf{x}$ includes the question and image features [2508.17595].

The ablation study isolates the contributions of curriculum learning and MoE:

| Configuration | Score (%) |
|---|---:|
| No MoE, Phase 1 only | 25.59 |
| No MoE, Phase 2 only | 63.65 |
| No MoE, Phase 1 + Phase 2 | 65.09 |
| MoE, Phase 2 only | 68.13 |
| MoE, Phase 1 + Phase 2 | 72.52 |

These numbers establish two technical points. First, normalized-answer supervision alone is substantially stronger than free-form-only training for the target evaluation. Second, the best reported validation score in the ablation table arises when both the MoE and the two-phase curriculum are active. A further nuance is that the actual competition submissions used the non-MoE version due to implementation constraints, whereas post-hoc ablations confirmed the benefits of MoE [2508.17595].

## 5. Empirical results and reported limitations

The principal reported competition result is a public leaderboard score of 66.8861 by the 64M base model, corresponding to 5th place. The paper also describes an 80M-parameter variant with expanded MoE capacity and states that it demonstrates improved performance on spatial reasoning tasks. In the more detailed summary, the public leaderboard position is given as 5th place out of 9, and the parameter counts are given as approximately 64M for the non-MoE base model and approximately 80M for the MoE-enabled version [2508.17595].

The per-task breakdown reported for the best models is 83.87% for object counting, 98.40% for left–right relation, 50.26% for distance estimation, and 35.01% for multiple-choice grounding. These figures indicate that the model is strongest on directional relation and counting tasks, while distance estimation and multiple-choice grounding remain harder. This suggests that not all forms of warehouse spatial reasoning are equally tractable under a compact parameter budget, even when depth information is available [2508.17595].

A second reported nuance concerns validation versus competition performance. The ablation discussion states that activating MoE increases validation score from 65.09% to 72.52%, and that more epochs after the competition reached 88.52% accuracy on validation. A plausible implication is that the challenge submission underestimates the headroom of the architecture as presented in the full paper. At the same time, the paper does not claim that the competition leaderboard score itself was obtained with the final MoE-enabled configuration; the official result corresponds to the non-MoE system [2508.17595].

## 6. Relation to the compact VLM literature

TinyGiantVLM belongs to the broader literature on small vision-language models, or sVLMs, which the survey “Small Vision-Language Models: A Survey on Compact Architectures and Techniques” organizes into transformer-based, mamba-based, and hybrid architectures. That survey emphasizes efficiency techniques such as knowledge distillation, lightweight attention mechanisms, and modality pre-fusion, and frames the field around trade-offs between accuracy, efficiency, and scalability, as well as persistent challenges including data biases and generalization to complex tasks [2503.10665]. Within that landscape, TinyGiantVLM is notable for specializing a compact model toward RGB-D spatial intelligence rather than toward general-purpose multimodal dialogue.

A useful comparison is Vary-toy, which uses Qwen-1.8B as the base language model and introduces a reinforced vision vocabulary by replacing negative samples of natural images with positive object detection samples during vocabulary generation. Vary-toy reports 65.6% ANLS on DocVQA, 59.1% accuracy on ChartQA, 88.1% accuracy on RefCOCO, and 29% on MMVet, and is described as feasible to deploy on a consumer GPU such as a GTX 1080Ti [2401.12503]. The comparison is instructive because both systems target resource-constrained multimodal inference, but they do so through different design priorities: Vary-toy emphasizes a small 1.8B language backbone and reinforced visual tokenization for OCR, VQA, and localization, whereas TinyGiantVLM emphasizes a far smaller 64M–80M architecture, RGB-D fusion, region injection, and warehouse-scale spatial reasoning.

The broader compact-model literature also includes work asking how small a VLM can be while still producing coherent and consistent text. “NanoVLMs: How small can we go and still make coherent Vision Language Models?” introduces the ShortDesc and LongDesc datasets and reports that VLMs can be trained that are significantly smaller, up to 10 times smaller than state-of-the-art small VLMs, while maintaining architectural simplicity; it further describes GPT-4o-based grading on creativity, meaningfulness, and consistency [2502.07838]. In relation to that line of inquiry, TinyGiantVLM can be understood as a domain-specific answer to the same general pressure toward compactness: rather than optimizing for broad multimodal fluency, it targets physically grounded warehouse reasoning with a tightly constrained parameter budget and explicit RGB-D structure.

Source: https://www.emergentmind.com/topics/tinygiantvlm