--- title: 'xGen-MM-Vid: Efficient Video Token Compression' url: https://www.emergentmind.com/topics/xgen-mm-vid-blip-3-video type: topic --- # xGen-MM-Vid: Efficient Video Token Compression xGen-MM-Vid (BLIP-3-Video) is a multimodal language model (VLM) designed for efficient video understanding by compressing temporal and spatial visual information into extremely compact token representations. The architecture introduces a temporal encoder that, together with a visual tokenizer, enables the mapping of entire video clips to as few as 32 visual tokens. This schema allows the model, with only 4B parameters, to achieve video question-answering and captioning performance on par with substantially larger models while using orders of magnitude fewer visual tokens [2410.16267]. ## 1. Visual Tokenization and Frame Encoding The foundation of xGen-MM-Vid’s video pipeline is its frame-wise visual tokenizer. For each raw frame $x_t \in \mathbb{R}^{H \times W \times 3}$ (where $H=W=384$), the image is partitioned into non-overlapping $16 \times 16$ patches, yielding $\ell=(H/p) \cdot (W/p)=24 \cdot 24=576$ patches per frame. Each patch $x_{t,i} \in \mathbb{R}^{16 \times 16 \times 3}$ is linearly embedded: $$ z_{t,i} = W_e \cdot \text{vec}(x_{t,i}) + b_e, \quad z_{t,i} \in \mathbb{R}^{1152} $$ A Perceiver-Resampler, functioning as a light-weight cross-attention module, reduces the set $\{z_{t,1},...,z_{t,576}\}$ to $N=128$ tokens per frame, compactly represented as $V_t \in \mathbb{R}^{128 \times d}$. For $T=8$ uniformly sampled frames, the tokenized video is concatenated as $V = [V_1;...;V_8] \in \mathbb{R}^{1024 \times d}$. ## 2. Temporal Encoder Mechanisms To obtain a highly compressed global representation of the entire video, xGen-MM-Vid employs either: **a) Learnable Spatio-Temporal Pooling (TokenLearner):** A pooling function $A: \mathbb{R}^{d \times K} \rightarrow \mathbb{R}^{M \times K}$ (for $K=1024$ input tokens and $M$ output tokens) is implemented as an MLP over the input token matrix. The learned attention matrix $A(V)$ is produced as $A(V)_{ij} = \text{softmax}(\alpha(V^T))_{ij}$ with $\alpha(V^T) = \text{MLP}_m(V^T) \in \mathbb{R}^{M \times K}$. The final $M$ spatio-temporal tokens are obtained by $X = A(V) \cdot V$, i.e., as a weighted sum that “soft-selects” informative tokens across frames and space. **b) Sequential Token Turing Machine (TTM):** A memory state $M_{t-1} \in \mathbb{R}^{G \cdot N \times d}$ maintains temporal context by recurrently integrating each frame’s tokens $V_t$. The memory updates are performed by a small 4-layer Transformer processor $f_\theta$, encompassing TokenLearner read/write modules: $$ M_t = f_\theta(M_{t-1}, V_t) $$ After processing all frames, the final memory $M_T \in \mathbb{R}^{512 \times d}$ is pooled again by a TokenLearner module to obtain $M$ global tokens, $X = A_{seq}(M_T) \cdot M_T$, with typical $M = 32$. ## 3. Token Compression Pipeline The token reduction is carried out in two abstraction stages. First, from $8 \times 729 = 5832$ raw patch embeddings per video, the Perceiver compresses to $8 \times 128 = 1024$ frame tokens. Next, either TokenLearner or TTM with pooling produces $M$ compressed visual tokens—commonly $M=32$—by applying a learned attention mechanism $A \in \mathbb{R}^{32 \times 1024}$ such that $X = A \cdot V$. Thus, the architecture achieves an extreme compression ratio, distilling thousands of patch features into a succinct set of global video tokens. ## 4. Pretraining and Instructional Tuning BLIP-3-Video’s training regime has three stages, each employing the standard next-token cross-entropy loss over the language model’s text outputs: **Stage 1: Image-Caption Pretraining** - Model initialized from BLIP-3; SigLIP and Perceiver weights are frozen; temporal encoder initialized randomly. **Stage 2: Video-Caption Pretraining** - 900K video-caption pairs from LLaVA-Hound-DPO, captions rephrased by GPT-4. - Loss: $\mathcal{L}_{cap} = - \sum_{i=1}^L \log P(w_i|w_{