InfiniteVL: Unbounded Vision-Language Modeling

• InfiniteVL is a framework for handling unbounded video and document streams using hybrid sparse and linear attention along with continuous long-term memory.
• It employs methodologies like Sliding Window Attention and Gated DeltaNet to overcome quadratic complexity and ensure real-time, scalable inference.
• Applications include infinite-context video QA and streaming multimodal understanding, achieving competitive performance with efficient memory and runtime.

InfiniteVL, or Infinite Vision-Language Modeling, refers to a collection of architectures and methodologies enabling vision-LLMs to process and comprehend arbitrarily long, potentially infinite streams of multimodal data—primarily videos and documents—while maintaining high performance, low memory footprint, and real-time inference capabilities. These systems synergize advances in sparse and linear attention, persistent memory, hierarchical representation, and efficient training paradigms to overcome the classical bottlenecks of quadratic complexity, degraded recall, and catastrophic loss of temporal context.

1. Motivation and Conceptual Scope

The key challenge addressed by InfiniteVL is the effective long-range modeling of video (and related multimodal) data where sequence length vastly exceeds the static context window of conventional Transformer models. At typical framerates, one minute of video can produce over a million visual tokens—far beyond the tractable context of standard attention-based VLMs. The ambition is to equip models with persistent long-term memory, scalable attention mechanisms, and streaming inference such that they can continuously ingest, encode, and reason about unbounded visual/textual streams spanning hours to days (Zhang et al., 11 Jul 2025, Tao et al., 9 Dec 2025).

InfiniteVL, as a term, encompasses hybrid architectures such as $\infty$-Video and InfiniteVL models; it also connects to the broader research program of infinite video understanding.

2. Hybrid Sparse and Linear Attention Mechanisms

InfiniteVL architectures decompose each decoder block into a synergistic combination of Sliding Window Attention (SWA) and Gated DeltaNet layers (Tao et al., 9 Dec 2025).

• SWA restricts queries to a fixed local window $w$ around each token, i.e., the attention span is $w \ll L$ with $L$ total sequence length, yielding $O(L w d)$ time and $O(L w)$ space complexity compared to $O(L^2 d)$ and $O(L^2)$ for full attention.

$$\mathrm{SWA}(Q,K,V)_t = \sum_{i=t-\lfloor w/2 \rfloor}^{t+\lfloor w/2 \rfloor} \mathrm{Softmax}\left(\frac{Q_t K_i^T}{\sqrt{d}}\right) V_i$$

• Gated DeltaNet is a state-space, linear attention block maintaining a constant-sized memory $S_t \in \mathbb{R}^{d \times d}$ per step, updated via gated recurrence:

$$S_t = S_{t-1} [\alpha_t (I-\beta_t k_t k_t^T)] + \beta_t v_t k_t^T$$

where $\alpha_t$ and $\beta_t$ are learnable gates, and $k_t$, $v_t$ are projected input features.

This hybrid enables constant memory for history while SWA preserves local fidelity, yielding the ability to sustain arbitrarily long context windows and streaming input.

3. Memory Consolidation and Long-Term State

Infinity-scale models rely on persistent memory mechanisms to retain, compress, and adaptively allocate representational capacity over time (Santos et al., 31 Jan 2025, Tao et al., 9 Dec 2025).

• Continuous-Time Long-Term Memory (LTM): As in $\infty$-Video, video embeddings from each chunk are compressed into a continuous function $x(t) = B^T \psi(t)$; memory contraction ($\tau < 1$) and sticky sampling prioritize segments with high historical attention, mirroring biological consolidation.
• Aggregation: Output tokens are aggregated via running averages so only a constant-sized semantic summary is passed to downstream modules.
• Cache Efficiency: Gated DeltaNet compresses global memory into a fixed-size state, and streaming architectures recycle key-value caches to avoid quadratic growth.

4. Training Strategies and Data Efficiency

Modern InfiniteVL models employ a multi-stage training strategy (Tao et al., 9 Dec 2025):

1. Distillation Pretraining: Aligns linear-space layers to full-attention (Transformer) teachers using a combination of layerwise MSE and end-to-end KL divergence losses.
2. Instruction Tuning (SFT): Teaches complex instruction following over diverse multimodal prompts.
3. Long-Sequence Supervised Fine-Tuning: Activates infinite-context capabilities using LoRA adapters, with context lengths up to $32768$ tokens and video QA pairs from large-scale datasets.

Total data required for competitive performance is less than $2\%$ of that used by leading Transformer VLMs—a substantial efficiency improvement.

5. Empirical Results and Benchmarking

InfiniteVL systems achieve near-Transformer performance while maintaining constant runtime and memory, evidenced on standard benchmarks (Tao et al., 9 Dec 2025):

Benchmark	InfiniteVL-4B	Cobra-3B (linear)	Qwen2.5-VL-3B (Transformer)
MME	2126	1346	2171
MMStar (%)	55.6	34.7	54.3
DocVQA/TextVQA/OCRBench	$\approx$91.7/78.5/79.8	lower	$\approx$equal

Long-sequence inference shows $3.6\times$ speedup over FlashAttention-2 baseline and stability at $24$ FPS for endless streaming video understanding.

6. Architectural Variants and Related Systems

Other notable infinite-scale video systems include:

• $\infty$-Video: A training-free wrapper that integrates continuous LTM mechanisms with frozen video Q-formers, enabling arbitrarily long-context video QA improvements and adaptive granularity allocation (Santos et al., 31 Jan 2025).
• StreamingVLM: Employs a fixed-size streaming key-value cache and contiguous RoPE for stable, low-latency, infinite-stream inference aligned by custom SFT strategy (Xu et al., 10 Oct 2025).
• Stable Video Infinity: Production-scale autoregressive video diffusion generator employing error-recycling fine-tuning for infinite-duration generation with robust scene coherence (Li et al., 10 Oct 2025).

7. Limitations, Open Problems, and Future Research Directions

Current infinite VL architectures must address trade-offs in granularity, forgetting factors, window size, and computational efficiency (Santos et al., 31 Jan 2025, Zhang et al., 11 Jul 2025). Integrals and regressions require nontrivial approximation (e.g., trapezoidal rule), and adaptive scheduling of memory hyperparameters is an open research direction.

Future work is anticipated on:

• Learned scheduling of temporal compression and sticky histograms.
• Integration with schema-driven symbolic memory and offline replay.
• Multilingual, multi-domain streaming datasets.
• Reinforcement learning from human feedback and adaptive attention gating.

InfiniteVL marks a paradigm shift toward unbounded, real-time, persistent multimodal reasoning, spanning both practical deployments and foundational research in memory, efficiency, and representational dynamics.