OneVision-Encoder: Unified Vision & Multimodal Encoding

• OneVision-Encoder is a unified architectural family for vision and multimodal representation that leverages codec-inspired sparsity and dynamic tokenization.
• It computes saliency using motion vectors and residuals, enabling efficient selection of high-information patches that reduce computational overhead.
• Its integration with large language models and unified 3D positional encoding enhances temporal grounding, spatial reasoning, and long-context perception.

A OneVision-Encoder is a unified architectural family for vision (and, in certain realizations, multimodal) representation, distinguished by codec-inspired sparsity, shared 3D positional encoding, and a tokenization pipeline that adapts dynamically to the structure and semantics of visual data. The defining principle is to minimize unnecessary computation on redundant spatial or temporal regions, instead allocating modeling capacity and token budget to high-entropy, event-bearing content. OneVision-Encoders enable direct, natively tokenized interactions between images, videos (including compressed bitstreams), and LLMs, facilitating high-fidelity temporal grounding, spatial reasoning, and long-context perception under fixed computation budgets. They are a foundational component across leading large multimodal models including LLaVA-OneVision and LLaVA-OneVision-2 (Tang et al., 9 Feb 2026, An et al., 25 May 2026), and are conceptually formalized and empirically validated in recent works arguing for information-theoretic alignment of vision architecture with codec principles.

1. Conceptual Foundations: Information-Theoretic and Codec Alignment

OneVision-Encoder design is rooted in the observation that real world video consists largely of highly redundant information, with temporal novelty and discriminative signal concentrated in sparse, high-entropy residuals—analogous to the partition of I-frames and P/B-frames in classical codecs. Per Shannon's theorem, the minimum data needed to encode a signal corresponds to its true entropy. Traditional vision models, which apply uniform computation across dense pixel grids regardless of underlying information content, are thus inherently inefficient for video (Tang et al., 9 Feb 2026).

Codec-aligned OneVision-Encoders address this by explicitly computing saliency per patch based on motion vectors and residual errors (often extracted directly from the video codec bitstream). Instead of modeling fixed, densely sampled patches across all frames, OV-Encoders select only a small, dynamically-determined subset of informative patches—frequently 3–25% of possible tokens—achieving compression ratios up to 96.9% over naïve alternatives at matched or superior accuracy (Tang et al., 9 Feb 2026, An et al., 25 May 2026).

2. Architectural Overview: Unified Spatiotemporal and Multimodal Encoding

The OneVision-Encoder comprises a natively sparse Vision Transformer backbone, windowed/local self-attention for scalability, group-aware visibility masks, and—at core—a tokenization front-end that adapts to input granularity and entropy (An et al., 25 May 2026, Tang et al., 9 Feb 2026, Li et al., 2024).

• Input Processing & Patchification: Each visual input (static image, video, or compressed stream) is partitioned into non-overlapping patches (commonly 16×16 or 32×32). While images and uniformly-sampled frames use dense or regularly downsampled tokens, codec-aligned video employs patch-wise saliency scores derived from motion vectors and residuals:

$s_\pi = \alpha M_\pi + \beta R_\pi$

where $M_\pi$ is average motion magnitude and $R_\pi$ is luma-residual energy within patch $\pi$.

• Sparse Token Selection: For P-frames, only the top $r$-fraction (e.g., 3.1%–25%) of highest-saliency patches are retained, drastically reducing token count without loss of key information (Tang et al., 9 Feb 2026).
• Native-Resolution ViT Backbone: Tokens (and, for compressed video, "canvases" or merged block patches) proceed through a stack of ViT blocks, each employing spatial windowed attention for tractable $O(Nw^2)$ per-layer complexity. Window size ($w$) is chosen to balance locality and efficiency (An et al., 25 May 2026).
• Group-Visible Attention: Each token is annotated with a group ID ($\kappa_u$) that restricts its self-attention to other tokens within its temporal group (e.g., GOP, sampled frame), supporting both dense and codec-stream scenarios.

3. Unified 3D RoPE: Positional Encoding across Modalities and Layouts

A key technical innovation is the use of a unified 3D Rotary Positional Embedding (RoPE), which encodes each token's spatial and temporal position—even across irregularly sampled or codec-selected layouts—into the model's attention mechanism (Bai et al., 2 May 2026, Tang et al., 9 Feb 2026, An et al., 25 May 2026). For any token $(t, h, w)$:

$$\theta_k(t, h', w') = \frac{t \cdot \omega_{k, t} + h' \cdot \omega_{k, h} + w' \cdot \omega_{k, w}}{10000^{2k/d}}$$

where $M_\pi$0 are remapped to guarantee modality separation (audio/visual) and $M_\pi$1 is the embedding dimension (Bai et al., 2 May 2026).

This approach:

• Enables seamless fusion of tokens from varying input types (images, sampled frames, codec-blocks, and audio/visual modalities).
• Supports irregular and sparse layouts, such as those resulting from codec-derived token selection.
• Prevents collisions in positional space across different modalities (audio, visual, continuous motion).

4. Codec-Stream Tokenization and Adaptive Grouping

The most advanced OneVision-Encoders (e.g., in LLaVA-OneVision-2 (An et al., 25 May 2026)) employ codec-stream tokenization, which leverages the compressed bit-cost stream of video to dynamically define both temporal groups (variable-length GOPs) and spatial token selection:

• Temporal Grouping: P/B-frames are clustered into groups by aggregate bit-cost (not fixed duration), with segment boundaries determined adaptively to concentrate token resources on content-rich intervals.
• Spatial Saliency & Packing: Within each temporal group, motion and residual statistics define a per-frame saliency map. Only the top scoring patch blocks (typically merged into "canvases") constitute the visual token set for that interval.
• Unified Token Interface: The encoder receives as input a 3-tuple $M_\pi$2—canvases, per-patch metadata, and group assignments—enabling native multi-view and long-context reasoning.

This codec-aligned strategy achieves extremely stable and efficient visual compression, with long-video token budgets maintaining fidelity and temporal grounding even for high-frequency motion regimes (An et al., 25 May 2026).

5. Training Objectives, Integration with LLMs, and Empirical Results

OneVision-Encoders are typically trained end-to-end under an autoregressive next-token objective in coordination with LLM decoders (An et al., 25 May 2026, Li et al., 2024):

$M_\pi$3

Some variants (notably in e-commerce and representation learning) deploy hierarchical residual quantization (VRQ), using contrastive, margin, commitment, and hierarchical consistency losses to produce discrete semantic IDs with strong recall and discriminative power (Zheng et al., 7 Oct 2025).

Integration with LLMs:

• Token embeddings from the OV-Encoder are projected into the LLM context space via a lightweight connector (typically 2-layer MLP), then concatenated with text tokens and consumed by standard transformer-decoder cross-attention (Li et al., 2024, An et al., 25 May 2026).
• Alignment to LLM embedding space is performed, sometimes via contrastive or instruction data (e.g., aligning OV-Encoder to Qwen3 LLM with a frozen backbone) (Tang et al., 9 Feb 2026).
• No additional vision-text fusion blocks are typically used: all fusion is handled via unified self/cross-attention, enabled by 3D RoPE and group visibility mechanisms.

Benchmarks and Efficiency:

• On JumpScore, a high-frequency temporal grounding benchmark, LLaVA-OneVision-2-8B achieves 74.9 mAP, surpassing Qwen3-VL-8B by +44.8 points.
• Across general video and spatial reasoning benchmarks, OneVision-Encoder models deliver consistent performance gains (e.g., +4.3 on video, +5.3 on spatial, and +15.6 J&F on tracking tasks under matched visual-token budgets) (An et al., 25 May 2026).
• Codec-aligned patch selection permits graceful degradation: 3.1% token budget achieves performance comparable to or exceeding baselines at full density (Tang et al., 9 Feb 2026).

6. Comparative Table: Principal OneVision-Encoder Variants

Paper/Model	Saliency/Compression Principle	3D Positional Encoding	Token Budget/Efficiency
(Tang et al., 9 Feb 2026) OV-Encoder	Codec-patchification (motion, res.)	3D RoPE (sparse, align)	3.1–25% tokens, up to 96.9% reduction
(An et al., 25 May 2026) LLaVA-OV-2	Codec-stream, bit-cost adaptive	Shared 3D RoPE	Stably compresses long video, adaptive tokenization
(Bai et al., 2 May 2026) Omni-Encoder	Uniform 25fps, token sparsification	3D RoPE (modal disamb.)	Tubelet downsampling/tokens selector
(Zheng et al., 7 Oct 2025) OneVision-VRQ	Residual quantization (multi-view)	Not emphasized	Dynamic pruning for 21% speedup
(Li et al., 2024) LLaVA-OneVision	CLIP/SigLIP dense	Learned 2D position	Standard ViT token budget, not codec-aligned

7. Implications, Limitations, and Future Directions

OneVision-Encoders instantiate the principle that architectural alignment to data structure (here, video codec statistics) affords not only greater efficiency but also improved accuracy and generalization, especially for long-horizon, high-frequency, and multimodal perception tasks (Tang et al., 9 Feb 2026, An et al., 25 May 2026, Bai et al., 2 May 2026). Their success demonstrates that:

• Dense uniform sampling is suboptimal for video; dynamic allocation of token budget sharply enhances both scalability and downstream reasoning.
• Unified 3D positional encodings, combined with group visibility, permit a single backbone to natively support dense images, sampled frames, variable-rate video, and compressed streams, while maintaining token ordering and event structure.
• Event-centric and semantic reasoning is achievable under tight compute constraints, enabling broad deployment in edge, real-time, and large-context LLM scenarios.

Plausible implications include broad adoption of codec-aligned patchification across vision-language architectures and increased research into self-organizing tokenization and semantic grouping for multimodal models.

A limitation is the reliance on codec-derived signals, which necessitates integration of video decoding infrastructure and careful tuning of saliency metrics. Additionally, some event types (e.g., extremely fine motion not captured by codec statistics) may benefit from hybrid sampling or auxiliary detection pipelines.

Future research is expected to focus on further generalization to fully open-set modalities (e.g., native audio–visual fusion as seen in Omni-Encoder (Bai et al., 2 May 2026)), more advanced causal compression strategies, and deeper synergy with sequence learners for world-modeling and long-range memory.

---

References:

• "OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence" (Tang et al., 9 Feb 2026)
• "LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence" (An et al., 25 May 2026)
• "OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder" (Bai et al., 2 May 2026)
• "OneVision: An End-to-End Generative Framework for Multi-view E-commerce Vision Search" (Zheng et al., 7 Oct 2025)
• "LLaVA-OneVision: Easy Visual Task Transfer" (Li et al., 2024)