HTC-VLM: Hybrid Token Compression in VLMs

• HTC-VLM is a hybrid token compression framework that splits visual data into continuous details and discrete semantic anchors to balance efficiency and fidelity.
• It integrates ViT patch embeddings with MGVQ-derived symbolic anchors into a 580-token hybrid sequence, later compressed to a single latent token using a disentanglement mask.
• Empirical results show an 87.2% average retention across seven benchmarks, demonstrating its effectiveness compared to continuous-only and discrete-only approaches.

Searching arXiv for the term and nearby variants to disambiguate usage and ground the article in the relevant papers. Search query: "HTC-VLM" HTC-VLM denotes a hybrid token compression framework for vision-LLMs in which semantics and appearance are disentangled through dual channels and then compressed into a single latent visual token for downstream language modeling (Zhang et al., 9 Dec 2025). In the formulation introduced in "HybridToken-VLM: Hybrid Token Compression for Vision-LLMs," the system combines a continuous pathway based on ViT patches with a discrete pathway based on MGVQ-derived symbolic anchors, fuses them into a 580-token hybrid sequence, and compresses that sequence into a single \<voco> token through a disentanglement attention mask and bottleneck (Zhang et al., 9 Dec 2025). Within the same literature stream, the acronym is not entirely unique: "HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-LLMs" notes that HTDC is "also called HTC-VLM in some contexts," but that usage refers to a decoding-time hallucination-mitigation method rather than to hybrid token compression (Liu, 13 Apr 2026).

1. Terminological scope and problem setting

HTC-VLM addresses the efficiency–fidelity tension in VLM inference. The motivating premise is that feeding hundreds of visual patch tokens into an LLM incurs quadratic computational costs and strains both memory and context windows, while existing compression strategies tend to sacrifice either high-level semantics or fine-grained appearance (Zhang et al., 9 Dec 2025). The framework is therefore organized around a division of labor: a continuous channel preserves textures and poses, whereas a discrete channel supplies symbolic anchors for object identities and scene layout.

A common misconception is that HTC-VLM is simply a continuous-token bottleneck with an auxiliary quantizer. The reported design is more specific. The model is described as a hybrid framework that "disentangles semantics and appearance through dual channels," and the compression objective is implemented through an explicit bottleneck token rather than by naïve pooling or mean aggregation (Zhang et al., 9 Dec 2025). Another possible source of confusion is nomenclatural rather than architectural: HTDC, a training-free calibration scheme for hallucination mitigation, has been called HTC-VLM in some contexts, but it operates at decoding time and does not define the hybrid tokenization architecture discussed here (Liu, 13 Apr 2026).

2. Dual-channel visual representation

The continuous pathway begins from an input image $I$, exemplified as $336\times 336$, and extracts ViT-L/14 patch embeddings through a frozen CLIP encoder $\mathcal{E}_v$. There are $N=576$ patches, and a trainable linear projector $\mathcal{P}_v$ maps each patch feature into the LLM embedding space:

$$V=\{v_i\}_{i=1}^{576}=\mathcal{P}_v(\mathcal{E}_v(I)), \qquad v_i\in\mathbb{R}^{4096}.$$

This channel is designated as the carrier of fine-grained detail $D$, including textures and poses (Zhang et al., 9 Dec 2025).

The discrete pathway uses an off-the-shelf MGVQ tokenizer $\mathcal{Q}$ with codebook size $K=16{,}384$, number of groups $G=8$, and spatial downsampling factor $336\times 336$0. The tokenizer produces a discrete code embedding $336\times 336$1, flattened to $336\times 336$2. A lightweight 2-layer MLP projector $336\times 336$3 with GELU then maps $336\times 336$4 into four symbolic semantic anchors $336\times 336$5, each of dimension $336\times 336$6:

$336\times 336$7

with $336\times 336$8 and $336\times 336$9 (Zhang et al., 9 Dec 2025).

The formal interpretation given for these two channels is explicitly information-theoretic. The continuous stream is intended to keep $\mathcal{E}_v$0 large, whereas the discrete stream is intended to maximize $\mathcal{E}_v$1, where $\mathcal{E}_v$2 denotes high-level semantics. This suggests that the framework is designed not merely for compression efficiency but for a controlled allocation of representational capacity across semantic and appearance subspaces (Zhang et al., 9 Dec 2025).

3. Hybrid sequence formation and disentanglement mechanism

The hybrid sequence is formed by prepending the four discrete tokens to the 576 continuous patch tokens:

$\mathcal{E}_v$3

A learnable \<voco> token is inserted after $\mathcal{E}_v$4, followed by the text tokens $\mathcal{E}_v$5. The key structural device is the disentanglement attention mask $\mathcal{E}_v$6, which blocks raw visual tokens from attending to one another and prevents text tokens from attending directly to the raw visual sequence (Zhang et al., 9 Dec 2025).

The mask is defined as

$\mathcal{E}_v$7

The supplied interpretation is equally specific: the mask enforces "no visual–visual self-attention," ensures that "text cannot see raw patches," and makes text attend to \<voco> instead (Zhang et al., 9 Dec 2025). In effect, the visual stream is forced through a star-shaped bottleneck centered on the single latent token.

After one cross-modal LLM layer under this mask, the hidden state of \<voco> is a latent $\mathcal{E}_v$8. The representation is cast in a VAE-style ELBO:

$\mathcal{E}_v$9

with $N=576$0 (Zhang et al., 9 Dec 2025). The accompanying interpretation states that $N=576$1 "primes" $N=576$2 with semantics, letting $N=576$3 reserve residual capacity for fine details. This suggests a structured bottleneck rather than an undifferentiated low-rank compression layer.

4. Objective function and forward computation

The training objective is the standard next-token cross-entropy, written in the supplied formulation as

$N=576$4

with an approximate ELBO interpretation:

$N=576$5

The stated gradient-level consequence is that gradients with respect to $N=576$6 encourage semantic clustering, while gradients with respect to $N=576$7 refine details (Zhang et al., 9 Dec 2025).

The forward sequence construction is fixed and explicit. First the model computes continuous patches $N=576$8. It then computes discrete codes $N=576$9, projects them into $\mathcal{P}_v$0, concatenates these with the continuous tokens to form $\mathcal{P}_v$1, and appends \<voco> followed by text embeddings $\mathcal{P}_v$2. The full input becomes

$\mathcal{P}_v$3

and the LLM processes $\mathcal{P}_v$4 under the mask $\mathcal{P}_v$5 (Zhang et al., 9 Dec 2025).

In computational terms, the framework is presented as a single-token compression scheme with "single-token inference ($\mathcal{P}_v$6 attention cost)." That cost claim is tied to the final bottlenecked visual representation rather than to the initial raw hybrid sequence, which still contains 580 visual tokens prior to aggregation (Zhang et al., 9 Dec 2025).

5. Benchmark behavior, retention, and ablation findings

The principal empirical claim is that HTC-VLM achieves an average performance retention of $\mathcal{P}_v$7 across seven benchmarks—GQA, VQAv2, MMBench, MME, POPE, SEED-Bench, and ScienceQA-Image—while using a 580-to-1 compression ratio (Zhang et al., 9 Dec 2025). In the reported single-token comparison, the upper bound with 576 visual tokens attains GQA $\mathcal{P}_v$8, VQAv2 $\mathcal{P}_v$9, MMBench $$V=\{v_i\}_{i=1}^{576}=\mathcal{P}_v(\mathcal{E}_v(I)), \qquad v_i\in\mathbb{R}^{4096}.$$0, MME $$V=\{v_i\}_{i=1}^{576}=\mathcal{P}_v(\mathcal{E}_v(I)), \qquad v_i\in\mathbb{R}^{4096}.$$1, POPE $$V=\{v_i\}_{i=1}^{576}=\mathcal{P}_v(\mathcal{E}_v(I)), \qquad v_i\in\mathbb{R}^{4096}.$$2, SEED $$V=\{v_i\}_{i=1}^{576}=\mathcal{P}_v(\mathcal{E}_v(I)), \qquad v_i\in\mathbb{R}^{4096}.$$3, and ScienceQA-Image $$V=\{v_i\}_{i=1}^{576}=\mathcal{P}_v(\mathcal{E}_v(I)), \qquad v_i\in\mathbb{R}^{4096}.$$4. VoCo-LLaMA at one token attains $$V=\{v_i\}_{i=1}^{576}=\mathcal{P}_v(\mathcal{E}_v(I)), \qquad v_i\in\mathbb{R}^{4096}.$$5, $$V=\{v_i\}_{i=1}^{576}=\mathcal{P}_v(\mathcal{E}_v(I)), \qquad v_i\in\mathbb{R}^{4096}.$$6, $$V=\{v_i\}_{i=1}^{576}=\mathcal{P}_v(\mathcal{E}_v(I)), \qquad v_i\in\mathbb{R}^{4096}.$$7, $$V=\{v_i\}_{i=1}^{576}=\mathcal{P}_v(\mathcal{E}_v(I)), \qquad v_i\in\mathbb{R}^{4096}.$$8, $$V=\{v_i\}_{i=1}^{576}=\mathcal{P}_v(\mathcal{E}_v(I)), \qquad v_i\in\mathbb{R}^{4096}.$$9, $D$0, and $D$1, corresponding to $D$2 average retention. HTC-VLM at one token reports $D$3, $D$4, $D$5, $D$6, $D$7, $D$8, and $D$9, corresponding to $\mathcal{Q}$0 average retention (Zhang et al., 9 Dec 2025).

The ablation results are central to interpreting the architecture. A continuous-only $\mathcal{Q}$1 configuration yields $\mathcal{Q}$2 average retention, a discrete-only $\mathcal{Q}$3 configuration yields $\mathcal{Q}$4, and the hybrid $\mathcal{Q}$5 configuration yields $\mathcal{Q}$6 (Zhang et al., 9 Dec 2025). This directly counters the view that either channel alone is sufficient. The number of discrete tokens is also non-monotonic: $\mathcal{Q}$7 gives $\mathcal{Q}$8, $\mathcal{Q}$9 gives $K=16{,}384$0, $K=16{,}384$1 gives $K=16{,}384$2, and $K=16{,}384$3 falls to $K=16{,}384$4. Similarly, the reported fusion strategy favors pre-fusion at $K=16{,}384$5, whereas both post-fusion and mean-fusion produce $K=16{,}384$6 (Zhang et al., 9 Dec 2025).

The attention analysis provides a mechanistic explanation for these outcomes. Figure 1 is described as visualizing \<voco>'s attention over the four discrete tokens and the first twelve image patches for sixteen MME examples. In HTC-VLM, \<voco> "strongly attends to the 4 discrete anchors," far more than to any continuous patch, whereas in VoCo-LLaMA attention is diffuse across many patches and lacks a clear semantic guide (Zhang et al., 9 Dec 2025). The probing study is consistent with this hybrid account: for Detail (D-10), $K=16{,}384$7 reaches $K=16{,}384$8, $K=16{,}384$9 reaches $G=8$0, and $G=8$1 reaches $G=8$2; for Semantic (S-10), $G=8$3 and $G=8$4 each reach $G=8$5, while $G=8$6 reaches $G=8$7 (Zhang et al., 9 Dec 2025). The hybrid latent $G=8$8 therefore performs best on both tasks in the reported probe setting.

6. Related usage, limits, and open directions

The most important clarification in adjacent literature is that HTC-VLM is not uniformly used to denote a single method family. In "HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-LLMs," the authors state that HTDC is "also called HTC-VLM in some contexts" (Liu, 13 Apr 2026). That framework is a training-free decoding method that identifies layer-wise hesitation and activates calibration only at hesitation-prone decoding steps. When triggered, it contrasts the full branch with a visual-nullification probe and a semantic-nullification probe, and on the MME benchmark it reports a baseline cost of $G=8$9 ms/token versus $336\times 336$00 ms/token for HTDC at trigger rate $336\times 336$01, compared with $336\times 336$02 ms/token for always-on contrastive methods (Liu, 13 Apr 2026). The overlap in naming can therefore obscure a substantive distinction between architectural compression and decoding-time calibration.

Within the HybridToken-VLM line itself, the stated limitations are concrete. The MGVQ anchors are pre-trained rather than jointly learned; the design is intended for single images; and extension to multi-view or video, including "temporal anchors," remains open (Zhang et al., 9 Dec 2025). A plausible implication is that the current formulation optimizes a specific single-image bottlenecking regime rather than a general-purpose multimodal sequence compressor.

The broader significance of HTC-VLM lies in the claim that a minimalist hybrid design can resolve, at least partially, the efficiency–fidelity trade-off that constrains VLM deployment. The evidence supplied for that claim is the combination of a 580-to-1 compression ratio, $336\times 336$03 average retention across seven benchmarks, and attention patterns in which the compressed token prioritizes the discrete anchor (Zhang et al., 9 Dec 2025). In that sense, HTC-VLM occupies a distinct position within VLM compression research: it is neither purely continuous compression nor purely discrete tokenization, but a structured bottleneck whose behavior is guided by discrete semantic anchors and refined by continuous detail channels.