CLIP Global Codebook: Semantic Structures

• CLIP Global Codebook is a framework that defines both explicit and implicit semantic prototypes derived from CLIP’s aligned image-text representations.
• It encompasses constructions such as dataset-global text embeddings, internal semantic bases in deep Vision Transformers, and per-image dynamic token pools for dense prediction.
• These approaches strike a balance between global semantic consistency and local contextual refinement, enhancing performance across various vision tasks.

“CLIP global codebook” denotes a family of constructions in which CLIP’s shared image–text space is used as a reusable semantic inventory for visual representation, matching, or compression. In the most explicit formulation, the codebook is a set of CLIP text embeddings $$\mathcal{C}=\{\boldsymbol{c}_1,\boldsymbol{c}_2,\ldots,\boldsymbol{c}_K\}$$ produced from class labels and reused as global prototypes for quantizing visual features. In broader usage, the same phrase refers to CLIP’s implicit global semantic structure: the global image–text manifold, global tokens and channel-wise bases in deep ViT layers, per-image global token pools used during sliding-window inference, and continuous latents trained to preserve CLIP semantics across tasks. The current literature therefore does not present a single canonical object called a “CLIP global codebook”; rather, it presents a spectrum of explicit and implicit prototype mechanisms built on CLIP’s aligned visual and textual spaces (Zhu et al., 24 Mar 2025, 2502.06818, Zeng et al., 27 Nov 2025, Yin et al., 8 Jan 2025, Lee et al., 24 Mar 2026, Zohra et al., 14 Dec 2025).

1. Conceptual scope

The literature suggests three recurring interpretations of a CLIP global codebook. First, it may be an explicit dataset-global prototype set in CLIP text space, with one shared prototype per semantic category. Second, it may denote implicit global semantic bases already present in CLIP, such as global tokens, dominant channels, or the shared global image–text manifold. Third, it may denote a dynamic per-image semantic memory, assembled from visual tokens across windows or regions and queried at inference time. These interpretations are compatible rather than mutually exclusive: each treats CLIP as providing a globally shared semantic coordinate system, but they differ in whether the codebook is discrete or continuous, fixed or query-conditioned, dataset-global or image-specific (Zhu et al., 24 Mar 2025, 2502.06818, Zeng et al., 27 Nov 2025, Lee et al., 24 Mar 2026).

Manifestation	Mechanism	Representative paper
Explicit dataset-global codebook	Text embeddings of class labels form $\mathcal{C}\in\mathbb{R}^{d\times K}$	VTD-CLIP (Zhu et al., 24 Mar 2025)
Implicit global semantic bases	Global tokens, channel bases, and global image–text manifold	GCLIP (2502.06818), HarmoCLIP (Zeng et al., 27 Nov 2025)
Dynamic per-image global pool	Key-value extension and proxy anchors across windows	GLA-CLIP (Lee et al., 24 Mar 2026)
Continuous CLIP-supervised latent code	Single latent $\hat{y}$ constrained by global, local, and instance CLIP losses	UG-ICM (Yin et al., 8 Jan 2025)
Hierarchical queryable codebook	Caption-, sentence-, and phrase-conditioned visual embeddings	$\beta$-CLIP (Zohra et al., 14 Dec 2025)

A central point across these variants is that “global” usually refers to shared semantic structure rather than universal coverage. VTD-CLIP states that its codebook is global across all videos within a dataset but not across datasets. GLA-CLIP builds a per-image global token pool across windows. HarmoCLIP treats the original CLIP global space as a manifold that must be preserved during local refinement. This suggests that globality in current usage is primarily a statement about scope of reuse and semantic consistency, not about a single universal ontology.

2. Explicit text-space codebooks

The clearest explicit formulation appears in VTD-CLIP, which “repurposes the frozen text encoder to construct a visual codebook from video class labels.” Given class labels $\boldsymbol{y}_k$, the codebook entries are

$$\boldsymbol{c}_k=\phi_t(\boldsymbol{y}_k;\theta_t),$$

and the codebook is

$$\mathcal{C}=\{\boldsymbol{c}_1,\boldsymbol{c}_2,\ldots,\boldsymbol{c}_K\}\in\mathbb{R}^{d\times K}.$$

The entries are prompt-structured: $$\boldsymbol{y}_k=[\boldsymbol{w}_1][\boldsymbol{w}_2]\cdots[\boldsymbol{w}_n][class_k],$$ where $\boldsymbol{w}_i$ are learnable text prompt tokens, with 16 prompt tokens in the reported setup. The codebook is “global across the dataset”: one vector per class, shared by all videos, and it lives in CLIP’s text embedding space while remaining aligned to CLIP’s visual space (Zhu et al., 24 Mar 2025).

VTD-CLIP operationalizes this codebook through hard assignment of frame embeddings to text prototypes. For frame features $\boldsymbol{x}_t$, it computes the cross-modal affinity matrix

$\mathcal{C}\in\mathbb{R}^{d\times K}$0

followed by

$\mathcal{C}\in\mathbb{R}^{d\times K}$1

Each frame is thereby mapped to a discrete textual token, either as the prototype index $\mathcal{C}\in\mathbb{R}^{d\times K}$2 or as the embedding $\mathcal{C}\in\mathbb{R}^{d\times K}$3. A video-level discrete prototype is then chosen by bag-of-prototypes voting,

$\mathcal{C}\in\mathbb{R}^{d\times K}$4

and fused with frame features through cross-attention and confidence-aware weighted pooling. The same global codebook is reused for discretization and for classification, and the training objective is a CLIP-style cross-entropy loss aligning the final video representation $\mathcal{C}\in\mathbb{R}^{d\times K}$5 with the correct text embedding $\mathcal{C}\in\mathbb{R}^{d\times K}$6 (Zhu et al., 24 Mar 2025).

This construction is explicit, discrete, and semantically interpretable. VTD-CLIP also contrasts it with VQ-VAE-style quantization: the codebook size $\mathcal{C}\in\mathbb{R}^{d\times K}$7 is fixed by the number of semantic categories, the entries are anchored to language, and there is no extra VQ-specific commitment loss or codebook usage penalty. Prompt tuning makes the codebook adaptive without unfreezing the encoders. In the 4-shot HMDB-51 / UCF-101 ablation, “Learnable prompt + {class}” yields 67.6 / 87.6, compared with 65.8 / 84.8 for “a photo of a ” + {class} and 63.9 / 81.5 for “{class} only,” indicating that adaptive prompt tuning improves alignment relative to a static text template (Zhu et al., 24 Mar 2025).

3. Implicit global codebooks inside CLIP

A different line of work treats CLIP’s internal representations themselves as a global codebook. GCLIP argues that CLIP’s patch-wise features in deep layers contain “global knowledge”: deeper blocks exhibit attention patterns in which many patches attend strongly to a few “summary” patches or to the [CLS] token, and late-layer patch tokens become dominated by image-level properties rather than fine local details. The method identifies “global tokens” as patches whose attention-map columns are bright across almost all rows, and defines the first global-token-emerging block $\mathcal{C}\in\mathbb{R}^{d\times K}$8 through a product criterion over the Query–Key attention map. These global tokens are described as image-level semantic hubs aligned with [CLS]-like semantics (2502.06818).

GCLIP makes this implicit codebook usable for training-free open-vocabulary semantic segmentation through two operations. Attention Map Fusion (AMF) mixes Query–Key maps from the global-token-emerging blocks with the last-block Query–Query attention: $\mathcal{C}\in\mathbb{R}^{d\times K}$9 This preserves local distinctness while reintroducing global context. Channel Suppression (CS) addresses “abnormal channels” in the FFN second linear layer, where one channel norm becomes abnormally large from a certain block onward. For the dominant channel $\hat{y}$0, GCLIP rescales its weight vector to the average norm of the remaining channels: $\hat{y}$1 The interpretation offered in the paper is that Value channels then become a more balanced set of semantic bases rather than being dominated by one global channel (2502.06818).

In this formulation, the codebook is not a learned list of discrete entries. Instead, the “global codebook” is CLIP’s latent basis structure: global tokens define image-level prototypes, and Value channels define semantic bases from which patch-level representations are composed. Quantitatively, GCLIP improves mIoU over ClearCLIP on all five reported benchmarks: PASCAL VOC 80.9 $\hat{y}$2 81.3, PASCAL Context 35.9 $\hat{y}$3 37.0, ADE20K 16.7 $\hat{y}$4 18.3, Cityscapes 30.0 $\hat{y}$5 32.8, and COCO Stuff 23.9 $\hat{y}$6 24.7. The ablations show complementary gains from AMF and CS, and the best setting fuses the first and second global-token blocks ($\hat{y}$7) (2502.06818).

This usage clarifies that a CLIP global codebook need not be explicit or discrete. It may instead be a structured set of emergent global relations and semantic bases already embedded in the transformer, which can be selectively reweighted to recover local discriminability without erasing global context.

4. Preserving the global manifold while adding local structure

HarmoCLIP and $\hat{y}$8-CLIP address a related but distinct problem: how to enrich CLIP with region- or phrase-level semantics without damaging the original global image–text geometry. HarmoCLIP explicitly formalizes three spaces—Image Global Space $\hat{y}$9, Image Region Space $\beta$0, and Text Global Space $\beta$1—and argues that prior region-aware methods improve $\beta$2 only indirectly by forcing $\beta$3 toward $\beta$4, thereby distorting the original $\beta$5 manifold. Its remedy is a three-term objective,

$\beta$6

where $\beta$7 is the standard global contrastive loss, $\beta$8 is a lexeme–region contrastive loss aligning region embeddings $\beta$9 with token-level text embeddings $\boldsymbol{y}_k$0, and $\boldsymbol{y}_k$1 aligns trainable region features to frozen CLIP region embeddings produced from cropped images. In effect, the original global manifold is treated as the codebook anchor, while local spaces are added in a way that does not collapse regions onto the same global points (Zeng et al., 27 Nov 2025).

The empirical motivation is explicit. On MSCOCO 5K retrieval, HarmoCLIP reports Recall@1 of 69.78 for image-to-text and 53.44 for text-to-image; on Flickr30K it reports 90.80 and 76.46. For zero-shot bounding-box classification with a ViT-B/16 backbone, it reports 43.2 on OVCOCO Top-1 and 22.2 on LVIS Top-1, while the abstract highlights a 3.2% Top-1 improvement on the region task of bounding-box classification. The paper’s framing is that local alignment must be added through parallel local spaces, not by bending the global codebook until region features collapse onto it (Zeng et al., 27 Nov 2025).

$\boldsymbol{y}_k$2-CLIP generalizes the same concern into a hierarchical alignment framework. It keeps CLIP’s global CLS/caption pathway, adds text-conditioned cross-attention pooling from caption-, sentence-, and phrase-level queries to image patches, and trains the resulting embeddings with the $\boldsymbol{y}_k$3-Contextualized Contrastive Alignment Loss ($\boldsymbol{y}_k$4-CAL). For a batch with $\boldsymbol{y}_k$5 images and $\boldsymbol{y}_k$6 queries per image, the CE variant assigns weight 1 to exact matches, weight $\boldsymbol{y}_k$7 to off-diagonal intra-image pairs, and 0 to cross-image pairs: $\boldsymbol{y}_k$8

This produces a shared space in which caption-, sentence-, and phrase-conditioned visual embeddings form a contextual but non-collapsed hierarchy. Reported results include 91.8% T2I and 92.3% I2T at R@1 on Urban1K, and 30.9% on FG-OVD (Hard), with the CE variant excelling on fine-grained retrieval and the BCE variant favoring long-text/global retrieval (Zohra et al., 14 Dec 2025).

Taken together, these methods suggest a refined meaning for “global codebook.” It is not only a set of prototypes; it is also the geometry of the shared global manifold. Under that reading, the major technical challenge is not merely to add local semantics, but to do so without destroying the global semantic arrangement that gives CLIP its zero-shot behavior.

5. Per-image global token pools in dense prediction

GLA-CLIP shifts the codebook notion from dataset-level or model-internal structure to a dynamic per-image construction. In training-free open-vocabulary semantic segmentation, high-resolution images exceeding CLIP’s 224×224 resolution are commonly processed by sliding-window inference. GLA-CLIP identifies “semantic discrepancy across windows” as the central failure mode of this strategy: each window is processed independently, so different parts of the same object can receive inconsistent labels and grid artifacts appear at window borders. The method addresses this with three components: Key-Value Extension (KVE), a Proxy Anchor, and Dynamic Norm (Lee et al., 24 Mar 2026).

KVE extends keys and values so that a query patch is not limited to tokens inside its own window but can attend to tokens from all windows. The Proxy Anchor is “constructed by aggregating tokens highly similar to the given query from all windows,” providing a unified semantic reference for both inner-window and outer-window patches. Dynamic Norm then adjusts attention strength according to object scale by dynamically scaling and thresholding the attention map. The paper also introduces BER (Boundary Error Rate) to capture sliding-window grid artifacts that mIoU overlooks (Lee et al., 24 Mar 2026).

Under a codebook interpretation, the all-window token pool is a per-image global memory, and the proxy anchor is a query-conditioned global prototype extracted from that memory. The reported ablations support the separation of these roles. On ProxyCLIP-based experiments, KVE with Fixed Norm gives 38.1 mIoU, KVE with Dynamic Norm gives 43.1, Proxy with Fixed Norm gives 42.9, Proxy with Dynamic Norm gives 43.0, KVE+Proxy with Fixed Norm gives 39.5, and KVE+Proxy with Dynamic Norm gives 44.0. On VOC21, ProxyCLIP reports 63.3 mIoU while the proposed method reports 66.3; on Context59, ProxyCLIP reports 38.8 while the proposed method reaches up to 39.9 under the listed inference settings (Lee et al., 24 Mar 2026).

A plausible implication is that a CLIP global codebook can be assembled at inference time rather than learned once during training. GLA-CLIP’s construction is global only within an image, but it shows that training-free dense prediction can benefit from global semantic prototypes derived directly from distributed CLIP patch tokens.

6. Continuous CLIP-supervised codes, misconceptions, and open questions

UG-ICM extends the codebook idea beyond recognition into learned compression. It trains an encoder to produce a single latent representation

$\boldsymbol{y}_k$9

which is entropy-coded into one bitstream and decoded either as a human-preferred reconstruction

$$\boldsymbol{c}_k=\phi_t(\boldsymbol{y}_k;\theta_t),$$0

or as a machine-preferred reconstruction

$$\boldsymbol{c}_k=\phi_t(\boldsymbol{y}_k;\theta_t),$$1

CLIP is used as a frozen supervision signal through a multi-scale loss composed of global, local, and instance terms: $$\boldsymbol{c}_k=\phi_t(\boldsymbol{y}_k;\theta_t),$$2 The resulting latent is not a VQ-style discrete codebook, but the paper explicitly describes it as a CLIP-informed global semantic code intended to generalize across unseen tasks. On MLIC++, the reported average gains over the backbone LIC are +2.8% for classification, +3.9% for object detection, +6.0% for semantic segmentation, and +3.2% for instance segmentation, while maintaining comparable PSNR, SSIM, and LPIPS for the human-preference decoding (Yin et al., 8 Jan 2025).

A common misconception is that a CLIP global codebook must be a universal, fixed, discrete dictionary. The surveyed work argues otherwise. VTD-CLIP’s codebook is explicit and discrete, but only global per dataset. GCLIP’s “global codebook” is implicit in global tokens and channel-wise bases. HarmoCLIP treats the original global manifold as the structure to preserve, not as a list of codewords. GLA-CLIP builds a per-image global token pool and proxy anchors at inference time. UG-ICM uses a continuous latent, and $$\boldsymbol{c}_k=\phi_t(\boldsymbol{y}_k;\theta_t),$$3-CLIP builds a hierarchical, queryable set of contextualized embeddings without introducing discrete entries (Zhu et al., 24 Mar 2025, 2502.06818, Zeng et al., 27 Nov 2025, Lee et al., 24 Mar 2026, Zohra et al., 14 Dec 2025).

A second misconception is that stronger local modeling necessarily requires suppressing globality. GCLIP explicitly criticizes methods that over-localize CLIP by removing global context; HarmoCLIP shows that naïve region alignment can distort the global image–text geometry; and GLA-CLIP shows that training-free dense prediction suffers when windows cannot exchange global information. Across these papers, the recurrent design problem is global–local balance rather than simple replacement of one by the other (2502.06818, Zeng et al., 27 Nov 2025, Lee et al., 24 Mar 2026).

The main unresolved issue is universality. VTD-CLIP states that its codebook is task-specific and “not yet a universal multimedia codebook.” GCLIP notes architecture dependence and heuristic choices for detecting global tokens and selecting channel-suppression blocks. HarmoCLIP depends on captions plus instance annotations and grounds noun-level tokens rather than adjectives or relations. $$\boldsymbol{c}_k=\phi_t(\boldsymbol{y}_k;\theta_t),$$4-CLIP remains continuous and query-conditioned rather than explicitly discrete. This suggests that a fully general CLIP global codebook would likely require a larger ontology, multi-granular codes, and mechanisms for preserving the original CLIP manifold while supporting task-specific adaptation (Zhu et al., 24 Mar 2025, 2502.06818, Zeng et al., 27 Nov 2025, Zohra et al., 14 Dec 2025).