WAVECLIP: Adaptive Wavelet Tokenization in CLIP

• The paper introduces a CLIP-derived model that replaces fixed patch embeddings with progressive wavelet-based tokenization for adaptive-resolution inference.
• It employs a multiresolution 2D discrete wavelet transform and causal cross-level attention with key-value caching, enabling confidence-based early exits to reduce compute.
• Using lightweight distillation from a frozen CLIP teacher, WAVECLIP achieves comparable zero-shot performance while offering flexible compute-accuracy trade-offs.

Searching arXiv for the WAVECLIP paper and closely related CLIP work to ground the article. arXiv search: query "WAVECLIP (Kimhi et al., 25 Sep 2025)" WAVECLIP is a CLIP-derived vision model for adaptive-resolution inference that replaces standard patch embeddings with wavelet-based tokenization and processes images progressively from coarse to fine. Rather than committing to a fixed token budget at full resolution, it begins with low-resolution tokens and refines only when needed, using key-value caching and causal cross-level attention to reuse computation. The model is presented as a single unified alternative to standard CLIP deployment: it preserves the standard CLIP text tower and standard ViT image encoder, requires only lightweight distillation from a frozen CLIP teacher, and exposes a runtime compute–accuracy trade-off through confidence-based early exits (Kimhi et al., 25 Sep 2025).

1. Problem setting and design objective

Standard CLIP ViTs are strong for zero-shot classification, but their cost is high because self-attention scales quadratically with the number of image tokens. For a $224\times224$ image with $16\times16$ patches, the image encoder processes 197 tokens including $[CLS]$, which is expensive. WAVECLIP is introduced to address two deployment constraints: fixed-resolution inference and the need for multiple models for different compute budgets (Kimhi et al., 25 Sep 2025).

In the fixed-resolution regime, standard CLIP always processes the full token set and cannot naturally spend less compute on easy inputs and more on hard ones. Prior efficient CLIP variants are described as usually training separate architectures for different compute budgets. WAVECLIP instead offers one model whose inference cost can be adjusted dynamically. The central architectural move is therefore not a redesign of the entire CLIP system, but a replacement of static patch tokenization with a progressive tokenization scheme.

This design implies a change in the role of tokenization itself. In standard CLIP, tokenization is a preprocessing step that produces a fixed set of tokens. In WAVECLIP, tokenization becomes a computational schedule: coarse information is processed first, and finer detail is introduced only when confidence is insufficient. A plausible implication is that the main systems contribution lies as much in scheduling inference as in modifying representation formation.

2. Wavelet tokenization and multiresolution representation

WAVECLIP replaces patchify with an $L$-level 2D discrete wavelet transform applied independently to Y, Cb, and Cr channels after RGB-to-YCbCr conversion. The decomposition is written as

$$\{(\mathrm{LL}^{(\ell,c)},\mathrm{LH}^{(\ell,c)},\mathrm{HL}^{(\ell,c)},\mathrm{HH}^{(\ell,c)})\}_{\ell=1}^{L}, \quad c \in \{Y,\mathrm{Cb},\mathrm{Cr}\}.$$

Here, $\mathrm{LL}^{(L,c)}$ is the coarsest approximation, while $\mathrm{LH}$, $\mathrm{HL}$, and $\mathrm{HH}$ provide detail information at each scale (Kimhi et al., 25 Sep 2025).

Tokens are formed by patchifying each subband into non-overlapping $P\times P$ patches and then ordering them coarse-to-fine. The model starts with only the coarsest tokens and appends finer detail tokens only if confidence is insufficient. The hierarchy is therefore dyadic and additive: the $16\times16$0 band contains a coarse summary of the image, and the detail bands add increasingly fine structure. Because each refinement step introduces only new information, the active token count is reduced early in inference.

The paper contrasts this design with prior wavelet tokenization work that keeps token count fixed by concatenating subbands along channels. WAVECLIP instead reduces active token count early in inference. For $16\times16$1, $16\times16$2, and $16\times16$3, the progressive token counts are reported as 4 tokens at the first stage, 14 at the second, 52 at the third, and 200 at the fourth. This numerical schedule is the basis of adaptive compute.

The multiresolution property of wavelets is operational rather than merely representational in this setting. Wavelet subbands do not simply encode frequency content; they define the stages by which the model can move from a coarse approximation to full-resolution processing. This suggests that WAVECLIP treats the image encoder as an anytime system whose stopping point is determined during inference rather than fixed at training time.

3. Progressive inference, causal cross-level attention, and early exit

WAVECLIP keeps the standard CLIP text tower and the standard ViT image encoder intact. The student image encoder consists of the wavelet-based tokenizer, the unchanged ViT backbone, and a progressive inference mechanism with coarse-to-fine token refinement, cross-level causal attention, key-value caching, and confidence-based gating for early exit (Kimhi et al., 25 Sep 2025).

Inference proceeds in stages. The model starts with $16\times16$4 coarse tokens, runs the transformer and classifies, exits early if confidence is high enough, and otherwise appends finer detail tokens, reuses cached computation, and refines the prediction. This allows easy images to terminate after inexpensive computation while preserving access to full detail for harder cases.

A technical challenge is progressive token addition without recomputing everything. WAVECLIP addresses this with a block-lower-triangular cross-level attention mask: tokens introduced at level $16\times16$5 may attend to tokens from levels $16\times16$6, while later tokens cannot change earlier computations. This preserves causality across levels. The use of cached key/value states means that only new tokens incur most of the extra cost, rather than forcing repeated processing of earlier tokens.

Classification at each level uses cosine similarity between normalized visual and text embeddings,

$16\times16$7

followed by a margin-based confidence gate,

$16\times16$8

The paper states that margin gating is more stable across datasets than thresholding top-1 probability because probability thresholds often require dataset-specific calibration. The threshold $16\times16$9 is the main inference-time control knob: a higher threshold leads to more refinement, higher accuracy, and more compute, whereas a lower threshold leads to earlier exits, lower compute, and potentially lower accuracy. The threshold is chosen relative to the number of classes, for example $[CLS]$0.

The stated deployment settings include mobile devices, low-battery modes, and dynamic runtime budgets. In those settings, WAVECLIP’s contribution is that the deployed artifact remains a single model rather than a family of separately trained operating points.

4. Distillation procedure and preservation of CLIP alignment

WAVECLIP does not require full CLIP-scale image-text pretraining. Its training setup is teacher–student distillation with a frozen CLIP ViT-B/16 teacher and a student image tower using the wavelet tokenizer (Kimhi et al., 25 Sep 2025).

The student’s $[CLS]$1 embedding at each level is aligned to the teacher embedding using cosine distance. This is described as a short distillation procedure that preserves zero-shot alignment while avoiding expensive multimodal pretraining. The text tower is unchanged, and the image-side modification is localized to tokenization and progressive inference. The system is therefore framed as a drop-in alternative to patch tokenization rather than a new CLIP architecture trained from scratch.

This training strategy matters conceptually because WAVECLIP inherits the teacher’s zero-shot semantic geometry instead of relearning it. A plausible implication is that the method’s empirical objective is efficiency-preserving adaptation of an existing CLIP model rather than re-optimization of the joint image–text space. That framing also explains why the paper emphasizes lightweight distillation and frozen-teacher alignment rather than large-scale contrastive training.

5. Empirical results and compute–accuracy frontier

The reported evaluation is zero-shot ImageNet-1k classification with standard CLIP prompts at input resolution $[CLS]$2. The primary metrics are top-1 accuracy, image-encoder GFLOPs, and average visual tokens processed. The main claim is that WAVECLIP yields a smooth accuracy–compute frontier by varying the gating threshold (Kimhi et al., 25 Sep 2025).

Method	GFLOPs	Top-1 IN1k
CLIP ViT-B/16	16.87	66.3
WAVECLIP-low	6.22	62.6
WAVECLIP-mid	7.8	64.2
WAVECLIP-high	14.03	66.3

At the high operating point, WAVECLIP-high matches standard CLIP ViT-B/16 accuracy at 66.3% while reducing image-encoder cost from 16.87 GFLOPs to 14.03 GFLOPs, which is described as about 16.8% compute reduction. Another reported point is 11.7 GFLOPs with $[CLS]$3, giving 66.12% top-1 accuracy and about 30.6% compute reduction. The low and mid settings show larger savings with moderate accuracy loss: 62.6% at 6.22 GFLOPs and 64.2% at 7.8 GFLOPs.

The comparison set includes CLIP ViT-B/16, OpenCLIP ViT-B/32, TinyCLIP, MobileCLIP, and MobileCLIP2. The distinction emphasized for WAVECLIP is not merely lower cost at one point, but the existence of multiple compute points from one model while keeping the CLIP architecture largely intact.

The paper also isolates the benefit of key-value caching. Naive progressive re-encoding incurs additional overhead of $[CLS]$4, $[CLS]$5, $[CLS]$6, and $[CLS]$7 GFLOPs across refinement levels, reaching up to about 20% overhead at the deepest point. This supports the claim that cross-level causality and key-value reuse are crucial: without them, progressive token addition would be much less efficient.

6. Scope, distinctions, and interpretive boundaries

WAVECLIP is specifically a visual adaptive-resolution CLIP model; it should not be conflated with Wav2CLIP, which is a teacher-student distillation method for learning audio representations from CLIP. Wav2CLIP freezes CLIP’s image encoder and trains an audio encoder to predict the same CLIP image embeddings from audio alone, thereby projecting audio into CLIP’s pre-existing image-text space and enabling zero-shot classification, cross-modal retrieval, and audio-guided image generation (Wu et al., 2021). WAVECLIP, by contrast, keeps the standard CLIP text tower and standard ViT image encoder intact and focuses on adaptive visual tokenization and inference cost control.

Several boundaries of interpretation are explicit in the reported results. First, the compute–accuracy trade-off is governed by the margin threshold $[CLS]$8: higher thresholds produce more refinement and more compute, whereas lower thresholds produce earlier exits and potentially lower accuracy. Second, the efficiency gains depend materially on key-value caching and causal cross-level attention; naive progressive re-encoding adds nontrivial overhead. Third, the reported experiments are on zero-shot ImageNet-1k classification at $[CLS]$9, so the evidence base is centered on adaptive visual inference rather than a wider range of downstream tasks.

The paper states that WAVECLIP would be preferred in resource-constrained or variable-budget settings and is likely most beneficial for higher-resolution inputs where progressive refinement can yield larger compute savings. This suggests a deployment-oriented interpretation of the method: it is meant to preserve CLIP-like zero-shot behavior while converting resolution into an adaptive resource rather than a fixed design choice.