Locked-Image Tuning (SigLiT) Overview

• The paper introduces a configuration where a frozen, pre-trained image encoder is paired with a trainable text tower, leading to superior zero-shot transfer and enhanced data efficiency.
• It replaces the traditional softmax-based contrastive loss with a pairwise sigmoid loss, enabling flexible batch-size scaling and reducing memory overhead.
• The method decouples vision representation from language alignment, allowing for multilingual extensions and robust performance on challenging image classification benchmarks.

Locked-Image Tuning (LiT) is a contrastive-tuning method for language–image models in which a pre-trained image encoder is kept frozen while a text encoder is trained to align with the image representation space. In the notation of the original work, LiT is the canonical Lu configuration: a locked, pre-trained image tower paired with a trainable, randomly initialized text tower (Zhai et al., 2021). The method was introduced as a modification of CLIP/ALIGN-style contrastive pre-training, with the central claim that preserving a strong vision backbone yields better zero-shot transfer than re-optimizing the image encoder on noisy web-scale image–text data (Zhai et al., 2021). In later usage, and explicitly in the SigLIP paper, “SigLiT” denotes the combination of this locked-image design with SigLIP’s pairwise sigmoid loss; the LiT paper itself does not formally define “SigLiT” as a separate method (Zhai et al., 2023).

1. Contrastive-tuning framework and the Lu regime

LiT was introduced within a broader contrastive-tuning framework in which each tower is specified by whether it is initialized from a pre-trained model and whether it is locked or trainable. The first letter denotes the image tower and the second denotes the text tower. L means locked and pre-trained, U means trainable and pre-trained, and u means trainable and randomly initialized (Zhai et al., 2021).

Config	Image tower	Text tower
uu	trainable, random init	trainable, random init
Uu	trainable, pre-trained	trainable, random init
LU	locked, pre-trained	trainable, pre-trained
Lu	locked, pre-trained	trainable, random init

The specific LiT configuration is Lu: the image tower is pre-trained and frozen, while the text tower and its projection are trained. The stated rationale is that pre-trained vision models already provide strong generic representations, whereas web-scale image–text data are useful for alignment to language but noisy for improving the underlying visual descriptor (Zhai et al., 2021). The paper reports that when the image tower is unlocked, training contrastive loss becomes lower on the image–text training set, but representation quality and out-of-distribution performance degrade; when the image tower is locked, training loss is worse, but representation quality and out-of-distribution performance are better (Zhai et al., 2021).

This design also decouples two problems that CLIP and ALIGN solve jointly: learning image representations and learning language alignment. LiT treats the former as largely solved by large-scale supervised or self-supervised vision pre-training, and the latter as a text-side readout problem over an existing visual space (Zhai et al., 2021). A plausible implication is that LiT is best viewed not as end-to-end multimodal pre-training, but as a modular retrofitting procedure for turning a vision backbone into a zero-shot image–text model.

2. Model architecture, embeddings, and training setup

The image tower in LiT can be any pre-trained vision model producing a fixed-dimensional embedding. The paper reports experiments with ResNet (BiT-M), Vision Transformers including ViT-B/16, ViT-B/32, ViT-L/16, and ViT-g/14, and MLP-Mixer, using both supervised and self-supervised pre-training regimes (Zhai et al., 2021). Writing the image encoder as $f_I(\cdot)$, an image $x$ is mapped to

$z_I = f_I(x)\in\mathbb{R}^{d_I}.$

If needed, a linear head maps to a shared space. In the canonical large-scale LiT setup, the image head is often omitted when dimensions already match, and the text head is retained as the trainable projection (Zhai et al., 2021).

The text tower is a Transformer-based encoder. Reported options include a ViT-style Transformer on token sequences, BERT-base, BERT-large, T5-base, and mT5-base, with SentencePiece used for ViT/T5/mT5 and WordPiece for BERT (Zhai et al., 2021). Given tokenized text $t$, the encoder $f_T$ produces

$z_T = f_T(t)\in\mathbb{R}^{d_T},$

followed by an optional projection

$$h_T(z_T)=W_T z_T,\quad W_T\in\mathbb{R}^{d\times d_T}.$$

For each image–text pair, LiT computes image and text embeddings, L2-normalizes them,

$$\tilde{u}_i = \frac{u_i}{\|u_i\|},\qquad \tilde{v}_i = \frac{v_i}{\|v_i\|},$$

and uses scaled cosine similarity

$$s_{ij} = \frac{\tilde{u}_i^\top \tilde{v}_j}{\tau},$$

with $\tau$ a learnable temperature parameter or fixed hyperparameter (Zhai et al., 2021). The original implementation uses a global contrastive loss across all devices, gathering embeddings from all accelerators to increase the effective number of negatives; an ablation in Appendix E is reported to show that this global loss consistently outperforms a per-device local loss (Zhai et al., 2021).

The large-scale runs used AdaFactor with $x$0, $x$1, $x$2, linear warmup for 10k steps, cosine decay, and batch sizes up to 32k, with a default training length of 55k steps and a best model trained for 550k steps (Zhai et al., 2021). Public-data runs on CC12M and YFCC100M used Adam with $x$3, $x$4, typically learning rate $x$5, decoupled weight decay $x$6, gradient clipping at global norm 1, and 20 epochs on CC12M with batch 16,384 (Zhai et al., 2021).

3. From LiT to SigLiT: softmax contrastive loss versus pairwise sigmoid loss

The original LiT objective is the symmetric CLIP-style InfoNCE loss. For a batch of $x$7 image–text pairs, with similarity matrix $x$8, the image-to-text and text-to-image losses are

$x$9

$z_I = f_I(x)\in\mathbb{R}^{d_I}.$0

and the total loss is

$z_I = f_I(x)\in\mathbb{R}^{d_I}.$1

This is a batch-level classification objective in which positives are the matched pairs and all mismatched in-batch pairs are negatives (Zhai et al., 2021).

SigLIP replaces this softmax-based loss with a pairwise sigmoid loss that operates on individual image–text pairs rather than on a batch-normalized softmax. With L2-normalized image embeddings $z_I = f_I(x)\in\mathbb{R}^{d_I}.$2, text embeddings $z_I = f_I(x)\in\mathbb{R}^{d_I}.$3, learnable temperature $z_I = f_I(x)\in\mathbb{R}^{d_I}.$4, learnable bias $z_I = f_I(x)\in\mathbb{R}^{d_I}.$5, and labels

$z_I = f_I(x)\in\mathbb{R}^{d_I}.$6

the logit is

$z_I = f_I(x)\in\mathbb{R}^{d_I}.$7

and the sigmoid loss is

$z_I = f_I(x)\in\mathbb{R}^{d_I}.$8

Because there is no batch-wise normalization, the loss does not require a global view of all pairwise similarities, which enables a chunked distributed implementation that keeps only a $z_I = f_I(x)\in\mathbb{R}^{d_I}.$9 submatrix in memory at a time and reduces memory from $t$0 to $t$1 (Zhai et al., 2023).

In the SigLIP paper, SigLiT = Sigmoid LiT = LiT + SigLIP-style sigmoid loss (Zhai et al., 2023). The image backbone is strictly locked, image embeddings are precomputed once, and only the text transformer plus the global temperature and bias are trainable. The reported initialization is

$t$2

with Scaling ViT–Adafactor, $t$3, learning rate $t$4, linear warmup over the first 200M examples, cosine decay, and weight decay $t$5 on the text tower (Zhai et al., 2023).

A common misconception is that SigLiT is a wholly separate family of models from LiT. The papers support a narrower interpretation: SigLiT is LiT under a different contrastive objective, retaining the locked-image design while replacing the softmax objective with the pairwise sigmoid formulation (Zhai et al., 2023).

4. Empirical results, transfer behavior, and scaling laws

LiT’s principal claim is strong zero-shot transfer from a locked image tower. With a transformer-based pre-trained ViT-g/14 model, the paper reports 85.2% zero-shot transfer accuracy on the ImageNet test set and 82.5% on the challenging out-of-distribution ObjectNet test set (Zhai et al., 2021). On additional zero-shot robustness benchmarks, the same model reaches 79.8 on ImageNet-v2, 94.9 on ImageNet-R, 81.8 on ImageNet-A, and 88.6 on ImageNet-ReaL (Zhai et al., 2021). The paper further states that ObjectNet zero-shot accuracy of 82.5% beats CLIP by 10.2% and outperforms a supervised ViT-g/14 fine-tuned on ImageNet, reported at 70.5% on ObjectNet (Zhai et al., 2021).

LiT also emphasizes data efficiency. The paper states that LiT reaches 81.7% ImageNet zero-shot accuracy with only 300M seen pairs, whereas CLIP needed 12.8B seen pairs to reach 76.2% (Zhai et al., 2021). Using only public components—ViT-L/16 from AugReg, trained on the union of YFCC100m-CLIP and CC12M—the reported ImageNet zero-shot accuracy is 75.7%, compared with 34.8% for the previous public-data best OpenCLIP and 31.3% for CLIP on the YFCC subset (Zhai et al., 2021).

SigLiT preserves most of LiT’s top-end performance while changing the objective and improving efficiency. The best SigLiT configuration reported in the SigLIP paper, using a B-sized text tower, a ViT-g image tower, 18B examples, and batch 32k, reaches 84.7% ImageNet zero-shot, compared with 85.2% for the original LiT using a much larger g-sized text model (Zhai et al., 2023). The paper also reports that with 4 TPUv4 chips, a frozen ViT-g/14 vision tower, batch size 20k, and 107k steps, SigLiT reaches 84.5% ImageNet zero-shot in less than two days; with a frozen ViT-AugReg-B/8 tower, batch size 32k, and 65k steps, it reaches 79.7–79.8% in about one day (Zhai et al., 2023).

The batch-size analysis in SigLIP is central to the meaning of SigLiT. The paper reports that sigmoid loss dominates softmax at small batch sizes, that the two are very close around 32k, and that benefits quickly diminish beyond that scale; it explicitly states that 32k is sufficient, even though the chunked implementation permits batch sizes up to 1,000,000 (Zhai et al., 2023). This suggests that SigLiT’s contribution is not only a new loss but also a practical decoupling of optimization quality from extremely large hardware budgets.

5. Multilingual extensions, scope, and practical limitations

The LiT paper extends the locked-image recipe to multilingual settings by appending a multilingual text encoder such as mT5 to an English-only image model, training on unfiltered YFCC100M including non-English text, and using multilingual prompt translations at inference (Zhai et al., 2021). The reported findings are that using the full multilingual dataset and a multilingual tokenizer drastically improves performance on non-Latin-script languages, that starting from a multilingual pre-trained text model gives further improvements especially for low-resource languages, and that English performance does not degrade when using multilingual data and models (Zhai et al., 2021).

It is in this context that the term “SigLiT” became loosely attached to multilingual locked-image systems. The LiT paper states that this line of work is often referred to informally as “SigLiT” in subsequent literature, with “Sig” usually pointing either to sigmoid-scaled learning rate variants for the image tower or to multilingual LiT variants using special learning-rate scheduling and tokenization; it also states that the paper itself does not formally define “SigLiT” as a separate method (Zhai et al., 2021). By contrast, the SigLIP paper provides a stricter definition: locked-image tuning plus sigmoid loss (Zhai et al., 2023). The coexistence of these usages is a terminological ambiguity rather than a methodological contradiction.

The limitations reported for LiT and SigLiT are equally specific. The experimental scope is concentrated on zero-shot image classification and retrieval, not zero-shot detection, segmentation, VQA, or captioning (Zhai et al., 2021). For retrieval, the benefit of locked-image tuning over unlocked alternatives is described as less clear-cut, and with very long schedules and large compute budgets Uu or UU can sometimes overtake Lu on retrieval metrics (Zhai et al., 2021). The method also presupposes access to a strong pre-trained image backbone such as ViT-L/16 or ViT-g/14, whose original training is itself expensive (Zhai et al., 2021). Finally, as with CLIP and ALIGN, web-scale image–text data introduce biases and toxic content from the web; the authors explicitly caution that the technique makes it easier to attach arbitrary text behavior, including undesirable behavior, to existing vision models (Zhai et al., 2021).

At the same time, freezing the image tower has practical advantages that are repeatedly emphasized: it reduces memory footprint and gradient computation, permits precomputing image embeddings once and reusing them across epochs, supports much larger contrastive batches, and makes it feasible to use very large image models under constrained budgets (Zhai et al., 2021).

6. Later optimization refinements and the status of SigLiT in subsequent work

Subsequent work on post-pretraining adaptation of CLIP and SigLIP reframed some of LiT’s design choices as part of a broader optimization problem. TuneCLIP studies self-supervised fine-tuning of open-weight CLIP and SigLIP models and argues that naive continuation training often causes immediate performance collapse because of cold-start bias in the optimizer state and because standard contrastive objectives over-penalize false negatives in web-scale image–text data (Mehta et al., 14 Jan 2026). It introduces a two-stage framework: Optimizer Statistics Recovery (OSR), in which weights remain frozen while momentum, variance, and global contrastive statistics are recovered, followed by fine-tuning with Hinged Global Contrastive Loss (HGCL), which stops pushing a negative farther away once the positive similarity exceeds it by a margin (Mehta et al., 14 Jan 2026).

For open-weight SigLIP ViT-B/16, TuneCLIP reports improvements from a baseline of 63.12% to 65.58% on ImageNet+variants mean, from 62.32% to 63.47% on DataComp average, and from 69.32% to 69.44% on retrieval mean; by contrast, FastCLIP and OpenCLIP baselines are reported to degrade strongly on DataComp for the same starting model (Mehta et al., 14 Jan 2026). The paper does not explicitly use partial freezing or locked-image tuning, but it states that the same optimization procedure applies if the image encoder is frozen and only the text encoder is trained (Mehta et al., 14 Jan 2026).

This later result does not redefine SigLiT, but it changes how locked-image tuning can be interpreted. The original LiT argument was that freezing the image tower preserves general visual features during multimodal alignment (Zhai et al., 2021). TuneCLIP adds that, even when only a subset of the model is adapted, optimizer-state mismatch and false-negative structure remain central failure modes (Mehta et al., 14 Jan 2026). A plausible implication is that modern SigLiT-style systems are best understood as lying at the intersection of three ideas: a frozen vision backbone, text-side or lightweight multimodal adaptation, and careful choice of contrastive optimization objective.

Across these papers, the stable core of Locked-Image Tuning is the decision to preserve the visual encoder and learn language alignment around it. LiT established that this choice is often better for zero-shot transfer than full end-to-end contrastive pre-training on noisy web data (Zhai et al., 2021). SigLiT, in the strict sense used by SigLIP, preserves that architecture while replacing softmax normalization with a pairwise sigmoid loss that scales more flexibly in batch size and compute (Zhai et al., 2023). Subsequent optimization work suggests that if SigLiT is used for self-supervised post-pretraining adaptation, recovery of optimizer statistics and protection against false negatives are likely to be consequential design variables rather than implementation details (Mehta et al., 14 Jan 2026).