---
title: Locked-Image Tuning (SigLiT) Overview
url: https://www.emergentmind.com/topics/locked-image-tuning-siglit
type: topic
---

# Locked-Image Tuning (SigLiT) Overview

Locked-Image Tuning (LiT) is a contrastive-tuning method for language–image models in which a pre-trained image encoder is kept frozen while a text encoder is trained to align with the image representation space. In the notation of the original work, LiT is the canonical `Lu` configuration: a locked, pre-trained image tower paired with a trainable, randomly initialized text tower [2111.07991]. The method was introduced as a modification of CLIP/ALIGN-style contrastive pre-training, with the central claim that preserving a strong vision backbone yields better zero-shot transfer than re-optimizing the image encoder on noisy web-scale image–text data [2111.07991]. In later usage, and explicitly in the SigLIP paper, “SigLiT” denotes the combination of this locked-image design with SigLIP’s pairwise sigmoid loss; the LiT paper itself does not formally define “SigLiT” as a separate method [2303.15343].

## 1. Contrastive-tuning framework and the `Lu` regime

LiT was introduced within a broader **contrastive-tuning** framework in which each tower is specified by whether it is initialized from a pre-trained model and whether it is locked or trainable. The first letter denotes the image tower and the second denotes the text tower. `L` means locked and pre-trained, `U` means trainable and pre-trained, and `u` means trainable and randomly initialized [2111.07991].

| Config | Image tower | Text tower |
|---|---|---|
| `uu` | trainable, random init | trainable, random init |
| `Uu` | trainable, pre-trained | trainable, random init |
| `LU` | locked, pre-trained | trainable, pre-trained |
| `Lu` | locked, pre-trained | trainable, random init |

The specific LiT configuration is `Lu`: the image tower is pre-trained and frozen, while the text tower and its projection are trained. The stated rationale is that pre-trained vision models already provide strong generic representations, whereas web-scale image–text data are useful for alignment to language but noisy for improving the underlying visual descriptor [2111.07991]. The paper reports that when the image tower is unlocked, training contrastive loss becomes lower on the image–text training set, but representation quality and out-of-distribution performance degrade; when the image tower is locked, training loss is worse, but representation quality and out-of-distribution performance are better [2111.07991].

This design also decouples two problems that CLIP and ALIGN solve jointly: learning image representations and learning language alignment. LiT treats the former as largely solved by large-scale supervised or self-supervised vision pre-training, and the latter as a text-side readout problem over an existing visual space [2111.07991]. A plausible implication is that LiT is best viewed not as end-to-end multimodal pre-training, but as a modular retrofitting procedure for turning a vision backbone into a zero-shot image–text model.

## 2. Model architecture, embeddings, and training setup

The image tower in LiT can be any pre-trained vision model producing a fixed-dimensional embedding. The paper reports experiments with ResNet (BiT-M), Vision Transformers including ViT-B/16, ViT-B/32, ViT-L/16, and ViT-g/14, and MLP-Mixer, using both supervised and self-supervised pre-training regimes [2111.07991]. Writing the image encoder as $f_I(\cdot)$, an image $x$ is mapped to
$$
z_I = f_I(x)\in\mathbb{R}^{d_I}.
$$
If needed, a linear head maps to a shared space. In the canonical large-scale LiT setup, the image head is often omitted when dimensions already match, and the text head is retained as the trainable projection [2111.07991].

The text tower is a Transformer-based encoder. Reported options include a ViT-style Transformer on token sequences, BERT-base, BERT-large, T5-base, and mT5-base, with SentencePiece used for ViT/T5/mT5 and WordPiece for BERT [2111.07991]. Given tokenized text $t$, the encoder $f_T$ produces
$$
z_T = f_T(t)\in\mathbb{R}^{d_T},
$$
followed by an optional projection
$$
h_T(z_T)=W_T z_T,\quad W_T\in\mathbb{R}^{d\times d_T}.
$$

For each image–text pair, LiT computes image and text embeddings, L2-normalizes them,
$$
\tilde{u}_i = \frac{u_i}{\|u_i\|},\qquad \tilde{v}_i = \frac{v_i}{\|v_i\|},
$$
and uses scaled cosine similarity
$$
s_{ij} = \frac{\tilde{u}_i^\top \tilde{v}_j}{\tau},
$$
with $\tau$ a learnable temperature parameter or fixed hyperparameter [2111.07991]. The original implementation uses a global contrastive loss across all devices, gathering embeddings from all accelerators to increase the effective number of negatives; an ablation in Appendix E is reported to show that this global loss consistently outperforms a per-device local loss [2111.07991].

The large-scale runs used AdaFactor with $\mathrm{LR}=0.001$, $\beta_1=0.9$, $\beta_2=0.999$, linear warmup for 10k steps, cosine decay, and batch sizes up to 32k, with a default training length of 55k steps and a best model trained for 550k steps [2111.07991]. Public-data runs on CC12M and YFCC100M used Adam with $\beta_1=0.9$, $\beta_2=0.999$, typically learning rate $0.001$, decoupled weight decay $10^{-4}$, gradient clipping at global norm 1, and 20 epochs on CC12M with batch 16,384 [2111.07991].

## 3. From LiT to SigLiT: softmax contrastive loss versus pairwise sigmoid loss

The original LiT objective is the symmetric CLIP-style InfoNCE loss. For a batch of $N$ image–text pairs, with similarity matrix $S_{ij}=s_{ij}$, the image-to-text and text-to-image losses are
$$
\mathcal{L}_{I\rightarrow T}
= - \frac{1}{N}\sum_{i=1}^N \log \frac{\exp(S_{ii})}{\sum_{j=1}^N \exp(S_{ij})},
$$
$$
\mathcal{L}_{T\rightarrow I}
= - \frac{1}{N}\sum_{i=1}^N \log \frac{\exp(S_{ii})}{\sum_{j=1}^N \exp(S_{ji})},
$$
and the total loss is
$$
\mathcal{L}=\frac{1}{2}\left(\mathcal{L}_{I\rightarrow T}+\mathcal{L}_{T\rightarrow I}\right).
$$
This is a batch-level classification objective in which positives are the matched pairs and all mismatched in-batch pairs are negatives [2111.07991].

SigLIP replaces this softmax-based loss with a pairwise sigmoid loss that operates on individual image–text pairs rather than on a batch-normalized softmax. With L2-normalized image embeddings $x_i$, text embeddings $y_j$, learnable temperature $t=e^{t'}$, learnable bias $b$, and labels
$$
z_{ij}=
\begin{cases}
+1 & i=j,\\
-1 & i\neq j,
\end{cases}
$$
the logit is
$$
\ell_{ij}=t\,x_i^\top y_j+b,
$$
and the sigmoid loss is
$$
\mathcal{L}_{\text{sigmoid}}
= -\frac{1}{|B|}\sum_{i=1}^{|B|}\sum_{j=1}^{|B|}\log \sigma\big(z_{ij}\ell_{ij}\big).
$$
Because there is no batch-wise normalization, the loss does not require a global view of all pairwise similarities, which enables a chunked distributed implementation that keeps only a $b\times b$ submatrix in memory at a time and reduces memory from $|B|^2$ to $b^2$ [2303.15343].

In the SigLIP paper, **SigLiT = Sigmoid LiT = LiT + SigLIP-style sigmoid loss** [2303.15343]. The image backbone is strictly locked, image embeddings are precomputed once, and only the text transformer plus the global temperature and bias are trainable. The reported initialization is
$$
t'=\log 10,\qquad b=-10,
$$
with Scaling ViT–Adafactor, $\beta_2=0.95$, learning rate $0.001$, linear warmup over the first 200M examples, cosine decay, and weight decay $0.0001$ on the text tower [2303.15343].

A common misconception is that SigLiT is a wholly separate family of models from LiT. The papers support a narrower interpretation: SigLiT is LiT under a different contrastive objective, retaining the locked-image design while replacing the softmax objective with the pairwise sigmoid formulation [2303.15343].

## 4. Empirical results, transfer behavior, and scaling laws

LiT’s principal claim is strong zero-shot transfer from a locked image tower. With a transformer-based pre-trained ViT-g/14 model, the paper reports **85.2% zero-shot transfer accuracy on the ImageNet test set** and **82.5% on the challenging out-of-distribution ObjectNet test set** [2111.07991]. On additional zero-shot robustness benchmarks, the same model reaches 79.8 on ImageNet-v2, 94.9 on ImageNet-R, 81.8 on ImageNet-A, and 88.6 on ImageNet-ReaL [2111.07991]. The paper further states that ObjectNet zero-shot accuracy of 82.5% beats CLIP by 10.2% and outperforms a supervised ViT-g/14 fine-tuned on ImageNet, reported at 70.5% on ObjectNet [2111.07991].

LiT also emphasizes data efficiency. The paper states that LiT reaches **81.7% ImageNet zero-shot accuracy with only 300M seen pairs**, whereas CLIP needed **12.8B seen pairs** to reach 76.2% [2111.07991]. Using only public components—ViT-L/16 from AugReg, trained on the union of YFCC100m-CLIP and CC12M—the reported ImageNet zero-shot accuracy is **75.7%**, compared with 34.8% for the previous public-data best OpenCLIP and 31.3% for CLIP on the YFCC subset [2111.07991].

SigLiT preserves most of LiT’s top-end performance while changing the objective and improving efficiency. The best SigLiT configuration reported in the SigLIP paper, using a B-sized text tower, a ViT-g image tower, 18B examples, and batch 32k, reaches **84.7%** ImageNet zero-shot, compared with **85.2%** for the original LiT using a much larger g-sized text model [2303.15343]. The paper also reports that with **4 TPUv4** chips, a frozen ViT-g/14 vision tower, batch size 20k, and 107k steps, SigLiT reaches **84.5%** ImageNet zero-shot in **less than two days**; with a frozen ViT-AugReg-B/8 tower, batch size 32k, and 65k steps, it reaches **79.7–79.8%** in about one day [2303.15343].

The batch-size analysis in SigLIP is central to the meaning of SigLiT. The paper reports that sigmoid loss dominates softmax at small batch sizes, that the two are very close around 32k, and that benefits quickly diminish beyond that scale; it explicitly states that **32k is sufficient**, even though the chunked implementation permits batch sizes up to **1,000,000** [2303.15343]. This suggests that SigLiT’s contribution is not only a new loss but also a practical decoupling of optimization quality from extremely large hardware budgets.

## 5. Multilingual extensions, scope, and practical limitations

The LiT paper extends the locked-image recipe to multilingual settings by appending a multilingual text encoder such as mT5 to an English-only image model, training on unfiltered YFCC100M including non-English text, and using multilingual prompt translations at inference [2111.07991]. The reported findings are that using the full multilingual dataset and a multilingual tokenizer drastically improves performance on non-Latin-script languages, that starting from a multilingual pre-trained text model gives further improvements especially for low-resource languages, and that **English performance does not degrade** when using multilingual data and models [2111.07991].

It is in this context that the term “SigLiT” became loosely attached to multilingual locked-image systems. The LiT paper states that this line of work is often referred to informally as “SigLiT” in subsequent literature, with “Sig” usually pointing either to **sigmoid-scaled learning rate** variants for the image tower or to multilingual LiT variants using special learning-rate scheduling and tokenization; it also states that the paper itself does not formally define “SigLiT” as a separate method [2111.07991]. By contrast, the SigLIP paper provides a stricter definition: locked-image tuning plus sigmoid loss [2303.15343]. The coexistence of these usages is a terminological ambiguity rather than a methodological contradiction.

The limitations reported for LiT and SigLiT are equally specific. The experimental scope is concentrated on zero-shot image classification and retrieval, not zero-shot detection, segmentation, VQA, or captioning [2111.07991]. For retrieval, the benefit of locked-image tuning over unlocked alternatives is described as less clear-cut, and with very long schedules and large compute budgets `Uu` or `UU` can sometimes overtake `Lu` on retrieval metrics [2111.07991]. The method also presupposes access to a strong pre-trained image backbone such as ViT-L/16 or ViT-g/14, whose original training is itself expensive [2111.07991]. Finally, as with CLIP and ALIGN, web-scale image–text data introduce biases and toxic content from the web; the authors explicitly caution that the technique makes it easier to attach arbitrary text behavior, including undesirable behavior, to existing vision models [2111.07991].

At the same time, freezing the image tower has practical advantages that are repeatedly emphasized: it reduces memory footprint and gradient computation, permits precomputing image embeddings once and reusing them across epochs, supports much larger contrastive batches, and makes it feasible to use very large image models under constrained budgets [2111.07991].

## 6. Later optimization refinements and the status of SigLiT in subsequent work

Subsequent work on post-pretraining adaptation of CLIP and SigLIP reframed some of LiT’s design choices as part of a broader optimization problem. TuneCLIP studies self-supervised fine-tuning of open-weight CLIP and SigLIP models and argues that naive continuation training often causes immediate performance collapse because of **cold-start bias in the optimizer state** and because standard contrastive objectives over-penalize **false negatives** in web-scale image–text data [2601.09859]. It introduces a two-stage framework: **Optimizer Statistics Recovery (OSR)**, in which weights remain frozen while momentum, variance, and global contrastive statistics are recovered, followed by fine-tuning with **Hinged Global Contrastive Loss (HGCL)**, which stops pushing a negative farther away once the positive similarity exceeds it by a margin [2601.09859].

For open-weight **SigLIP ViT-B/16**, TuneCLIP reports improvements from a baseline of **63.12%** to **65.58%** on ImageNet+variants mean, from **62.32%** to **63.47%** on DataComp average, and from **69.32%** to **69.44%** on retrieval mean; by contrast, FastCLIP and OpenCLIP baselines are reported to degrade strongly on DataComp for the same starting model [2601.09859]. The paper does not explicitly use partial freezing or locked-image tuning, but it states that the same optimization procedure applies if the image encoder is frozen and only the text encoder is trained [2601.09859].

This later result does not redefine SigLiT, but it changes how locked-image tuning can be interpreted. The original LiT argument was that freezing the image tower preserves general visual features during multimodal alignment [2111.07991]. TuneCLIP adds that, even when only a subset of the model is adapted, optimizer-state mismatch and false-negative structure remain central failure modes [2601.09859]. A plausible implication is that modern SigLiT-style systems are best understood as lying at the intersection of three ideas: a frozen vision backbone, text-side or lightweight multimodal adaptation, and careful choice of contrastive optimization objective.

Across these papers, the stable core of Locked-Image Tuning is the decision to preserve the visual encoder and learn language alignment around it. LiT established that this choice is often better for zero-shot transfer than full end-to-end contrastive pre-training on noisy web data [2111.07991]. SigLiT, in the strict sense used by SigLIP, preserves that architecture while replacing softmax normalization with a pairwise sigmoid loss that scales more flexibly in batch size and compute [2303.15343]. Subsequent optimization work suggests that if SigLiT is used for self-supervised post-pretraining adaptation, recovery of optimizer statistics and protection against false negatives are likely to be consequential design variables rather than implementation details [2601.09859].

Source: https://www.emergentmind.com/topics/locked-image-tuning-siglit