---
title: 'TULIP Models: Diverse ML Applications'
url: https://www.emergentmind.com/topics/tulip-models
type: topic
---

# TULIP Models: Diverse ML Applications

Searching arXiv for the TULIP papers and adjacent uses of the acronym to ground the article in current literature.
“TULIP models” does not denote a single research lineage. In contemporary arXiv usage, the name labels several technically unrelated systems across multimodal learning, uncertainty estimation, domain-adapted language modeling, optimization, statistical software, and programming-language semantics. The most prominent recent machine-learning uses include **“TULIP: Token-length Upgraded CLIP”** [2410.10034], **“TULIP: Towards Unified Language-Image Pretraining”** [2503.15485], **“Transitional Uncertainty with Layered Intermediate Predictions”** [2405.17494], **“TULiP: Test-time Uncertainty Estimation via Linearization and Weight Perturbation”** [2505.16923], and **“TULIP: Adapting Open-Source Large Language Models for Underrepresented Languages and Specialized Financial Tasks”** [2508.16243].

## 1. Scope and nomenclature

The label is best treated as a polysemous acronym rather than a unified architecture family. In some papers it denotes a specific neural model, in others a training pipeline, a warm-start algorithm, a software toolbox, or a program logic.

| TULIP designation | Domain | Core description |
|---|---|---|
| Token-length Upgraded CLIP [2410.10034] | Vision-language modeling | Extends CLIP-like text encoders beyond the 77-token cap |
| Towards Unified Language-Image Pretraining [2503.15485] | Multimodal pretraining | Unifies image-text, image-image, text-text contrastive learning and reconstruction |
| Transitional Uncertainty with Layered Intermediate Predictions [2405.17494] | Uncertainty estimation | Uses intermediate-layer exits for single-pass uncertainty |
| Test-time Uncertainty via Linearization and Weight Perturbation [2505.16923] | OOD detection | Post-hoc uncertainty from linearized training dynamics and weight perturbation |
| Turkish finance TULIP models [2508.16243] | Domain-adapted LLMs | Continual pre-training and SFT for Turkish financial tasks |
| TULIP-algorithm [2412.10098] | Stochastic mixed-integer optimization | Two-step warm start via scenario reduction and cut reuse |

This multiplicity matters because secondary literature can otherwise conflate incompatible objects: a CLIP upgrade, a SigLIP-like multimodal pretraining framework, and an OOD detector named TULiP are not variants of one another. A plausible implication is that “TULIP models” should be disambiguated by paper title or domain whenever the term appears in bibliographies or benchmark tables.

## 2. TULIP as Token-length Upgraded CLIP

In **“TULIP: Token-length Upgraded CLIP”**, TULIP is a method for converting a standard CLIP-like contrastive vision-language model with a 77-token text limit into a model that can process arbitrarily long captions while preserving image-text alignment [2410.10034]. The stated bottleneck is CLIP’s fixed, absolute positional embedding table, which truncates inputs beyond 77 positions and degrades performance on dense captions and long compositional descriptions.

The architectural intervention is the replacement of absolute positional encoding with **Rotary Position Embedding (RoPE)** inside every transformer block of the text encoder. Standard self-attention uses
$$
q_m = W_q x_m,\qquad k_n = W_k x_n,
$$
with attention weights derived from
$$
\mathrm{softmax}\!\left(\frac{q_m^T k_n}{\sqrt{d}}\right).
$$
TULIP instead injects position through rotations,
$$
q_m = R_{\Theta,m} W_q x_m,\qquad k_n = R_{\Theta,n} W_k x_n,
$$
so that relative displacement is encoded through phase differences rather than a fixed-length embedding table. The paper’s interpretation is that this improves pairwise token modeling and makes the encoder, in principle, length-agnostic.

The training procedure is explicitly two-stage. The first stage, **relative position distillation**, keeps the original CLIP text encoder as teacher and trains the RoPE-based student on the first 77 tokens of captions using cosine similarity between teacher and student text embeddings. The second stage, **relative position expansion**, fine-tunes the distilled text encoder together with the image encoder on full long captions using a mixture of short- and long-caption contrastive losses. For extrapolation, the method uses **NTK-aware scaled RoPE** with original length \(T_f=77\), new context length \(T_g\), and hyperparameter \(\alpha\). In the main experiments, the method is implemented on OpenAI CLIP / OpenCLIP **ViT-B/16** and **ViT-L/14**, trained on **ShareGPT4V**, with 20 epochs of distillation and one epoch of expansion.

Evaluation spans short-caption retrieval, long-caption retrieval, and text-to-image generation. The long-caption benchmarks are **ShareGPT4V test**, **Urban-1K**, and **Long-DCI**, where Long-DCI is introduced as a 7,000-pair benchmark with captions averaging about 200 tokens. On long-caption retrieval, TULIP is reported to outperform CLIP and Long-CLIP. With **ViT-L/14**, it reaches **55.7 / 56.4** on **Long-DCI**, **99.0 / 99.0** on **ShareGPT4V**, and **90.1 / 91.1** on **Urban-1K**; with **ViT-B/16**, it reaches **50.2 / 50.6**, **98.6 / 98.6**, and **88.1 / 86.6**, respectively. On **ViT-L/14 Long-DCI**, the reported baselines are **35.0 / 37.0** for CLIP, **51.6 / 50.7** for fine-tuned CLIP, **54.0 / 46.1** for Long-CLIP, and **55.7 / 56.4** for TULIP. Short-caption retrieval remains competitive; for example, on **Flickr30k** with **ViT-L/14**, TULIP achieves **56.7 / 79.5** for image-to-text \(R@1/R@5\) and **41.6 / 64.3** for text-to-image \(R@1/R@5\).

The paper also reports qualitative transfer to **Stable Diffusion XL** by replacing the standard CLIP text encoder without retraining the diffusion model. The cited examples show better preservation of details beyond token 77, including phrases such as “the red tulip inside a wooden box,” “an old man sits on a rock,” “playful grin,” and “inside a large glass jar.” Ablations support three design choices: **RoPE outperforms CoPE**, context expansion helps up to roughly **231 tokens** before a mild plateau or decline at **308**, and **cosine loss** outperforms **L2** and **MSE** during distillation. The authors also report that useful image-relevant information is distributed throughout long captions, reinforcing the argument against truncation.

## 3. TULIP as unified language-image pretraining

In **“TULIP: Towards Unified Language-Image Pretraining”**, TULIP is an open-source, drop-in replacement for CLIP-like models intended to preserve global language-image alignment while substantially improving fine-grained visual understanding [2503.15485]. The motivating claim is that CLIP/SigLIP-style contrastive training often privileges high-level semantics at the expense of counting, depth estimation, localization, fine-grained recognition, multi-view reasoning, and compositional reasoning.

The framework combines five elements: **image-text contrastive learning**, **image-image contrastive learning**, **text-text contrastive learning**, **reconstruction regularization**, and **generative data augmentation**. The contrastive core uses a **SigLIP-style** pairwise sigmoid objective. The total contrastive term is
$$
L_\text{cont} = L_\text{I-T} + L_\text{I-I} + L_\text{T-T},
$$
where \(L_\text{I-I}\) operates on augmented image views and \(L_\text{T-T}\) on augmented text views. Reconstruction is added through an **MAE-style decoder** for images and a **causal decoder based on T5** for text, giving
$$
L_\text{recons} = \lambda_i L_\text{image-recons} + \lambda_t L_\text{text-recons},
$$
and the total objective
$$
L_\text{TULIP} = \lambda_c L_\text{cont} + \lambda_r L_\text{recons}.
$$

A central component is **GeCo** (**GEnerative COntrastive view augmentation**). For text, GeCo uses **Llama-3.1-8B-Instruct** to generate positive paraphrases and negative paraphrases with minimal semantic changes. For images, it fine-tunes an instruction-based image editing model with soft prompting, using sources such as **WebVid-10M**, **MVImgNet**, and **InstructPix2Pix**-style semantic edits to produce positive and negative views. This extends contrastive supervision beyond handcrafted augmentations like crop, flip, or color jitter.

The training corpus includes **DataComp-1B**, described as about **1.4 billion image-text pairs**, together with **Recap-DataComp-1B** recaptioned data. The paper states that average caption length increases from about **10.22 words** to **49.43 words**, and that **20%** of original captions are replaced with recaptioned data during augmentation, while **50%** replacement is used for text reconstruction. Optimization uses **Adam**, learning rate \(10^{-4}\), weight decay \(10^{-4}\), gradient clipping norm **2**, batch size **49,152**, and training on up to **32 A100 GPUs**. The framework scales to **over 1B parameters**.

The reported results are broad. In zero-shot classification, **ViT-B/16** TULIP reaches **79.5** on **ImageNet-1K val**, **73.0** on **ImageNet-v2**, **86.2** on **ImageNet-ReaL**, **74.2** on **ObjectNet**, and **73.8** on **ImageNet 10-shot**. Larger variants report **85.0** and **85.3** on ImageNet val for **So/14, 384** and **g/16, 384**, respectively. Retrieval also improves: for **B/16**, the paper reports **54.2** / **70.1** on **COCO** \(T\!\to\!I / I\!\to\!T\) and **81.8** / **93.9** on **Flickr**. The strongest vision-centric evidence appears in linear probing and multimodal evaluation. The paper emphasizes **almost \(2\times\)** improvement over SigLIP on **RxRx1**, and on **MMVP** reports a progression for **SigLIP (ViT-So/14)** from **5.9** at baseline to **17.4** with image-image/text-text contrastive learning, **18.2** with added reconstruction, and **20.3** with full GeCo. On **Winoground**, TULIP is reported as the only contrastive image-text model listed to exceed random chance on **group score**, reaching **17.75** for **ViT-So/14, 384** and **18.50** for **ViT-G/16, 384**, compared with random chance at **16.67**.

Conceptually, this TULIP is distinct from the long-caption CLIP upgrade. The former is a **general pretraining framework** for improving visual fidelity while preserving alignment; the latter is a **retrofit procedure** for extending CLIP-like text context length. Their overlap is limited to the broader CLIP-like multimodal setting.

## 4. TULIP and TULiP in uncertainty estimation

The acronym also appears in two separate uncertainty-estimation lines that share neither architecture nor derivation. **“Transitional Uncertainty with Layered Intermediate Predictions”** defines TULIP as a **single-pass uncertainty estimation framework**, whereas **“Test-time Uncertainty Estimation via Linearization and Weight Perturbation”** defines TULiP as a **post-hoc OOD detector** grounded in linearized training dynamics [2405.17494] [2505.16923].

The single-pass TULIP paper argues that many deterministic uncertainty estimators, such as **SNGP** and **DUQ**, preserve feature distances at the final output in a way that can oppose the deep network’s compression objective. Its remedy is **transitional feature preservation**: rather than forcing the final representation to retain all uncertainty-relevant geometry, the method places **internal classifiers (ICs)** at intermediate layers and extracts uncertainty before subsequent layers collapse information. Training uses a weighted shallow-deep objective,
$$
L_{SDN}(\mathbf{x}, y) = p_0 L_s(h_w(\mathbf{x}), y) + \sum_{i=1}^{N_{IC}} p_i L_s(f^i_{w_l}(\mathbf{x}), y),
$$
with equal weighting \(p_k=\frac{1}{N_{IC}+1}\) often reported to work well. Per-exit uncertainties are then aggregated as
$$
u_{final}(\mathbf{x}) = \frac{1}{\sum_{i=1}^{N_{IC}} r_i} \sum_{i=1}^{N_{IC}} r_i\, u_s(f^i_{w_i}(\mathbf{x})),
$$
using a GP-style / Dempster-Shafer score
$$
u_s(\mathbf{x}) = \frac{K}{K + \sum_{k=1}^K \exp(g^k(\mathbf{x}))}.
$$
Combination weights are fit by logistic regression on proxy labels derived from prediction switches among internal exits, without requiring OOD validation data. The paper reports competitiveness or superiority over **DNN**, **Energy-Based**, **DUN**, **Early Exit**, **SNGP**, and **DUQ** across **CIFAR10/CIFAR100**, **ImageNet**, and **MedMNIST CT** settings, especially under class imbalance, deeper architectures, and modality shift.

By contrast, TULiP formulates OOD uncertainty as the sensitivity of a trained network’s final prediction to a hypothetical perturbation introduced before convergence. The theory is stated in an **NTK / lazy-training** regime. A linearized model
$$
f(\cdot;\theta) := f_{\mathrm{Init}}(\cdot) + \left.\nabla_{\theta} f_{\mathrm{True}}(\cdot;\theta)\right|_{\theta=\theta^\ast} (\theta-\theta^\ast)
$$
is perturbed, and the discrepancy between converged perturbed and unperturbed solutions is bounded by a gradient-space distance to the training set. The implementation then approximates this uncertainty post hoc by perturbing model parameters, generating surrogate predictions, and computing predictive entropy. The final score is
$$
U(z)= -\sum_{y=1}^o p_y\log p_y
$$
for the averaged predictive distribution over sampled perturbed outputs. The paper reports defaults **\(M=10\)** samples, **\(\epsilon=0.005\)**, **\(\delta=8\)**, **\(\lambda=1.25\)**, perturbation of all parameters including biases and BatchNorm \(\beta,\gamma\), and evaluation on **OpenOOD v1.5** using **ResNet-18** and **ResNet-50**. On **CIFAR-10**, TULiP (ENT) is reported at **FPR@95 40.77 / 28.40** and **AUROC 89.36 / 92.25** for near/far OOD; on **ImageNet-1K**, it reports **FPR@95 64.96 / 48.01** and **AUROC 78.38 / 88.85**. The paper’s stated emphasis is that gains are strongest on **near-distribution OOD**.

Taken together, these works show that the TULIP acronym spans both **architectural single-pass uncertainty estimation** and **test-time perturbative uncertainty estimation**. This suggests that citation by acronym alone is especially error-prone in safety- or robustness-oriented surveys.

## 5. TULIP as Turkish-finance-adapted large language models

In **“TULIP: Adapting Open-Source Large Language Models for Underrepresented Languages and Specialized Financial Tasks”**, TULIP denotes a pair of Turkish-finance-adapted LLM families based on **Llama 3.1 8B** and **Qwen 2.5 7B**, together with their instruction-tuned variants [2508.16243]. The paper’s framing is application-driven: finance requires privacy-preserving, on-premise deployment, while Turkish is described as an underrepresented language for general-purpose open models.

The development pipeline has five stages: **data collection**, **continual pre-training (CPT)**, **benchmark design**, **synthetic data generation**, and **supervised fine-tuning (SFT)**. The CPT corpus totals about **2.19 billion tokens**, with the following reported breakdown: **Academic sources 1.1B**, **Financial institutions 150M**, **Textbooks and study materials 200M**, **Market and business data 350M**, **Legislation and regulations 50M**, and **Other reports and documents 340M**. For SFT, the raw text sources are **Academic**, **Central Bank**, **News**, and **Trade Registry Gazette**.

CPT uses a **QLoRA-style** setup with learning rate \(2\times10^{-5}\), optimizer `paged_adamw_8bit`, `bf16` precision, **4-bit** quantization, **LoRA alpha 128**, and **LoRA rank 64**, with all linear layers, the head, and embedding layers optimized. SFT keeps the same general setup but reduces the learning rate to \(2\times10^{-6}\). The resulting model names are **TULIP-Llama-3.1**, **TULIP-Qwen2.5**, **TULIP-Llama-3.1-IT**, and **TULIP-Qwen2.5-IT**.

Because no suitable evaluation suite existed, the paper introduces two benchmarks. **FINTR-EXAMS** is a 5-shot multiple-choice benchmark based on undergraduate finance courses across seven domains: **Banking and Insurance**, **Foreign Trade and International Finance**, **Economics**, **Finance and Investment**, **Private Sector Finance**, **Public Finance and Tax**, and **Accounting and Financial Reporting**. The paper states a human passing threshold of **60/100**. The second benchmark is a **Turkish Trade Registry Gazette QA** dataset with **880 manually annotated question-answer pairs**, **9 annotators**, and **Cohen’s kappa = 0.91**. The event types are **Change of Capital**, **Change of Management**, **Composition with Creditors**, and **Notice to Creditors**.

Synthetic SFT data are generated rather than translated from English datasets. The paper states that translation quality was inadequate for specialized financial terminology and that human translation was too expensive. The synthetic procedure uses random source chunks, seed prompts, GPT-4o rephrasing, and filtering. The final SFT set contains about **23K instruction-answer pairs**, with source distribution **Academic 54%**, **Central Bank 5%**, **News 21%**, **Trade Registry Gazette 20%**, and task distribution including **fill-in-the-blank 19%**, **multiple-choice QA 16%**, **summarization 16%**, **true/false 14%**, **sentiment analysis 14%**, and smaller shares for the remaining task types.

The results emphasize the value of CPT. On **FINTR-EXAMS**, **TULIP-Llama3.1** improves mean accuracy from **0.544** to **0.583**, and **TULIP-Qwen2.5** from **0.600** to **0.668**. The corresponding instruction-tuned variants reach **0.573** and **0.659**. On the **Trade Registry Gazette QA** benchmark, **Llama3.1-Instruct** reports mean **0.866**, **TULIP-Llama3.1-IT 0.907**, and **TULIP-Llama3.1-Instruct 0.918**; **Qwen2.5-Instruct** reports **0.775**, **TULIP-Qwen2.5-IT 0.858**, and **TULIP-Qwen2.5-Instruct 0.818**. The paper also reports that untuned **Qwen2.5** sometimes switches languages mid-answer or emits English/Chinese, a behavior said to disappear after adaptation. The authors note several limitations: SFT does not always improve over CPT alone, hard numeric reasoning remains difficult, and specialized terminology such as *memzuç* and *temdit* still challenges general models.

This TULIP family is therefore not a generic foundational model in the CLIP sense. It is a **domain- and language-adaptation recipe** for finance-focused, underrepresented-language deployment under privacy constraints.

## 6. Other TULIP usages and disambiguation outside model nomenclature

Several additional arXiv works use TULIP in ways that are not “models” in the narrow neural-network sense. In **“A Two-Step Warm Start Method Used for Solving Large-Scale Stochastic Mixed-Integer Problems”**, TULIP is a two-phase optimization framework: first reduce the scenario set and solve only the root node of the reduced problem to harvest cuts, then reuse those cuts to warm-start the full problem [2412.10098]. The paper demonstrates this on the **Stochastic Capacitated Vehicle Routing Problem** and the **Two-Stage Stochastic Steiner Forest Problem**, positioning TULIP as a preprocessing mechanism for branch-and-cut rather than a predictive model.

In statistics, **“TULIP: A Toolbox for Linear Discriminant Analysis with Penalties”** presents TULIP as an **R package** integrating **DSDA**, **ROAD**, **SOS**, **SeSDA**, **MSDA**, and **CATCH** for high-dimensional, semiparametric, multiclass, tensor-valued, and covariate-adjusted discriminant analysis [1904.03469]. Here the acronym denotes software for sparse discriminant estimation rather than a single classifier. In programming languages, **“A denotationally-based program logic for higher-order store”** uses TULIP, written as \(\LMuRef{}\), for a higher-order separation logic over the typed equational theory of higher-order store [2308.02906]. In graph rewriting, **“Graph Creation, Visualisation and Transformation”** describes **GraphPaper** as an editor built on the graph-visualization engine **TULIP**, which provides efficient graph storage, iteration, and dynamic visualisation [1003.4326].

A related but secondary usage appears in LiDAR super-resolution. **“Real-Time LiDAR Super-Resolution via Frequency-Aware Multi-Scale Fusion”** treats **TULIP** as its strongest transformer-based baseline, describing it as a Swin Transformer–style U-Net on range images with **row-based patching**, **non-square windows**, and **circular padding** [2511.07377]. In the reported comparison on **KITTI**, that baseline achieves **MAE 0.4354**, **CD 0.1342**, **IoU 0.3819**, and **F1 0.5502**, while FLASH improves these to **0.3899**, **0.1161**, **0.3972**, and **0.5661**. Since the baseline is described from the perspective of FLASH, this use is best understood as a benchmark reference rather than a fully reconstructed TULIP paper in its own right.

Finally, the word **“tulip”** also appears in combustion literature, but there it refers to **tulip flames**, a flame morphology rather than an acronymic model family. Recent papers analyze tulip flame formation through rarefaction-wave-driven hydrodynamics, ignition geometry, wall heat loss, or reduced chemical-diffusive modeling [2502.00895] [2308.12904] [2309.05893] [2604.08027] [2604.19812]. That literature is terminologically adjacent but conceptually separate from TULIP acronyms in machine learning and computation.

Across these usages, the commonality is nominal rather than methodological. TULIP may designate a long-context CLIP retrofit, a unified multimodal pretraining framework, a single-pass uncertainty method, a perturbative OOD estimator, a Turkish-finance LLM adaptation pipeline, a stochastic-programming warm start, a statistical software toolbox, a program logic, or a graph-visualization backend. For technical reading, the paper title and domain are therefore indispensable disambiguators.

Source: https://www.emergentmind.com/topics/tulip-models