---
title: 'GIRCSE: Iterative Refinement for Text Embeddings'
url: https://www.emergentmind.com/topics/gircse
type: topic
---

# GIRCSE: Iterative Refinement for Text Embeddings

GIRCSE, short for **Generative Iterative Refinement for Contrastive Sentence Embeddings**, is a framework for LLM-based text embeddings that replaces the usual encoder-style, single-pass extraction paradigm with an autoregressive refinement procedure. Instead of treating a large language model as a static feature extractor whose hidden states are pooled once, GIRCSE turns embedding construction into a generative process: the model produces a sequence of **soft refinement tokens**, feeds them back into the model, and supervises the resulting intermediate representations at every step under an **Iterative Contrastive Refinement (ICR)** objective. The central claim is that standard LLM embedders underuse **autoregressive generation**, whereas GIRCSE uses generation itself as the mechanism for progressively improving semantic representations and thereby captures latent concepts and implicit semantics that encoder-only methods often miss [2509.24291].

## 1. Conceptual position within LLM-based embedding research

The motivation for GIRCSE is the observation that most existing LLM-based embedding models remain effectively **encoder-style feature extractors**: they run the model once, pool hidden states, and optimize a contrastive objective. In that setup, the input is compressed into a fixed vector in a **single pass**. GIRCSE departs from this assumption by arguing that embeddings should not be read off immediately; rather, the model should be allowed to **speak an embedding language** composed of tokens that need not be human-readable but are optimized to improve semantic representation [2509.24291].

The framework differs from standard LLM embedding approaches in three explicit respects. First, it is **generative rather than purely extractive**: instead of pooling only the input’s hidden states, it generates auxiliary tokens that are fed back into the model. Second, the generated tokens are **soft, not discrete**, which preserves differentiability and avoids a mismatch between human-language generation and embedding optimization. Third, training **supervises every generation step**, rather than only the final embedding, so the refinement trajectory itself is learned [2509.24291].

In the comparison discussed by the paper, prior methods such as **E5-Mistral**, **GritLM**, **LLM2Vec**, and **NV-Embed** remain encoder-like in their final embedding extraction, whereas GIRCSE is identified as the only method in that table combining **generation + soft tokens + test-time scaling** [2509.24291]. This suggests that the contribution is not merely architectural substitution, but a redefinition of embedding inference as a bounded autoregressive computation.

## 2. Architecture and embedding formation

Let the input sequence be $\mathbf{T} = (t_1,\ldots,t_N)$. GIRCSE uses a pretrained language model with parameters $\psi=\{\mathbf{E},\theta,\phi\}$, where $\mathbf{E}$ is the token embedding matrix, $\theta$ is the Transformer decoder, and $\phi$ is the LM head. The input is embedded as

$$
\mathbf{X} = (\mathbf{E}[t_1], \mathbf{E}[t_2], \ldots, \mathbf{E}[t_N]) \in \mathbb{R}^{N\times d}.
$$

The model then autoregressively generates $K$ auxiliary soft tokens according to

$$
p_\psi(\mathbf{S} \mid \mathbf{T}) = \prod_{k=1}^{K} p_\psi(\mathbf{s}_k \mid \mathbf{T}, \mathbf{S}_{<k}),
$$

where $\mathbf{S}=(\mathbf{s}_1,\ldots,\mathbf{s}_K)$ and $\mathbf{S}_{<k}$ denotes previously generated soft tokens [2509.24291].

At each step, GIRCSE concatenates the original input embeddings $\mathbf{X}$ with the embeddings of the generated soft tokens $\mathbf{D}=(\mathbf{d}_1,\ldots,\mathbf{d}_K)$, runs the Transformer, and collects the last-layer hidden states corresponding to generated positions:

$$
\mathbf{H} = f_{\theta}([\mathbf{X}; \mathbf{D}]) \in \mathbb{R}^{(N+K)\times d},
$$

$$
\mathbf{G} = (\mathbf{g}_1,\ldots,\mathbf{g}_K) = (\mathbf{h}_{N+1}^{(L)},\ldots,\mathbf{h}_{N+K}^{(L)}).
$$

The final sentence embedding is pooled from those generated-token hidden states with mean pooling as default:

$$
\mathbf{z} = \mathcal{P}(\mathbf{G}) = \frac{1}{K}\sum_{i=1}^{K}\mathbf{g}_i.
$$

A crucial clarification is that this is **not external text generation**. The appendix-level algorithm described in the paper makes explicit that the loop embeds the input tokens, initializes an empty list of generated embeddings, repeatedly runs the decoder on the input plus previously generated soft embeddings, obtains the hidden state at the current generation position, computes a vocabulary distribution with the LM head, converts it to a soft embedding by weighted averaging over the token embedding matrix, and appends that embedding before a final pooling pass [2509.24291]. The generated sequence therefore functions as an internal refinement trajectory rather than an output utterance.

## 3. Soft token mechanism and the “embedding language”

GIRCSE does not sample hard discrete tokens during training. At generation step $k$, the LM head produces a distribution over the vocabulary:

$$
\mathbf{s}_k = \text{softmax}(\mathbf{W}_\phi \mathbf{h}'_{k-1} + \mathbf{b}_\phi),
$$

where $\mathbf{h}'_{k-1}$ is the final-layer hidden state at the current generation position. That soft token is then mapped back into embedding space via a convex combination of vocabulary embeddings:

$$
\mathbf{d}_k = \sum_{i=1}^{|\mathcal{V}|} s_{k,i}\mathbf{e}_i.
$$

This **soft token** mechanism serves two stated purposes. The first is **differentiability**, since gradients flow through the entire generation chain. The second is **semantic richness**, because the model retains the full probability distribution instead of collapsing to a single argmax token [2509.24291].

The paper’s core intuition is that a soft token can encode an “embedding-language” representation more effectively than a hard natural-language token, because the representation is optimized for downstream similarity rather than fluency. This does not imply interpretability in ordinary linguistic terms. On the contrary, one of the stated limitations is that the generated tokens are **not directly interpretable as natural language**, and the learned “embedding language” remains abstract [2509.24291].

A common misunderstanding is to equate GIRCSE with instruction-conditioned text generation followed by standard embedding extraction. That interpretation is inconsistent with the described mechanism. The generated objects are auxiliary soft representations inside the embedding pipeline, not discrete textual outputs. A plausible implication is that the method occupies an intermediate space between representation learning and generative latent computation.

## 4. Iterative Contrastive Refinement objective

The main training objective is **Iterative Contrastive Refinement (ICR)**, consisting of two components: **stepwise contrastive loss** and **iterative refinement regularization** [2509.24291].

For each prefix of generated tokens, the model forms an intermediate embedding

$$
\mathbf{z}_k = \mathcal{P}(\mathbf{G}_{1:k}).
$$

For a query–positive document pair $(q,d^+)$, GIRCSE applies a standard InfoNCE-style contrastive objective at every step:

$$
\mathcal{L}_{\text{contrast}} = \sum_{k=1}^{K} \mathcal{L}_k,
$$

$$
\mathcal{L}_k = - \log \frac{ \exp\!\big(\sigma(\mathbf{z}_{k}^{q}, \mathbf{z}_{k}^{d^+})/\tau\big) }{ \sum_{d \in \mathcal{B}} \exp\!\big(\sigma(\mathbf{z}_{k}^{q}, \mathbf{z}_{k}^{d})/\tau\big) },
$$

where $\mathcal{B}$ is the batch of documents including positives and negatives, $\sigma$ is cosine similarity, and $\tau$ is the temperature. The explicit purpose is to ensure that **each generation step must already be useful for retrieval or similarity**, rather than only the terminal state [2509.24291].

The regularization term addresses the observation that generating more tokens does not automatically improve embedding quality; later tokens can become redundant or nearly identical. To enforce actual progress, the model uses a monotonic refinement penalty:

$$
\mathcal{L}_{\text{reg}} = \frac{1}{K-1}\sum_{k=1}^{K-1} \max\!\big(\log \mathcal{L}_{k+1} - \log \mathcal{L}_k,\, 0\big).
$$

This penalizes cases where later steps fail to improve over earlier ones. The total objective is

$$
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{contrast}} + \lambda \mathcal{L}_{\text{reg}},
$$

with $\lambda$ a balancing coefficient [2509.24291].

The design hypothesis supported by the reported ablations is cumulative: **generation only** improves over **no generation**; **generation + stepwise loss (SL)** improves further; and **generation + SL + ICR** performs best overall. The strongest gains from SL and ICR appear on instruction-following tasks, where the model must gradually distill a task-dependent semantic signal [2509.24291]. This suggests that the objective is not merely stabilizing training but shaping a refinement process with measurable functional depth.

## 5. Training configuration and evaluation protocol

The implementation fine-tunes open-source LLMs with **LoRA** rather than full fine-tuning. The principal backbones are **Mistral-7B** and **Qwen2.5-7B**, with Mistral-7B as the primary backbone and Qwen2.5-7B used for validation. Training data comes from the **bge-icl** dataset, which mixes supervised pairs and hard negatives. Owing to compute limits, the experiments use **20% of that dataset, about 0.2M examples** [2509.24291].

The reported hyperparameters are: LoRA rank $=64$, LoRA scaling $\alpha = 32$, learning rate $= 1\text{e-}5$ generally, **Llama-3B uses $1\text{e-}4$**, contrastive temperature $\tau = 0.02$, $\lambda = 1$, batch size $=2$, gradient accumulation $=8$, effective batch size $=16$, and training for **1 epoch** [2509.24291]. A particularly important detail is that GIRCSE is trained with **$K=5$** generation steps, but at inference it can generate more tokens, up to **$K=20$** in the experiments.

The evaluation covers **MTEB English v2**, comprising **41 datasets across 7 task categories**: retrieval, reranking, clustering, pair classification, classification, STS, and summarization. It also includes two instruction-following benchmarks from Inbedder, **IntentEmotion** and **NYTClustering**, with additional results on **TREC**, **NanoBEIR**, and BEIR-style settings reported in the appendix [2509.24291].

Comparison systems include non-LLM models such as **E5-Large**, **GTE-Large**, **BGE-Large**, and **UAE-Large**; LLM-based embedding models such as **E5-Mistral**, **SFR-Embedding-2**, **gte-Qwen2**, **LLM2Vec**, **GritLM**, and **NV-Embed-v1**; generative baselines including two-stage generation variants such as **E5-Mistral (w/ gen.)**, **GritLM (w/ gen.)**, **E5-Large (w/ gen.)**, and **Inbedder**; and fair re-implementations, notably **Causal-EOS** and **Bidirectional-Avg** [2509.24291]. The fair baselines are emphasized because they control for backbone and training data scale, thereby isolating the effect of the generative refinement mechanism itself.

## 6. Empirical results, test-time scaling, and stated limitations

On **MTEB**, the reported headline results are **67.83** average for **GIRCSE-Mistral** (rank 5) and **67.67** for **GIRCSE-Qwen2** (rank 6). These scores are described as competitive with or better than strong LLM embedding baselines such as **GritLM: 67.07** and **LLM2Vec: 64.57**, while using only **0.2M examples**, much less than many competing 7B embedding systems. The strongest MTEB result in the comparison table is **gte-Qwen2 at 70.72**, but GIRCSE is presented as more balanced because it performs much better on instruction-following tasks than many top generic embedders [2509.24291].

On the two instruction-following benchmarks, **GIRCSE-Mistral** achieves **62.97** average (rank 2) and **GIRCSE-Qwen2** achieves **62.48** (rank 3). **Inbedder** attains the best instruction-following score, **77.17**, but performs very poorly on MTEB, **50.32**, illustrating a strong trade-off. GIRCSE is reported to avoid that collapse while keeping both sides reasonably strong, and the authors state that it achieves the **best overall ranking across benchmarks**, balancing MTEB and instruction-following quality better than prior methods [2509.24291].

The paper further claims that GIRCSE outperforms the fair baselines **Causal-EOS** and **Bidirectional-Avg**, as well as competitive LLM embedders including **LLM2Vec**, **GritLM**, **NV-Embed-v1**, **E5-Mistral**, and **SFR-Embedding-2**, together with generative baselines such as **Inbedder** and two-stage generation variants. The stated comparative advantage is strongest when the backbone is matched and training data is limited, because under those conditions gains can more plausibly be attributed to generative refinement itself [2509.24291].

A key novelty is the reported **emergent test-time scaling** property. The experiments evaluate $K \in \{1,3,5,10,15,20\}$ and find that performance increases monotonically with $K$ for GIRCSE on many tasks, including beyond the training regime, since the model is trained with $K=5$ but still improves at $K=10,15,20$. By contrast, standard non-generative baselines do not improve with extra generation and often degrade [2509.24291]. The authors interpret this as evidence that GIRCSE has learned a genuine iterative refinement process rather than a fixed-depth transformation.

The qualitative analysis gives a concrete example. For the input “Why is it so hard to track down this card?”, early generated tokens are content words such as “why,” “hard,” “track,” and “card,” while later tokens evolve into task-specific expansions. Under an intention prompt, examples include “seek,” “elusive,” “question,” and “inquiry”; under an emotion prompt, examples include “frustrating,” “struggle,” “challenging,” and “perseverance” [2509.24291]. The paper interprets this as evidence that early tokens capture coarse, input-aligned semantics, whereas later tokens refine toward more task- and instruction-specific concepts. This suggests a semantic chain-of-thought analogue for embeddings, although the paper does not claim natural-language transparency for the soft-token process.

The method’s efficiency profile is explicitly qualified. Autoregressive generation is acknowledged to be more expensive than a single encoder pass. The appendix provides a rough theoretical analysis stating that without caching, GIRCSE costs about **$K+1$ times** the baseline in compute when $K \ll N$. However, **peak memory grows only modestly**, **KV caching reduces inference FLOPs to nearly baseline levels**, and the reported FLOP multiplier with caching is about **1.0–1.1×** in the experiments, which the paper presents as making test-time scaling practical [2509.24291].

The stated limitations are fivefold: **higher complexity than standard embeddings**; dependence on careful training of soft tokens and ICR; the fact that training–inference length mismatch is **not free** even though the model generalizes beyond $K=5$; evaluation that is primarily on **English benchmarks**; and the lack of direct interpretability of the generative tokens as natural language [2509.24291]. Taken together, these limitations define the current boundary of the approach: GIRCSE is not a drop-in simplification of encoder-style embedding extraction, but a more complex representation-learning framework whose main empirical distinction is that additional inference-time computation can improve embedding quality rather than merely re-encode the same information.

Source: https://www.emergentmind.com/topics/gircse