GIRCSE: Iterative Refinement for Text Embeddings

• GIRCSE is a framework that transforms conventional one-pass embedding extraction into an iterative, autoregressive process using soft token refinement.
• It employs an Iterative Contrastive Refinement objective that supervises each generation step to enhance semantic accuracy.
• Empirical results demonstrate that increasing generation steps consistently improves embedding quality while keeping computational costs manageable.

GIRCSE, short for Generative Iterative Refinement for Contrastive Sentence Embeddings, is a framework for LLM-based text embeddings that replaces the usual encoder-style, single-pass extraction paradigm with an autoregressive refinement procedure. Instead of treating a LLM as a static feature extractor whose hidden states are pooled once, GIRCSE turns embedding construction into a generative process: the model produces a sequence of soft refinement tokens, feeds them back into the model, and supervises the resulting intermediate representations at every step under an Iterative Contrastive Refinement (ICR) objective. The central claim is that standard LLM embedders underuse autoregressive generation, whereas GIRCSE uses generation itself as the mechanism for progressively improving semantic representations and thereby captures latent concepts and implicit semantics that encoder-only methods often miss (Tsai et al., 29 Sep 2025).

1. Conceptual position within LLM-based embedding research

The motivation for GIRCSE is the observation that most existing LLM-based embedding models remain effectively encoder-style feature extractors: they run the model once, pool hidden states, and optimize a contrastive objective. In that setup, the input is compressed into a fixed vector in a single pass. GIRCSE departs from this assumption by arguing that embeddings should not be read off immediately; rather, the model should be allowed to speak an embedding language composed of tokens that need not be human-readable but are optimized to improve semantic representation (Tsai et al., 29 Sep 2025).

The framework differs from standard LLM embedding approaches in three explicit respects. First, it is generative rather than purely extractive: instead of pooling only the input’s hidden states, it generates auxiliary tokens that are fed back into the model. Second, the generated tokens are soft, not discrete, which preserves differentiability and avoids a mismatch between human-language generation and embedding optimization. Third, training supervises every generation step, rather than only the final embedding, so the refinement trajectory itself is learned (Tsai et al., 29 Sep 2025).

In the comparison discussed by the paper, prior methods such as E5-Mistral, GritLM, LLM2Vec, and NV-Embed remain encoder-like in their final embedding extraction, whereas GIRCSE is identified as the only method in that table combining generation + soft tokens + test-time scaling (Tsai et al., 29 Sep 2025). This suggests that the contribution is not merely architectural substitution, but a redefinition of embedding inference as a bounded autoregressive computation.

2. Architecture and embedding formation

Let the input sequence be $\mathbf{T} = (t_1,\ldots,t_N)$. GIRCSE uses a pretrained LLM with parameters $\psi=\{\mathbf{E},\theta,\phi\}$, where $\mathbf{E}$ is the token embedding matrix, $\theta$ is the Transformer decoder, and $\phi$ is the LM head. The input is embedded as

$$\mathbf{X} = (\mathbf{E}[t_1], \mathbf{E}[t_2], \ldots, \mathbf{E}[t_N]) \in \mathbb{R}^{N\times d}.$$

The model then autoregressively generates $K$ auxiliary soft tokens according to

$$p_\psi(\mathbf{S} \mid \mathbf{T}) = \prod_{k=1}^{K} p_\psi(\mathbf{s}_k \mid \mathbf{T}, \mathbf{S}_{<k}),$$

where $\mathbf{S}=(\mathbf{s}_1,\ldots,\mathbf{s}_K)$ and $\mathbf{S}_{<k}$ denotes previously generated soft tokens (Tsai et al., 29 Sep 2025).

At each step, GIRCSE concatenates the original input embeddings $\psi=\{\mathbf{E},\theta,\phi\}$0 with the embeddings of the generated soft tokens $\psi=\{\mathbf{E},\theta,\phi\}$1, runs the Transformer, and collects the last-layer hidden states corresponding to generated positions:

$\psi=\{\mathbf{E},\theta,\phi\}$2

$\psi=\{\mathbf{E},\theta,\phi\}$3

The final sentence embedding is pooled from those generated-token hidden states with mean pooling as default:

$\psi=\{\mathbf{E},\theta,\phi\}$4

A crucial clarification is that this is not external text generation. The appendix-level algorithm described in the paper makes explicit that the loop embeds the input tokens, initializes an empty list of generated embeddings, repeatedly runs the decoder on the input plus previously generated soft embeddings, obtains the hidden state at the current generation position, computes a vocabulary distribution with the LM head, converts it to a soft embedding by weighted averaging over the token embedding matrix, and appends that embedding before a final pooling pass (Tsai et al., 29 Sep 2025). The generated sequence therefore functions as an internal refinement trajectory rather than an output utterance.

3. Soft token mechanism and the “embedding language”

GIRCSE does not sample hard discrete tokens during training. At generation step $\psi=\{\mathbf{E},\theta,\phi\}$5, the LM head produces a distribution over the vocabulary:

$\psi=\{\mathbf{E},\theta,\phi\}$6

where $\psi=\{\mathbf{E},\theta,\phi\}$7 is the final-layer hidden state at the current generation position. That soft token is then mapped back into embedding space via a convex combination of vocabulary embeddings:

$\psi=\{\mathbf{E},\theta,\phi\}$8

This soft token mechanism serves two stated purposes. The first is differentiability, since gradients flow through the entire generation chain. The second is semantic richness, because the model retains the full probability distribution instead of collapsing to a single argmax token (Tsai et al., 29 Sep 2025).

The paper’s core intuition is that a soft token can encode an “embedding-language” representation more effectively than a hard natural-language token, because the representation is optimized for downstream similarity rather than fluency. This does not imply interpretability in ordinary linguistic terms. On the contrary, one of the stated limitations is that the generated tokens are not directly interpretable as natural language, and the learned “embedding language” remains abstract (Tsai et al., 29 Sep 2025).

A common misunderstanding is to equate GIRCSE with instruction-conditioned text generation followed by standard embedding extraction. That interpretation is inconsistent with the described mechanism. The generated objects are auxiliary soft representations inside the embedding pipeline, not discrete textual outputs. A plausible implication is that the method occupies an intermediate space between representation learning and generative latent computation.

4. Iterative Contrastive Refinement objective

The main training objective is Iterative Contrastive Refinement (ICR), consisting of two components: stepwise contrastive loss and iterative refinement regularization (Tsai et al., 29 Sep 2025).

For each prefix of generated tokens, the model forms an intermediate embedding

$\psi=\{\mathbf{E},\theta,\phi\}$9

For a query–positive document pair $\mathbf{E}$0, GIRCSE applies a standard InfoNCE-style contrastive objective at every step:

$\mathbf{E}$1

$\mathbf{E}$2

where $\mathbf{E}$3 is the batch of documents including positives and negatives, $\mathbf{E}$4 is cosine similarity, and $\mathbf{E}$5 is the temperature. The explicit purpose is to ensure that each generation step must already be useful for retrieval or similarity, rather than only the terminal state (Tsai et al., 29 Sep 2025).

The regularization term addresses the observation that generating more tokens does not automatically improve embedding quality; later tokens can become redundant or nearly identical. To enforce actual progress, the model uses a monotonic refinement penalty:

$\mathbf{E}$6

This penalizes cases where later steps fail to improve over earlier ones. The total objective is

$\mathbf{E}$7

with $\mathbf{E}$8 a balancing coefficient (Tsai et al., 29 Sep 2025).

The design hypothesis supported by the reported ablations is cumulative: generation only improves over no generation; generation + stepwise loss (SL) improves further; and generation + SL + ICR performs best overall. The strongest gains from SL and ICR appear on instruction-following tasks, where the model must gradually distill a task-dependent semantic signal (Tsai et al., 29 Sep 2025). This suggests that the objective is not merely stabilizing training but shaping a refinement process with measurable functional depth.

5. Training configuration and evaluation protocol

The implementation fine-tunes open-source LLMs with LoRA rather than full fine-tuning. The principal backbones are Mistral-7B and Qwen2.5-7B, with Mistral-7B as the primary backbone and Qwen2.5-7B used for validation. Training data comes from the bge-icl dataset, which mixes supervised pairs and hard negatives. Owing to compute limits, the experiments use 20% of that dataset, about 0.2M examples (Tsai et al., 29 Sep 2025).

The reported hyperparameters are: LoRA rank $\mathbf{E}$9, LoRA scaling $\theta$0, learning rate $\theta$1 generally, Llama-3B uses $\theta$2, contrastive temperature $\theta$3, $\theta$4, batch size $\theta$5, gradient accumulation $\theta$6, effective batch size $\theta$7, and training for 1 epoch (Tsai et al., 29 Sep 2025). A particularly important detail is that GIRCSE is trained with $\theta$8 generation steps, but at inference it can generate more tokens, up to $\theta$9 in the experiments.

The evaluation covers MTEB English v2, comprising 41 datasets across 7 task categories: retrieval, reranking, clustering, pair classification, classification, STS, and summarization. It also includes two instruction-following benchmarks from Inbedder, IntentEmotion and NYTClustering, with additional results on TREC, NanoBEIR, and BEIR-style settings reported in the appendix (Tsai et al., 29 Sep 2025).

Comparison systems include non-LLM models such as E5-Large, GTE-Large, BGE-Large, and UAE-Large; LLM-based embedding models such as E5-Mistral, SFR-Embedding-2, gte-Qwen2, LLM2Vec, GritLM, and NV-Embed-v1; generative baselines including two-stage generation variants such as E5-Mistral (w/ gen.), GritLM (w/ gen.), E5-Large (w/ gen.), and Inbedder; and fair re-implementations, notably Causal-EOS and Bidirectional-Avg (Tsai et al., 29 Sep 2025). The fair baselines are emphasized because they control for backbone and training data scale, thereby isolating the effect of the generative refinement mechanism itself.

6. Empirical results, test-time scaling, and stated limitations

On MTEB, the reported headline results are 67.83 average for GIRCSE-Mistral (rank 5) and 67.67 for GIRCSE-Qwen2 (rank 6). These scores are described as competitive with or better than strong LLM embedding baselines such as GritLM: 67.07 and LLM2Vec: 64.57, while using only 0.2M examples, much less than many competing 7B embedding systems. The strongest MTEB result in the comparison table is gte-Qwen2 at 70.72, but GIRCSE is presented as more balanced because it performs much better on instruction-following tasks than many top generic embedders (Tsai et al., 29 Sep 2025).

On the two instruction-following benchmarks, GIRCSE-Mistral achieves 62.97 average (rank 2) and GIRCSE-Qwen2 achieves 62.48 (rank 3). Inbedder attains the best instruction-following score, 77.17, but performs very poorly on MTEB, 50.32, illustrating a strong trade-off. GIRCSE is reported to avoid that collapse while keeping both sides reasonably strong, and the authors state that it achieves the best overall ranking across benchmarks, balancing MTEB and instruction-following quality better than prior methods (Tsai et al., 29 Sep 2025).

The paper further claims that GIRCSE outperforms the fair baselines Causal-EOS and Bidirectional-Avg, as well as competitive LLM embedders including LLM2Vec, GritLM, NV-Embed-v1, E5-Mistral, and SFR-Embedding-2, together with generative baselines such as Inbedder and two-stage generation variants. The stated comparative advantage is strongest when the backbone is matched and training data is limited, because under those conditions gains can more plausibly be attributed to generative refinement itself (Tsai et al., 29 Sep 2025).

A key novelty is the reported emergent test-time scaling property. The experiments evaluate $\phi$0 and find that performance increases monotonically with $\phi$1 for GIRCSE on many tasks, including beyond the training regime, since the model is trained with $\phi$2 but still improves at $\phi$3. By contrast, standard non-generative baselines do not improve with extra generation and often degrade (Tsai et al., 29 Sep 2025). The authors interpret this as evidence that GIRCSE has learned a genuine iterative refinement process rather than a fixed-depth transformation.

The qualitative analysis gives a concrete example. For the input “Why is it so hard to track down this card?”, early generated tokens are content words such as “why,” “hard,” “track,” and “card,” while later tokens evolve into task-specific expansions. Under an intention prompt, examples include “seek,” “elusive,” “question,” and “inquiry”; under an emotion prompt, examples include “frustrating,” “struggle,” “challenging,” and “perseverance” (Tsai et al., 29 Sep 2025). The paper interprets this as evidence that early tokens capture coarse, input-aligned semantics, whereas later tokens refine toward more task- and instruction-specific concepts. This suggests a semantic chain-of-thought analogue for embeddings, although the paper does not claim natural-language transparency for the soft-token process.

The method’s efficiency profile is explicitly qualified. Autoregressive generation is acknowledged to be more expensive than a single encoder pass. The appendix provides a rough theoretical analysis stating that without caching, GIRCSE costs about $\phi$4 times the baseline in compute when $\phi$5. However, peak memory grows only modestly, KV caching reduces inference FLOPs to nearly baseline levels, and the reported FLOP multiplier with caching is about 1.0–1.1× in the experiments, which the paper presents as making test-time scaling practical (Tsai et al., 29 Sep 2025).

The stated limitations are fivefold: higher complexity than standard embeddings; dependence on careful training of soft tokens and ICR; the fact that training–inference length mismatch is not free even though the model generalizes beyond $\phi$6; evaluation that is primarily on English benchmarks; and the lack of direct interpretability of the generative tokens as natural language (Tsai et al., 29 Sep 2025). Taken together, these limitations define the current boundary of the approach: GIRCSE is not a drop-in simplification of encoder-style embedding extraction, but a more complex representation-learning framework whose main empirical distinction is that additional inference-time computation can improve embedding quality rather than merely re-encode the same information.