---
title: Granite Embedding Models for Retrieval
url: https://www.emergentmind.com/topics/granite-embedding-models
type: topic
---

# Granite Embedding Models for Retrieval

Granite Embedding Models are a family of encoder-based neural architectures developed for information retrieval, enabling both dense and sparse retrieval across text, code, long documents, conversational data, and tabular modalities. Initially introduced in 2025, these models emphasize production-ready deployment for enterprise-scale applications and are distinguished by rigorous governance, public Apache 2.0 licensing, and state-of-the-art (SOTA) retrieval performance across a wide spectrum of academic and industrial benchmarks [2502.20204][2508.21085].

## 1. Model Architectures

Granite Embedding Models encompass several architectural configurations to balance retrieval effectiveness and efficiency. The core designs are:

- **Bi-encoder retrievers:** Independently encode queries and passages into fixed-length embedding vectors, enabling fast approximate nearest neighbor search.
    - *Granite-Encoder-English (base):* 22-layer Transformer, hidden size 768, 149M parameters.
    - *Granite-Encoder-Small-English:* 12 layers, hidden size 384, 47M parameters.
    - *First-gen Granite (base):* 12 layers, hidden size 768 (125M English, 278M multilingual).
    - *Distilled models:* 6 layers, hidden size 384 (30M English, 107M multilingual, and a 30M sparse variant).

- **Cross-encoder reranker:** Processes joint query-document pairs for precise reranking.
    - *Granite-Embedding-Reranker-English-R2:* 22-layer ModernBERT, hidden size 768, context window up to 8192 tokens, “\([CLS]\, q\, [SEP]\, d\)” input with classifier head.

All models integrate rotary positional embeddings (RoPE), with R2 expanding context windows to 8192 tokens via RoPE scaling and masked language modeling (MLM) for context extension. ModernBERT architecture is employed, featuring alternating global attention and FlashAttention-optimized computations [2508.21085].

| Model type           | Layers | Hidden size | Parameters | Context window |
|----------------------|--------|-------------|------------|---------------|
| Bi-encoder (R2 base) | 22     | 768         | 149M       | 8192          |
| Bi-encoder (R2 small)| 12     | 384         | 47M        | 8192          |
| Reranker (R2)        | 22     | 768         | 149M       | 8192          |
| First-gen (base)     | 12     | 768         | 125M       | 512           |
| Distilled (small)    | 6      | 384         | 30/107M    | 512           |

## 2. Training Methodologies and Objectives

Granite models leverage a multistage training pipeline:

- **Retrieval-Oriented Pretraining (RetroMAE):** Moderate masking applied to encoder inputs to generate embeddings; aggressive masking and a 1-layer decoder reconstruct masked tokens using the embedding and remaining tokens. The loss for masked positions \( M \) is:
  $$
  \mathcal{L}_{\mathrm{RetroMAE}} = -\sum_{t \in M} \log P_{\mathrm{dec}}(x_t \mid \mathbf{e}, \tilde x_{\setminus t})
  $$
  [2502.20204]

- **Contrastive Fine-Tuning:** Uses enhanced InfoNCE-style objective with in-batch negatives, hard negatives, and bidirectional scoring:
  $$
  s(u, v) = \frac{1}{\tau} \frac{\mathbf{E}(u) \cdot \mathbf{E}(v)}{\|\mathbf{E}(u)\| \|\mathbf{E}(v)\|}
  $$
  $$
  \mathcal{L}_C = -\frac{1}{N}\sum_{i=1}^N \log \frac{\exp(s(q_i, p_{i0}))}{Z_i}
  $$
  $$
  Z_i = \exp(s(q_i, p_{i0})) + \alpha \sum_{j>0} \exp(s(q_i, p_{ij})) + \beta \sum_{i' \neq i} \exp(s(q_i, q_{i'})) + \gamma \sum_{j>0} \exp(s(p_{i0}, p_{ij}))
  $$
  Adapted for domain and retrieval specifics [2502.20204][2508.21085].

- **Knowledge Distillation:** Trains compact models via soft target distributions from a large Mistral-7B-Instruct teacher:
  $$
  \mathcal{L}_{KD} = - \sum_{i=1}^N \sum_{j=0}^K P_t(q_i, p_{ij}) \log P_s(q_i, p_{ij})
  $$
  Two-stage distillation: weak pairs + in-batch negatives followed by annotated pairs + hard negatives; includes temperature annealing [2502.20204][2508.21085].

- **List-wise Rank Loss (Reranker):** The PListMLE with position weights aligned to NDCG:
  $$
  L_{rank}(z, y) = \sum_{i=1}^n \alpha(i)\left[-z_{y_i} + \log \sum_{k=i}^n e^{z_{y_k}}\right]
  $$
  with $\alpha(i) = 2^{\text{rel}(y_i)} - 1$ [2508.21085].

- **Model Merging (LM-Cocktail):** Blends domain-adapted and general-purpose weights via simple averaging, $\lambda=0.5$, preserving cross-domain retrieval strength.

## 3. Data Resources and Governance

Granite models are distinguished by stringent enterprise data governance and transparent data provenance. All data used is fully licensed, screened for compliance, and documented for content, license, usage, and sensitivity.

**Training sources include:**
- **Text/code:** GneissWeb, Wikipedia, BookCorpus, StackExchange, PubMed, CodePile, Project CodeNet, CoNaLa.
- **Tabular:** WikiTables, ArXiv-tables, PubTables, GitTables, FinTabNet, NQTables, with synthetic summaries from LLMs.
- **Conversational IR:** MultiDoc2Dial, plus synthetic dialog from Mixtral-8x22B [2508.21085].
- **Annotated and synthetic queries:** Generated by models such as Mixtral and Phi-3.5-MoE, with hard negative mining from IBM internal embedders [2502.20204].

**Preprocessing:** Personal or sensitive information is filtered; sequences are tokenized and suitably truncated (e.g., 512 tokens for base models) or chunked for long-context models (e.g., 512-token chunks with 100-token overlap).

## 4. Context Handling and Input Representation

The R2 generation expands support for extended contexts (up to 8192 tokens), achieved by scaling RoPE frequencies and targeted context-extension pretraining. Chunking with overlapping windows and aggregation (average or max-pooling) is used for long-document representation, ensuring robust retrieval for large enterprise documents [2508.21085]. Sequence packing, FlashAttention, and kernel optimizations are adopted to maximize input throughput and maintain comparability across context lengths.

## 5. Performance Across Benchmarks

Granite Embedding Models report competitive or SOTA zero-shot performance across multiple domains:

- **Text retrieval (MTEB-v2, BEIR):** R2 achieves nDCG@10 avg. 53.1–56.4, comparing favorably to gte-ModernBERT and BGE-base given stricter training data governance.
- **Code retrieval (COIR):** R2 reaches nDCG@10 of 54.8 (vs. 71.5 gte-ModernBERT, which trains on task data).
- **Long-context retrieval:** R2 reaches LongEmbed avg. 67.8 (acc.@1 & nDCG@10).
- **Table-IR benchmarks:** Achieve Recall@5/Match@5 of 78.5 avg.
- **Multi-turn conversational:** Recall@5 = 57.6 on MT-RAG (vs. 54.5 BGE-Large).
- **Reranking:** NDCG@10 on BEIR: 55.4 (vs. 56.1 gte-reranker).
- **Internal enterprise tasks:** Outperform or match open-source baselines on all in-house IBM search tasks [2502.20204][2508.21085].

**Efficiency:** R2 base models process 144 docs/sec on an H100 GPU (batch=128, 1873 tokens per doc), 19–44% faster than e5 or BGE models. Small models deliver up to 199 docs/sec at modest accuracy tradeoff (~3–5% nDCG@10) [2508.21085].

## 6. Licensing and Distribution

All Granite Embedding Models are publicly available under the Apache 2.0 license, permitting unrestricted research and commercial use. End-to-end recipes, tokenizers, full provenance, and governance records are hosted with the models at https://huggingface.co/collections/ibm-granite, ensuring transparent, production-safe deployment [2502.20204][2508.21085].

## 7. Implications and Best Practices

Granite Embedding Models integrate advances in pretraining, contrastive learning, knowledge distillation, and architectural engineering to set new deployment standards in enterprise and open-source dense retrieval. Empirical results indicate that 12-layer base models establish new baselines on IBM tasks and match or surpass open-source ensembles given licensing constraints. Six-layer distilled models provide 4× speedup with only moderate retrieval degradation, supporting real-time production needs.

Key techniques validated in these models include CLS pooling, enhanced contrastive objectives, domain-aware hard negative mining, two-stage knowledge distillation, and weight merging via LM-Cocktail. Rigorous data governance and transparent provenance further enable immediate enterprise adoption.

A plausible implication is that the Granite series design and training strategies, given their public license and governance, will inform future open and enterprise embedding model development.

Source: https://www.emergentmind.com/topics/granite-embedding-models