ChatRAG-Hi: Hindi RAG Benchmark

• ChatRAG-Hi is a Hindi benchmark for retrieval-augmented, document-grounded conversational QA, testing models on multi-turn, evidence-based responses.
• It employs a translate-and-verify pipeline with CHRF++ gating and heuristic filtering to ensure high translation quality of context and answers.
• The benchmark and its hierarchical RAG design highlight that retrieval granularity and parent-level aggregation are vital for improving grounded QA accuracy.

Searching arXiv for the benchmark paper and closely related RAG work cited in the provided material. Searching arXiv for "Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis". ChatRAG-Hi is a Hindi benchmark for retrieval-augmented, document-grounded conversational question answering introduced as one of five evaluation datasets in “Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis” (Kamath et al., 27 Aug 2025). Within that suite, it functions as the main stress test for grounded generation in Hindi: models are expected to answer questions from provided context, operate in multi-turn dialogue settings, and produce fact-based responses that remain faithful to underlying documents. In related RAG systems literature, the same label is also used descriptively for a hierarchical, chat-oriented RAG design pattern whose architecture closely tracks parent–child retrieval pipelines for multi-turn conversations, especially the design analyzed in H-RAG (Elchafei et al., 1 May 2026). The term therefore denotes both a concrete Hindi benchmark and, in a broader methodological sense, a class of chat-focused RAG systems concerned with retrieval granularity, grounding, and conversational context.

1. Definition and scope

ChatRAG-Hi is the Hindi adaptation of the English “ChatRAG Bench” and is specifically designed to test Hindi LLMs in retrieval-augmented, document-grounded conversational question answering (Kamath et al., 27 Aug 2025). Within the benchmark suite that also includes IFEval-Hi, MT-Bench-Hi, GSM8K-Hi, and BFCL-Hi, its capability focus is the use of provided context to answer questions in Hindi, including multi-turn dialogue settings in which context evolves over turns, faithful fact-based generation, and handling both short answers and longer free-form responses.

The original English ChatRAG Bench is described as a composite benchmark integrating multiple datasets focused on conversational QA over documents and tables, and the Hindi version preserves that composite nature while adapting it linguistically to Hindi. This means that ChatRAG-Hi is not merely a translation of isolated questions. It is intended to probe RAG behavior under conversational and document-grounded conditions, including settings where a model must answer using passages, dialogues, or documents rather than relying on parametric recall alone.

A plausible implication is that ChatRAG-Hi sits at the intersection of two evaluation axes that are often studied separately: multilingual adaptation and grounded conversational QA. The benchmark therefore exposes failure modes that may not appear in purely reasoning-oriented or instruction-following evaluations, particularly when a model must remain close to retrieved evidence in Hindi rather than generate a plausible but unsupported answer.

2. Dataset composition and construction

ChatRAG-Hi is built by translating the English ChatRAG Bench, which itself integrates ten diverse source datasets (Kamath et al., 27 Aug 2025). The included subsets are listed as INSCIT (450 examples), Doc2Dial (498), QuAC, QReCC, TopiocQA, CoQA, HybriDial, SQA, DoQA (Cooking, Travel, Movies), and ConvFinQA (500 each). In total, ChatRAG-Hi has 5,948 examples. The dataset coverage spans conversational QA, task-oriented dialogue, domain-specific QA, financial table/document QA, and open-domain or structured QA.

Its construction uses a translate-and-verify pipeline with differential treatment of long contexts and short answer segments. All extensive context passages are translated using Google Cloud Platform machine translation and are not filtered post-translation as long as they pass the high-level QA process. Answers and conversation turns are handled more conservatively through a two-tier procedure. In Tier 1, each English answer or turn is translated with GCP and then back-translated to English; the Hindi translation is retained only if the back-translation reaches $\text{CHRF++} \geq 90$. In Tier 2, if $\text{CHRF++} < 90$, the GCP translation is discarded and Llama-3.1-405B is used to translate the short answer or turn with access to the broader GCP-translated Hindi context, after which heuristic filtering removes low-quality outputs.

Around 10% of the final Hindi data is manually checked, and the reported error rates across subsets are in the 1–5% range. The stated rationale for this asymmetric pipeline is that long contexts can tolerate minor stylistic differences, whereas short answers and dialogue turns must remain exactly faithful to the English originals, especially for short answers and numerical values. This suggests that ChatRAG-Hi is not only a benchmark for answer generation, but also a test of whether a model can stay aligned to narrow lexical targets in a language setting where translation noise is non-negligible.

3. Task format and evaluation protocol

A typical ChatRAG-Hi example consists of a user question in Hindi, partial or full supporting context in Hindi, and a single ground-truth answer in Hindi (Kamath et al., 27 Aug 2025). Depending on the source dataset, some instances are single-turn document QA, while others are multi-turn conversational QA in which conversation history and retrieved documents form part of the input. The paper’s evaluation setup treats the task as single-answer generation: the model is given the question and context and must produce an answer in Hindi.

ChatRAG-Hi reports the F1-Score. The benchmark-level score is defined as

$$\text{Score}_{\text{ChatRAG-Hi}} = \frac{1}{N} \sum_{i=1}^{N} F1_i,$$

where $N = 5948$ is the number of test cases and $F1_i$ is the F1 score for example $i$, computed between the model’s Hindi answer and the reference Hindi answer. The paper further gives the usual QA formulation: $$\text{Precision} = \frac{|P \cap G|}{|P|}, \quad \text{Recall} = \frac{|P \cap G|}{|G|},$$ and

$$F1 = \begin{cases} \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} & \text{if } \text{Precision} + \text{Recall} > 0,\ 0 & \text{otherwise.} \end{cases}$$

The use of token-level F1 is important because ChatRAG-Hi mixes single-word, numeric, and longer explanatory answers. A lexical metric does not directly score discourse naturalness, but it does strongly penalize unsupported paraphrase, mistaken numerals, or short-answer deviations. In that sense, the benchmark operationalizes grounded QA conservatively: a fluent answer that drifts away from the reference can score poorly even if it appears superficially reasonable.

4. Comparative results on Hindi LLMs

The benchmarking study evaluates both SLMs and larger LLMs that support Hindi (Kamath et al., 27 Aug 2025). Among SLMs, the reported ChatRAG-Hi F1 scores are 29.89 for Gemma-2-2b-it, 32.60 for Llama-3.2-3B-Instruct, 27.32 for Nemotron-Mini-4B-Instruct, 36.07 for Nemotron-4-Mini-Hindi-4B-Instruct, 38.03 for Llama-3.1-8B-Instruct, 30.15 for Aya-expanse-8b, 40.97 for Gemma-2-9b-it, and 37.48 for Krutrim-2-instruct. Among LLMs, the reported scores are 26.16 for GPT-OSS-20B, 37.92 for Mistral-Small-3.2-24B-Instruct, 40.14 for Sarvam-M, 45.23 for Gemma-3-27b-it, 29.85 for GPT-OSS-120B, 32.47 for Qwen3-235B-A22B-FP8, and 47.46 for Llama-3.1-405B.

The paper explicitly notes that Gemma-2-9b-it is the best ChatRAG-Hi performer within the SLM category, while Llama-3.1-405B is the best overall model on ChatRAG-Hi. It also observes that models that perform strongly on GSM8K-Hi or IFEval-Hi do not automatically lead on ChatRAG-Hi. GPT-OSS-120B, for example, reaches 93.41 accuracy on GSM8K-Hi but only 29.85 F1 on ChatRAG-Hi.

This pattern indicates that Hindi grounded QA is a distinct capability cluster rather than a trivial by-product of scale, math reasoning, or generic instruction-following. The benchmark also highlights several difficulty sources: hallucination versus grounding, sensitivity to short or numeric answers, domain-specific technical phrasing in subsets such as ConvFinQA, and conversational discourse quality in Hindi for datasets such as Doc2Dial, HybriDial, and DoQA. The paper further notes that direct translation often tests a model’s ability to comprehend translated English rather than its native fluency, implying that performance in Hindi RAG is systematically more fragile than in English.

5. Architectural interpretation: hierarchical and chat-oriented RAG

In a broader systems sense, ChatRAG-Hi is also used as a shorthand for a hierarchical, chat-focused RAG design, and this usage maps closely to H-RAG’s parent–child retrieval architecture for SemEval-2026 Task 8 (Elchafei et al., 1 May 2026). H-RAG separates retrieval granularity from generation granularity: documents are segmented into overlapping child chunks using a sliding window of 3 sentences with stride 2, while full documents are preserved as parent units. Each child chunk is embedded with BAAI/bge-large-en-v1.5 and indexed in a Weaviate hybrid vector store, while parent documents are kept in a document store with metadata linking children to parents.

Retrieval combines dense and sparse scoring through

$$s(c, q) = \alpha \, s_{\text{dense}(c, q)} + (1 - \alpha)\, s_{\text{sparse}(c, q)},$$

with $\alpha = 0.7$ in the submitted configuration, followed by embedding-based rescoring with BAAI/bge-reranker-v2-m3. Child relevance is then projected to parents using max pooling,

$\text{CHRF++} < 90$0

and the top parent documents are supplied to the generator. The generation component uses OpenAI GPT-5 in zero-shot mode with temperature $\text{CHRF++} < 90$1 and max_completion_tokens = 4096; query rewriting is also done with GPT-5 at $\text{CHRF++} < 90$2 using the last three Q/A turns to produce a standalone query.

H-RAG achieves an nDCG@5 score of 0.4271 on Task A and a harmonic mean score of 0.3241 on Task C, with RB_agg = 0.2488, RL_F = 0.2703, and RB_llm = 0.6508. Its ablations show that enabling parent-level rescoring improves retrieval by mean gains of +0.0197 nDCG@5 and +0.0108 Recall@5, that variation in mean nDCG@5 across $\text{CHRF++} < 90$3 is less than 0.003, and that performance peaks at $\text{CHRF++} < 90$4 rather than $\text{CHRF++} < 90$5. A plausible implication is that, for chat-oriented RAG systems of the kind informally labeled ChatRAG-Hi, retrieval hierarchy and aggregation strategy matter more than marginal changes in dense–sparse interpolation.

6. Methodological significance, related directions, and limitations

ChatRAG-Hi’s methodological significance lies in how it connects benchmark design, multilingual evaluation, and practical RAG system design. As a benchmark, it offers a replicable blueprint for low-resource-language RAG evaluation through a hybrid translate-and-verify pipeline with CHRF++ gating, heuristic filtering, and human sampling (Kamath et al., 27 Aug 2025). As a systems concept, it points toward hierarchical retrieval, parent-level aggregation, and conversation-aware query handling as central design elements for multi-turn grounded generation (Elchafei et al., 1 May 2026).

Several limitations are explicit. First, ChatRAG-Hi is largely translation-based rather than fully human-authored, so some linguistic or cultural nuance may be absent. Second, it assumes fixed translated contexts and does not separately measure retrieval quality in Hindi; the retrieval side is fixed or implicit from the original dataset. Third, coverage remains incomplete for many India-specific domains such as legal, medical, or governmental texts. Fourth, F1 is purely lexical and does not capture discourse naturalness or subtle semantic differences between near-synonyms in Hindi.

Related RAG research sharpens these limitations rather than removing them. CoRAG shows that collaborative passage stores can improve low-resource QA, but also that hard negatives can degrade collaborative RAG if they are introduced without contrastive handling (Muhamed et al., 2 Apr 2025). KohakuRAG shows that hierarchical document indexing, prompt ordering, retry mechanisms, and ensemble voting can materially stabilize grounded answers and citation quality, with prompt ordering contributing +80% relative improvement, retry mechanisms +69%, and ensemble voting with blank filtering +1.2 percentage points on WattBot 2025 (Yeh et al., 8 Mar 2026). This suggests that future ChatRAG-Hi-like systems could extend beyond benchmarked answer generation toward end-to-end retrieval, richer evidence attribution, conversation summarization for long dialogues, and more explicit controls for faithfulness.

In that broader sense, ChatRAG-Hi is both a concrete Hindi benchmark and a focal point for a wider research agenda: multilingual grounded QA requires not only stronger models, but also better retrieval architectures, stricter evidence use, and evaluation regimes that separate fluency from grounding.