---
title: Hindi Analogy Test Set (HATS)
url: https://www.emergentmind.com/topics/hindi-analogy-test-set-hats
type: topic
---

# Hindi Analogy Test Set (HATS)

Hindi Analogy Test Set (HATS) is a benchmark for evaluating the reasoning abilities of large language models (LLMs) in Hindi through semantic analogical reasoning. Introduced in "HATS: Hindi Analogy Test Set for Evaluating Reasoning in Large Language Models" [2507.13238], it comprises 405 multiple-choice questions sourced from Indian government and competitive examinations and standardized into the proportional form $A:B::C:?$. HATS was created to address the limited availability of native Hindi reasoning resources, particularly the absence of in-situ semantic analogy benchmarks for Indic languages. Its associated study benchmarks multilingual LLMs under several prompting strategies, including a grounded Chain of Thought formulation motivated by cognitive theories of analogical reasoning, and reports that English-only prompts consistently yield the highest performance even when the analogy itself is in Hindi.

## 1. Conceptual scope and benchmark rationale

HATS is explicitly designed as a **Hindi semantic analogy benchmark** rather than a translated lexical or syntactic analogy set. The motivating premise is that analogy solving is a strong test of reasoning because it requires a model to infer an implicit relation between two concepts, abstract that relation, and transfer it to a new pair. In the proportional format $A:B::C:D$, this is substantially more demanding than surface lexical matching or memorization [2507.13238].

The benchmark targets two gaps. First, native Hindi reasoning resources remain underdeveloped. Second, previously used Hindi analogy datasets are described as being largely derived by translating English analogies and as concentrating on syntactic relations such as singular/plural or inflectional families. Those resources were often used to evaluate word embeddings or Hindi language models trained on corpora, rather than richer semantic reasoning. HATS therefore reorients evaluation toward semantic relation transfer in naturally occurring Hindi questions.

The benchmark’s design further reflects the claim that translated analogy sets do not capture semantic relations that are specific to Hindi usage and Indian examination contexts. HATS instead uses real analogy questions from Indian government and competitive examinations. This gives the benchmark a native and situationally grounded character. A plausible implication is that performance on HATS is less reducible to success on imported English templates than performance on translated analogy datasets.

## 2. Dataset composition, provenance, and format

HATS contains **405 multiple-choice analogy questions**. The questions were scraped using **BeautifulSoup** from **national and state-level administrative service examinations** and their **preparatory materials**, including sources for **UPSC**, **SSC**, **PSC**, **Clerk**, **Defense**, **Railway**, and **Banking** examinations [2507.13238]. The questions are described as **in-situ semantic analogies** intended to assess **aptitude and reasoning abilities**.

The source material appeared in varied formats. The authors state that all questions were standardized into the form
$$
A : B :: X : Y
$$
then converted for model input by replacing $Y$ with a question mark while retaining the original four answer options. The resulting benchmark is therefore a **four-option multiple-choice analogy completion task**.

A representative example is:

> **भोपाल : मध्य प्रदेश :: भुवनेश्वर : ?**

with options **गुजरात**, **उड़ीसा**, **राजस्थान**, and **अरुणाचल प्रदेश**. The correct answer is **उड़ीसा**, since Bhubaneshwar is the capital of Odisha just as Bhopal is the capital of Madhya Pradesh.

Although the study does not provide a formal category inventory for all 405 questions, the worked examples in the appendix show that HATS includes several semantic relation types. These include **capital \(\rightarrow\) state/country**, **instance \(\rightarrow\) category**, **product \(\rightarrow\) process of obtaining/producing**, **tool \(\rightarrow\) function**, and **animal \(\rightarrow\) habitat**. This suggests that HATS is intended to probe relation transfer across heterogeneous semantic classes rather than a narrow lexical template family.

The curation details are intentionally limited. The paper reports scraping, standardization, and preservation of original answer options, but it does **not** provide a detailed annotation schema, inter-annotator agreement, per-category counts, source-by-source distribution, train/dev/test split, or formal filtering heuristics. Everything reported about usage indicates that HATS functions as a **single evaluation set** rather than a training corpus.

## 3. Benchmark tasks, models, and scoring protocol

The experimental study evaluates three multilingual or open LLM families: **Aya-Expanse-8B**, **Llama-3.1-8B / Llama-3.1-8B-Instruct**, and **Gemma-2-9B / Gemma-2-9b-it**. The specific Hugging Face model names are `CohereForAI/aya-expanse-8b`, `meta-Llama/Llama-3.1-8B-Instruct`, and `google/gemma-2-9b-it`. The reported inference configuration uses `max_new_tokens = 1200`, `torch_dtype = torch.float16`, `device_map = "auto"`, `do_sample = False`, and evaluation mode only [2507.13238].

The paper defines multiple evaluation tasks. **Task A** is a probability-based forced-choice setting. The model sees a truncated analogy of the form
$$
A : B :: X :
$$
along with a fixed answer set, and the most likely option is selected via direct probability measurement. Because the authors wished to avoid metalinguistic judgment, they used non-instruct variants in this setting. The reported Task A accuracies are **46.17** for **Llama 3.1-8B**, **42.96** for **Aya Expanse-8B**, and **43.20** for **Gemma 2-9B**.

**Tasks B and C** are generation-based. They use a system prompt, a user prompt, the analogy question, the answer options, and a required terminal answer format. The prompting language is varied under three configurations: **Hindi-only (Hi+Hi)**, with both system and user prompts in Hindi; **English-only (En+En)**, with both prompts in English; and **Mixed (En+Hi)**, with an English system prompt and Hindi user prompt.

To reduce positional bias in multiple-choice generation, the authors apply **cyclic option rotation**. For a question with $n$ options, the evaluation generates $n$ variants by cyclically rotating the answer order, collects the model’s answer to each variant, and uses majority vote. A faithful formalization given in the summary is:
$$
\hat{y}_q = \operatorname{Maj}(r_1,\dots,r_n)
$$
with item-level correctness
$$
\mathrm{score}(q)=
\begin{cases}
1 & \text{if } \hat{y}_q = y_q \\
0 & \text{otherwise}
\end{cases}
$$
and overall accuracy
$$
\mathrm{Accuracy} = \frac{1}{|Q|}\sum_{q \in Q}\mathrm{score}(q)\times 100.
$$

The paper notes that these accuracies are computed **only for valid analogies**. This restriction matters because models sometimes produced invalid responses such as **“I don’t know”** or **“None of the above”** even when valid options were available.

## 4. Prompting strategies and cognitive grounding

The prompting study begins with a basic **0-shot** instruction-following baseline. In English, the system prompt is of the form:

> Complete the analogy: You will end your answer with:  
> `###Final Answer: < Your chosen option>`

A corresponding Hindi version is also used. For Task B, all instructions were presented in the system prompt, and the mixed setting was not evaluated separately because it was effectively equivalent to English-only in practice [2507.13238].

The study then tests several Chain-of-Thought variants. The simplest is **0-shot CoT**, which appends **“Let’s think step by step.”** in English or **“आइये कदम दर कदम सोचें”** in Hindi. More significant is the paper’s main methodological contribution, **Grounded 0-shot Chain of Thought**. Rather than merely asking the model to reason step by step, the prompt imposes a fixed three-stage analogy-solving scaffold grounded in **Minnameier (2010)**:

1. **Identify the specific relationship between the first two terms \((A, B)\)**.
2. **Apply this same relationship to the third term \((C)\)**.
3. **Examine each option and select the one that best completes the analogy**.

The prompt further instructs the model to consider multiple possible relationships before deciding and to be explicit in its reasoning. The authors connect these steps to **abductive structure identification**, **inductive concept mapping**, and **adequacy-based evaluation**. A faithful reconstruction in the summary expresses the relation inference step as
$$
r^\* = \arg\max_{r \in \mathcal{R}} P(r \mid A,B)
$$
followed by option scoring
$$
s_j = \mathrm{Adequacy}\big(r^\*, C, o_j\big)
$$
and final selection
$$
\hat{D} = \arg\max_{o_j \in \mathcal{O}} s_j.
$$

Two few-shot extensions are also evaluated. **Grounded Few-Shot CoT** adds **5 worked-out examples**, generated by **Claude-3.7-Sonnet** in Hindi and then **verified and corrected by a Hindi language expert**. These examples cover the relation types **instance \(\rightarrow\) category**, **product \(\rightarrow\) production process**, **capital \(\rightarrow\) country**, **tool \(\rightarrow\) function**, and **animal \(\rightarrow\) habitat**. **Few-Shot CoT with Translation** implements a three-step pipeline: **Translation** of the Hindi analogy and options into English, **Solution** in English using the grounded method, and **Mapping** of the English answer back to the correct Hindi option. The prompt explicitly restricts the solution phase to English words only.

## 5. Empirical results and benchmark behavior

The central empirical pattern is that **English-only prompts consistently led to the highest performance**, irrespective of prompting strategy [2507.13238]. This is true even though the underlying analogy questions remain Hindi problems. The best configuration for **Aya-Expanse-8B** is **65.67** with **0-shot, En+En**; for **Llama-3.1-8B-Instruct**, **74.56** with **Grounded 0-shot CoT, En+En**; and for **Gemma-2-9b-it**, **79.75** with **Grounded 0-shot CoT, En+En**.

| Model | Task A accuracy (%) | Best prompted result |
|---|---:|---|
| Aya-Expanse-8B | 42.96 | 65.67 with 0-shot, En+En |
| Llama-3.1-8B | 46.17 | 74.56 with Grounded 0-shot CoT, En+En |
| Gemma-2-9B | 43.20 | 79.75 with Grounded 0-shot CoT, En+En |

The paper reports that moving from baseline **0-shot CoT** to **Grounded 0-shot CoT** yields an **average improvement of +0.27 points across all models and settings**. The gain is therefore modest on average, but it is substantial in some cases. In **En+En**, **Gemma** improves from **78.52** to **79.75**, and **Llama** improves from **66.91** to **74.56**. The reported interpretation is that explicit reasoning scaffolds can materially improve analogy performance, especially for stronger models.

**Gemma** is the strongest model overall. The study states that it **consistently outperformed other models** by an **average margin of 11.46 points across all tasks** and showed **minimal performance drop across prompt settings**. By contrast, **Llama** is more prompt-sensitive: it rises from **46.17** in Task A to **74.56** under grounded English prompting. **Aya** is the weakest performer, with a peak of **65.67**, and grounded prompting often hurts it, especially in few-shot settings.

Several negative findings are equally important. **0-shot CoT** is not uniformly beneficial. In **En+En**, Aya drops from **65.67** to **65.43**, Llama from **71.85** to **66.91**, and Gemma from **78.77** to **78.52**. **Few-shot prompting** is not automatically superior either: for **Gemma** in **En+En**, **Grounded 0-shot CoT** reaches **79.75**, whereas **Grounded FS-CoT** reaches **77.28**; for **Llama** in **En+En**, **74.56** under grounded 0-shot is slightly above **74.07** under grounded few-shot. Likewise, **FS Translate-CoT** does not outperform the best direct English prompting. Its scores are **62.46** for Aya, **72.83** for Llama, and **77.04** for Gemma, all below the strongest **En+En Grounded 0-shot CoT** results for Llama and Gemma.

These findings correct several common assumptions. Multilingual instruction following does not imply equivalent reasoning performance across prompt languages. Simple requests to “think step by step” do not reliably improve performance. More examples do not necessarily help. Explicit translation into English does not automatically dominate native-script prompting.

## 6. Error patterns, limitations, and position within analogy-benchmark research

The discussion identifies several recurring failure modes. The first is that models often identify the source relation in $A:B$ but fail to transfer it correctly to $C:D$. This localizes a weakness in **structured analogical mapping** rather than in lexical comprehension alone [2507.13238]. A second failure mode appears in the translation-based setup, where **Aya** and **Llama** sometimes mistranslate crucial terms because of phonetic confusion. The paper’s concrete example is **फूल : माला :: ईंट : ?**, where **ईंट** (“brick”) is misread as **Eat**; the authors state that this occurred **consistently across all 10 sampled failures**. This suggests that translation pipelines introduce additional brittleness in low-resource or script-sensitive settings.

The paper also notes model-specific differences in output language. **Aya** responds in Hindi under Hindi settings and in English under **En+En**. **Llama** often replies in Hindi even in some **En+En** settings for simpler tasks, but switches to English for grounded and few-shot English tasks. **Gemma** notably replies in English even in mixed-like settings and consistently in English under **En+En**. A plausible implication is that prompt-language sensitivity is entangled with response-language preferences and with the robustness of instruction following in Hindi.

The explicit limitations include a **model size limitation**: only **smaller open models in the 8B–9B range** were tested because of **resource and hardware constraints**. The paper also leaves several benchmark-level issues unresolved: there is **no train/dev/test split**, **no category-wise breakdown of all 405 questions**, **no inter-annotator agreement or annotation protocol details**, **no confidence intervals or significance testing**, **no evaluation of larger proprietary frontier models on HATS itself**, and **no deep per-relation error analysis**. The ethics statement adds that the data comes from **publicly available national-level question papers and preparatory material**, and therefore the dataset is described as free from **anonymity concerns**, **obscenities**, and, according to the authors, **stereotyping or bias**.

Within the broader literature on analogy benchmarks for low-resource languages, HATS occupies a position aligned with native, semantically grounded evaluation rather than translation-based replication. The Persian framework "SAT Based Analogy Evaluation Framework for Persian Word Embeddings" [2106.15674] is relevant methodological background because it likewise argues that low-resource languages lack rich semantic benchmarks and that translation is not a satisfactory way to build analogy datasets. That framework uses **hand crafted Persian SAT based analogy data**, a **colloquial test set**, and a broad semantic taxonomy for word-embedding evaluation. HATS differs in task design and model class: it evaluates multilingual LLM reasoning on Hindi multiple-choice analogy completion rather than vector-offset behavior in Persian word embeddings. Even so, both benchmarks converge on the view that native analogy resources are necessary if one wishes to evaluate semantic reasoning rather than shallow morphological regularity.

HATS’s stated contributions are therefore threefold: it introduces a **405-question Hindi semantic analogy test set** built from real Indian examination material; it benchmarks multilingual LLMs under multiple prompting regimes; and it proposes a **grounded Chain-of-Thought** strategy rooted in cognitive theories of analogy. The benchmark further highlights a central mismatch in contemporary multilingual systems: broad multilingual coverage does not yet imply equally strong native-language reasoning, since the best results on Hindi analogies arise when the prompting language is English rather than Hindi.

Source: https://www.emergentmind.com/topics/hindi-analogy-test-set-hats