Hindi Analogy Test Set (HATS)

• The paper introduces a 405-question Hindi semantic analogy benchmark built from authentic Indian exam materials to test reasoning abilities.
• HATS standardizes natural analogies into a proportional A:B::C:? format, focusing on semantic relation transfer over lexical similarity.
• Results show that English prompts yield higher accuracy, underscoring challenges in native-language reasoning for multilingual models.

Hindi Analogy Test Set (HATS) is a benchmark for evaluating the reasoning abilities of LLMs in Hindi through semantic analogical reasoning. Introduced in "HATS: Hindi Analogy Test Set for Evaluating Reasoning in LLMs" (Gupta et al., 17 Jul 2025), it comprises 405 multiple-choice questions sourced from Indian government and competitive examinations and standardized into the proportional form $A:B::C:?$. HATS was created to address the limited availability of native Hindi reasoning resources, particularly the absence of in-situ semantic analogy benchmarks for Indic languages. Its associated study benchmarks multilingual LLMs under several prompting strategies, including a grounded Chain of Thought formulation motivated by cognitive theories of analogical reasoning, and reports that English-only prompts consistently yield the highest performance even when the analogy itself is in Hindi.

1. Conceptual scope and benchmark rationale

HATS is explicitly designed as a Hindi semantic analogy benchmark rather than a translated lexical or syntactic analogy set. The motivating premise is that analogy solving is a strong test of reasoning because it requires a model to infer an implicit relation between two concepts, abstract that relation, and transfer it to a new pair. In the proportional format $A:B::C:D$, this is substantially more demanding than surface lexical matching or memorization (Gupta et al., 17 Jul 2025).

The benchmark targets two gaps. First, native Hindi reasoning resources remain underdeveloped. Second, previously used Hindi analogy datasets are described as being largely derived by translating English analogies and as concentrating on syntactic relations such as singular/plural or inflectional families. Those resources were often used to evaluate word embeddings or Hindi LLMs trained on corpora, rather than richer semantic reasoning. HATS therefore reorients evaluation toward semantic relation transfer in naturally occurring Hindi questions.

The benchmark’s design further reflects the claim that translated analogy sets do not capture semantic relations that are specific to Hindi usage and Indian examination contexts. HATS instead uses real analogy questions from Indian government and competitive examinations. This gives the benchmark a native and situationally grounded character. A plausible implication is that performance on HATS is less reducible to success on imported English templates than performance on translated analogy datasets.

2. Dataset composition, provenance, and format

HATS contains 405 multiple-choice analogy questions. The questions were scraped using BeautifulSoup from national and state-level administrative service examinations and their preparatory materials, including sources for UPSC, SSC, PSC, Clerk, Defense, Railway, and Banking examinations (Gupta et al., 17 Jul 2025). The questions are described as in-situ semantic analogies intended to assess aptitude and reasoning abilities.

The source material appeared in varied formats. The authors state that all questions were standardized into the form

$A : B :: X : Y$

then converted for model input by replacing $Y$ with a question mark while retaining the original four answer options. The resulting benchmark is therefore a four-option multiple-choice analogy completion task.

A representative example is:

भोपाल : मध्य प्रदेश :: भुवनेश्वर : ?

with options गुजरात, उड़ीसा, राजस्थान, and अरुणाचल प्रदेश. The correct answer is उड़ीसा, since Bhubaneshwar is the capital of Odisha just as Bhopal is the capital of Madhya Pradesh.

Although the study does not provide a formal category inventory for all 405 questions, the worked examples in the appendix show that HATS includes several semantic relation types. These include capital $\rightarrow$ state/country, instance $\rightarrow$ category, product $\rightarrow$ process of obtaining/producing, tool $\rightarrow$ function, and animal $\rightarrow$ habitat. This suggests that HATS is intended to probe relation transfer across heterogeneous semantic classes rather than a narrow lexical template family.

The curation details are intentionally limited. The paper reports scraping, standardization, and preservation of original answer options, but it does not provide a detailed annotation schema, inter-annotator agreement, per-category counts, source-by-source distribution, train/dev/test split, or formal filtering heuristics. Everything reported about usage indicates that HATS functions as a single evaluation set rather than a training corpus.

3. Benchmark tasks, models, and scoring protocol

The experimental study evaluates three multilingual or open LLM families: Aya-Expanse-8B, Llama-3.1-8B / Llama-3.1-8B-Instruct, and Gemma-2-9B / Gemma-2-9b-it. The specific Hugging Face model names are CohereForAI/aya-expanse-8b, meta-Llama/Llama-3.1-8B-Instruct, and google/gemma-2-9b-it. The reported inference configuration uses max_new_tokens = 1200, torch_dtype = torch.float16, device_map = "auto", do_sample = False, and evaluation mode only (Gupta et al., 17 Jul 2025).

The paper defines multiple evaluation tasks. Task A is a probability-based forced-choice setting. The model sees a truncated analogy of the form

$A : B :: X :$

along with a fixed answer set, and the most likely option is selected via direct probability measurement. Because the authors wished to avoid metalinguistic judgment, they used non-instruct variants in this setting. The reported Task A accuracies are 46.17 for Llama 3.1-8B, 42.96 for Aya Expanse-8B, and 43.20 for Gemma 2-9B.

Tasks B and C are generation-based. They use a system prompt, a user prompt, the analogy question, the answer options, and a required terminal answer format. The prompting language is varied under three configurations: Hindi-only (Hi+Hi), with both system and user prompts in Hindi; English-only (En+En), with both prompts in English; and Mixed (En+Hi), with an English system prompt and Hindi user prompt.

To reduce positional bias in multiple-choice generation, the authors apply cyclic option rotation. For a question with $A:B::C:D$0 options, the evaluation generates $A:B::C:D$1 variants by cyclically rotating the answer order, collects the model’s answer to each variant, and uses majority vote. A faithful formalization given in the summary is:

$A:B::C:D$2

with item-level correctness

$A:B::C:D$3

and overall accuracy

$A:B::C:D$4

The paper notes that these accuracies are computed only for valid analogies. This restriction matters because models sometimes produced invalid responses such as “I don’t know” or “None of the above” even when valid options were available.

4. Prompting strategies and cognitive grounding

The prompting study begins with a basic 0-shot instruction-following baseline. In English, the system prompt is of the form:

Complete the analogy: You will end your answer with: ###Final Answer: < Your chosen option>

A corresponding Hindi version is also used. For Task B, all instructions were presented in the system prompt, and the mixed setting was not evaluated separately because it was effectively equivalent to English-only in practice (Gupta et al., 17 Jul 2025).

The study then tests several Chain-of-Thought variants. The simplest is 0-shot CoT, which appends “Let’s think step by step.” in English or “आइये कदम दर कदम सोचें” in Hindi. More significant is the paper’s main methodological contribution, Grounded 0-shot Chain of Thought. Rather than merely asking the model to reason step by step, the prompt imposes a fixed three-stage analogy-solving scaffold grounded in Minnameier (2010):

1. Identify the specific relationship between the first two terms $A:B::C:D$5.
2. Apply this same relationship to the third term $A:B::C:D$6.
3. Examine each option and select the one that best completes the analogy.

The prompt further instructs the model to consider multiple possible relationships before deciding and to be explicit in its reasoning. The authors connect these steps to abductive structure identification, inductive concept mapping, and adequacy-based evaluation. A faithful reconstruction in the summary expresses the relation inference step as

$A:B::C:D$7

followed by option scoring

$A:B::C:D$8

and final selection

$A:B::C:D$9

Two few-shot extensions are also evaluated. Grounded Few-Shot CoT adds 5 worked-out examples, generated by Claude-3.7-Sonnet in Hindi and then verified and corrected by a Hindi language expert. These examples cover the relation types instance $A : B :: X : Y$0 category, product $A : B :: X : Y$1 production process, capital $A : B :: X : Y$2 country, tool $A : B :: X : Y$3 function, and animal $A : B :: X : Y$4 habitat. Few-Shot CoT with Translation implements a three-step pipeline: Translation of the Hindi analogy and options into English, Solution in English using the grounded method, and Mapping of the English answer back to the correct Hindi option. The prompt explicitly restricts the solution phase to English words only.

5. Empirical results and benchmark behavior

The central empirical pattern is that English-only prompts consistently led to the highest performance, irrespective of prompting strategy (Gupta et al., 17 Jul 2025). This is true even though the underlying analogy questions remain Hindi problems. The best configuration for Aya-Expanse-8B is 65.67 with 0-shot, En+En; for Llama-3.1-8B-Instruct, 74.56 with Grounded 0-shot CoT, En+En; and for Gemma-2-9b-it, 79.75 with Grounded 0-shot CoT, En+En.

Model	Task A accuracy (%)	Best prompted result
Aya-Expanse-8B	42.96	65.67 with 0-shot, En+En
Llama-3.1-8B	46.17	74.56 with Grounded 0-shot CoT, En+En
Gemma-2-9B	43.20	79.75 with Grounded 0-shot CoT, En+En

The paper reports that moving from baseline 0-shot CoT to Grounded 0-shot CoT yields an average improvement of +0.27 points across all models and settings. The gain is therefore modest on average, but it is substantial in some cases. In En+En, Gemma improves from 78.52 to 79.75, and Llama improves from 66.91 to 74.56. The reported interpretation is that explicit reasoning scaffolds can materially improve analogy performance, especially for stronger models.

Gemma is the strongest model overall. The study states that it consistently outperformed other models by an average margin of 11.46 points across all tasks and showed minimal performance drop across prompt settings. By contrast, Llama is more prompt-sensitive: it rises from 46.17 in Task A to 74.56 under grounded English prompting. Aya is the weakest performer, with a peak of 65.67, and grounded prompting often hurts it, especially in few-shot settings.

Several negative findings are equally important. 0-shot CoT is not uniformly beneficial. In En+En, Aya drops from 65.67 to 65.43, Llama from 71.85 to 66.91, and Gemma from 78.77 to 78.52. Few-shot prompting is not automatically superior either: for Gemma in En+En, Grounded 0-shot CoT reaches 79.75, whereas Grounded FS-CoT reaches 77.28; for Llama in En+En, 74.56 under grounded 0-shot is slightly above 74.07 under grounded few-shot. Likewise, FS Translate-CoT does not outperform the best direct English prompting. Its scores are 62.46 for Aya, 72.83 for Llama, and 77.04 for Gemma, all below the strongest En+En Grounded 0-shot CoT results for Llama and Gemma.

These findings correct several common assumptions. Multilingual instruction following does not imply equivalent reasoning performance across prompt languages. Simple requests to “think step by step” do not reliably improve performance. More examples do not necessarily help. Explicit translation into English does not automatically dominate native-script prompting.

6. Error patterns, limitations, and position within analogy-benchmark research

The discussion identifies several recurring failure modes. The first is that models often identify the source relation in $A : B :: X : Y$5 but fail to transfer it correctly to $A : B :: X : Y$6. This localizes a weakness in structured analogical mapping rather than in lexical comprehension alone (Gupta et al., 17 Jul 2025). A second failure mode appears in the translation-based setup, where Aya and Llama sometimes mistranslate crucial terms because of phonetic confusion. The paper’s concrete example is फूल : माला :: ईंट : ?, where ईंट (“brick”) is misread as Eat; the authors state that this occurred consistently across all 10 sampled failures. This suggests that translation pipelines introduce additional brittleness in low-resource or script-sensitive settings.

The paper also notes model-specific differences in output language. Aya responds in Hindi under Hindi settings and in English under En+En. Llama often replies in Hindi even in some En+En settings for simpler tasks, but switches to English for grounded and few-shot English tasks. Gemma notably replies in English even in mixed-like settings and consistently in English under En+En. A plausible implication is that prompt-language sensitivity is entangled with response-language preferences and with the robustness of instruction following in Hindi.

The explicit limitations include a model size limitation: only smaller open models in the 8B–9B range were tested because of resource and hardware constraints. The paper also leaves several benchmark-level issues unresolved: there is no train/dev/test split, no category-wise breakdown of all 405 questions, no inter-annotator agreement or annotation protocol details, no confidence intervals or significance testing, no evaluation of larger proprietary frontier models on HATS itself, and no deep per-relation error analysis. The ethics statement adds that the data comes from publicly available national-level question papers and preparatory material, and therefore the dataset is described as free from anonymity concerns, obscenities, and, according to the authors, stereotyping or bias.

Within the broader literature on analogy benchmarks for low-resource languages, HATS occupies a position aligned with native, semantically grounded evaluation rather than translation-based replication. The Persian framework "SAT Based Analogy Evaluation Framework for Persian Word Embeddings" (Mahmoudi et al., 2021) is relevant methodological background because it likewise argues that low-resource languages lack rich semantic benchmarks and that translation is not a satisfactory way to build analogy datasets. That framework uses hand crafted Persian SAT based analogy data, a colloquial test set, and a broad semantic taxonomy for word-embedding evaluation. HATS differs in task design and model class: it evaluates multilingual LLM reasoning on Hindi multiple-choice analogy completion rather than vector-offset behavior in Persian word embeddings. Even so, both benchmarks converge on the view that native analogy resources are necessary if one wishes to evaluate semantic reasoning rather than shallow morphological regularity.

HATS’s stated contributions are therefore threefold: it introduces a 405-question Hindi semantic analogy test set built from real Indian examination material; it benchmarks multilingual LLMs under multiple prompting regimes; and it proposes a grounded Chain-of-Thought strategy rooted in cognitive theories of analogy. The benchmark further highlights a central mismatch in contemporary multilingual systems: broad multilingual coverage does not yet imply equally strong native-language reasoning, since the best results on Hindi analogies arise when the prompting language is English rather than Hindi.