---
title: 'IndicIFEval-Hi: Multilingual Instruction Benchmark'
url: https://www.emergentmind.com/topics/ifeval-hi
type: topic
---

# IndicIFEval-Hi: Multilingual Instruction Benchmark

Searching arXiv for the benchmark lineage and related instruction-following evaluation work.
IFEval-Hi refers, in the cited usage, to **IndicIFEval**, a benchmark for **verifiable instruction-following evaluation in 14 Indic languages**. It extends the English IFEval paradigm to a multilingual, script-diverse, and often low-resource setting by using **automatically verifiable, rule-based instructions** rather than human or LLM-judge scoring. The benchmark is designed to test whether large language models can satisfy simple, checkable constraints across Indic languages, with particular attention to the gap between English performance and non-English constrained generation. Its construction combines a translated-and-localized subset derived from IFEval with a native, synthetically generated subset grounded in Indic contexts, and its evaluation emphasizes **prompt-level accuracy**, **instruction-level accuracy**, and cross-lingual parity [2602.22125].

## 1. Definition and benchmark lineage

IndicIFEval is introduced as a multilingual extension of the English **IFEval** framework for objective instruction-following evaluation. The underlying premise is that instruction-following benchmarks have been predominantly English-centric, even though instruction adherence in non-English settings is frequently weaker and more difficult to assess reliably. IndicIFEval preserves the IFEval emphasis on **verifiable instructions** and **rule-based evaluation**, thereby avoiding the expense, noise, and potential bias associated with human evaluation or LLM-as-a-judge pipelines [2602.22125].

Within the broader benchmark lineage, IFEval is positioned as a benchmark that evaluates instruction following using **deterministic, programmatic evaluation** rather than subjective judgments [2502.04688]. IndicIFEval inherits that design principle but adapts it to languages with substantial variation in script, morphology, and resource availability. This makes the benchmark not merely a translation exercise, but an attempt to probe whether models can follow explicit constraints in linguistically heterogeneous environments [2602.22125].

A plausible implication is that the term “IFEval-Hi” can be misleading if interpreted narrowly as a Hindi-only benchmark. In the cited source, the term is associated with **IndicIFEval as a whole**, while **Hindi** is one of the 14 covered languages rather than the sole target language [2602.22125].

## 2. Dataset composition and construction

IndicIFEval is organized into **two complementary subsets**. The first, **IndicIFEval-Trans**, is a translated-and-localized version of IFEval prompts. The authors do not simply translate source prompts mechanically. They inspect the original IFEval prompts, localize culturally specific references where needed, explicitly handle English-specific constraints such as capitalization by adding “Respond in English.” when necessary, extract keywords before translation so that prompt metadata and prompt text remain aligned, and then apply regex-based automatic checks followed by manual fixes for formatting problems. Because some original prompts are too fragile or unsuitable for clean translation, the source pool is reduced from **541** to **490 translated examples per language** before verification [2602.22125].

The second subset, **IndicIFEval-Ground**, is generated from **native Indic contexts** rather than English templates. It focuses on the constraint types that are hardest to translate faithfully: **Keyword Inclusion, Keyword Prohibition, Keyword Frequency, Paragraph Count, Sentence Count, and First Word**. To build this subset, the authors mine keywords from the **Sangraha** corpus using TF-IDF ranking, retrieve text segments that naturally satisfy a given constraint, and use **Gemini-2.5** to generate a prompt that would plausibly elicit that response. They diversify prompt style through one of sixteen generation task types inspired by UltraChat, including news articles, essays, dialogues, summaries, formal rewrites, and creative stories [2602.22125].

The benchmark covers **14 Indic languages**: **Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Sanskrit, Tamil, Telugu, and Urdu**. Across both subsets, the final dataset contains **around 800 human-verified examples per language**. For the translated subset, there are **321 parallel prompts per language** that are correct in all languages, enabling cross-lingual comparison. Human verification is performed by **one qualified native speaker per language**, who checks semantic correctness, naturalness, and constraint validity, while the paper explicitly notes that single-annotator verification is a limitation [2602.22125].

## 3. Instruction taxonomy and evaluation protocol

IndicIFEval retains the original IFEval style of **automatically verifiable constraints**, but adapts the evaluation stack to Indic scripts and linguistic structures. The paper groups the original **25 IFEval instruction templates** into five higher-level categories for analysis: **English, Keyword, Format, Length, and Language**. The **English** category includes capitalization-related instructions; **Keyword** includes inclusion, exclusion, frequency, punctuation-related keyword constraints, and related detectable-content instructions; **Format** includes JSON formatting, numbered bullet lists, highlighted sections, and multiple responses; **Length** includes sentence, paragraph, and word counts; and **Language** specifies the required response language [2602.22125].

This category structure makes clear that the benchmark is not limited to general semantic compliance. It also tests precise surface realization. In the grounded subset, the emphasis on keyword constraints, sentence and paragraph counts, and first-word constraints reflects the authors’ view that these are especially sensitive to translation artifacts and therefore require native prompt construction [2602.22125].

Scoring follows the original IFEval setup and reports both **prompt-level accuracy** and **instruction-level accuracy**. Prompt-level accuracy is the percentage of prompts for which **all constraints are satisfied**, while instruction-level accuracy is the percentage of individual instructions satisfied across prompts. The paper emphasizes **loose** scoring, which tolerates minor formatting discrepancies. For category-wise analysis, it computes accuracy per instruction and then averages across instructions, so the prompt-level versus instruction-level distinction does not affect those category scores [2602.22125].

To make structural verification reliable for Indic scripts, the benchmark replaces the original NLTK-based sentence and word checks with the **Indic NLP Library**, which provides better tokenization and sentence segmentation for the target languages. Evaluations are run through the **Language Model Evaluation Harness** and use **greedy decoding** [2602.22125].

## 4. Models evaluated and empirical findings

The benchmark evaluates both open-weight and proprietary models. The open-weight families are **Llama** (3.1, 3.2, 3.3, and 4 variants), **Gemma 3** (1B, 4B, 12B, 27B), **Aya-Expanse** (8B and 32B), and **Qwen 3** (0.6B through 32B). Proprietary models include **GPT-5**, **GPT-5-mini**, **Gemini-3-Pro**, and **Gemini-3-Flash**. The evaluated systems span roughly **0.5B to 70B+** parameters and include both reasoning and non-reasoning configurations [2602.22125].

The main empirical pattern is that **proprietary models lead overall** on **IndicIFEval-Trans**, with **Gemini-3-Pro/Flash** and **GPT-5** at the top, while **Gemma-3-27B-IT** is the strongest open-weight model overall. Performance generally improves with model scale, though gains begin to diminish around the **12B–14B** range. The smallest models, particularly **Llama-3.2-1B** and **Gemma-3-1B**, perform poorly, suggesting a minimum capacity threshold for stable verifiable instruction adherence [2602.22125].

A notable result is that **Gemma** shows the **smallest gap between English and Indic performance**, often substantially smaller than **Llama** or **Aya**. **Qwen** and **Llama** form a middle tier, while **Aya-Expanse** tends to show the largest degradation from English, especially in lower-resource languages. This suggests that multilingual grounding quality varies substantially even among high-capacity systems [2602.22125].

This broader multilingual pattern is consistent with findings from **M-IFEval**, which showed that model rankings can vary across languages, that **Japanese is consistently the hardest language** in that benchmark, and that English-only evaluation can mis-rank models for non-English use cases [2502.04688]. The connection is methodological rather than dataset-identical: both benchmarks argue that objective, programmatic instruction-following evaluation must extend beyond English.

## 5. Difficulty structure across languages and constraint types

IndicIFEval reports that task difficulty is highly non-uniform. Models are relatively strong on **formatting constraints** such as JSON and bullet-list style structure, but they struggle much more with **lexical constraints**, especially **keyword inclusion**. The paper explicitly notes that models are much better at **excluding** forbidden words than at **including** required ones. This asymmetry indicates that suppressing concepts is easier than generating exact required surface forms [2602.22125].

The hardest category is **cross-lingual tasks**, where a prompt is in an Indic language but the required output is **English**. Performance drops sharply in this setting. Even for high-resource languages such as **Hindi**, models may continue answering in the prompt language or answer in English while violating capitalization or related English-specific constraints. The paper highlights **Gemma** as the standout model here, nearly closing the cross-lingual gap in some cases, while also observing that strict evaluation can penalize otherwise reader-helpful additions such as parenthetical translations or transliterations [2602.22125].

The benchmark also finds substantial variation across languages. **Hindi** is repeatedly the strongest Indic language, often showing the smallest gap to English and sometimes nearing parity, especially for structural constraints such as format, length, and language control. By contrast, lower-resource or more divergent languages, including **Sanskrit, Odia, or Assamese** in some settings, show larger disparities, particularly under English-output requirements [2602.22125].

This suggests that “IFEval-Hi” is important not because Hindi alone defines the benchmark, but because Hindi serves as one of the clearest cases where high-resource Indic instruction-following approaches English performance more closely than other Indic languages. A plausible implication is that Hindi can function as a relatively favorable probe of Indic instruction-following, while the full benchmark reveals how poorly that generalizes to the broader language family [2602.22125].

## 6. Relation to adjacent instruction-following benchmarks

IndicIFEval belongs to a rapidly expanding family of benchmarks that decompose instruction-following into distinct, objectively testable dimensions. **M-IFEval** extends IFEval to **French, Japanese, and Spanish**, adding language-specific instructions rather than merely translating prompts, and shows that models are often weak at **character-, script-, and orthography-level control** [2502.04688]. IndicIFEval is aligned with this perspective but addresses a larger, script-diverse regional family and places greater emphasis on localization and grounded native prompts [2602.22125].

**IHEval** addresses a different axis: the ability of models to follow the **instruction hierarchy** among **system messages, user messages, conversation history, and tool outputs**. Its central finding is that models degrade sharply when instructions conflict across priority levels, with the most competitive open-source model achieving only **48% accuracy** in resolving such conflicts [2502.08745]. IndicIFEval does not test hierarchy resolution; instead, it isolates whether models can satisfy verifiable constraints in multilingual generation. The contrast underscores that “instruction following” is not a single capability, but a bundle of separable behaviors.

**Speech-IFEval** similarly separates **speech perception** from **instruction following** in speech-aware language models, measuring whether models obey explicit output constraints independently of speech-task correctness [2505.19037]. **IFEval-FC** applies the same verifiable-instruction philosophy to **function calling**, where constraints are embedded in JSON schema descriptions and evaluation is fully algorithmic [2509.18420]. Across these variants, the common design principle is the replacement of subjective evaluation with **deterministic compliance checks**, while each benchmark probes a different failure mode or deployment context.

## 7. Significance, limitations, and future directions

The central significance of IndicIFEval is that it demonstrates that **multilingual constrained generation remains substantially harder than English constrained generation**, even when the instructions are simple and automatically checkable. It shows that translation-only multilingual evaluation can miss important failures, because **localization matters** and **grounded native prompts** expose weaknesses that translated prompts may hide. In that sense, IndicIFEval is not merely “IFEval in Indic scripts,” but a benchmark for studying instruction following under conditions of script diversity, morphological complexity, and uneven resource availability [2602.22125].

The benchmark also has explicit limitations. **IndicIFEval-Ground** covers only a subset of possible constraint types. Human verification relies on **a single annotator per language**, without inter-annotator agreement. Reasoning-mode experiments are limited to the **Qwen** family. The authors also identify possible future extensions, including **romanization constraints**, **digit-versus-word numeral formatting**, and **unit-system constraints** [2602.22125].

More broadly, the benchmark reinforces a recurring conclusion across recent instruction-following research: English-only benchmarks are incomplete, and surface-level task success can obscure failures in exact compliance. IndicIFEval therefore serves both as a multilingual diagnostic instrument and as evidence that progress in instruction following should be measured not only by semantic adequacy, but also by precise, automatically verifiable adherence to explicit constraints in the languages where models are actually deployed [2602.22125].

Source: https://www.emergentmind.com/topics/ifeval-hi