IndicIFEval-Hi: Multilingual Instruction Benchmark

• The paper introduces a multilingual benchmark that extends the English-centric IFEval framework to objectively evaluate instruction following in 14 Indic languages.
• It constructs a dual dataset with translated-localized prompts (IndicIFEval-Trans) and native, synthetically generated prompts (IndicIFEval-Ground) to assess diverse linguistic constraints.
• Evaluation reveals that while proprietary models generally outperform open-weight ones, significant challenges remain in cross-lingual and precise constraint adherence.

Searching arXiv for the benchmark lineage and related instruction-following evaluation work. IFEval-Hi refers, in the cited usage, to IndicIFEval, a benchmark for verifiable instruction-following evaluation in 14 Indic languages. It extends the English IFEval paradigm to a multilingual, script-diverse, and often low-resource setting by using automatically verifiable, rule-based instructions rather than human or LLM-judge scoring. The benchmark is designed to test whether LLMs can satisfy simple, checkable constraints across Indic languages, with particular attention to the gap between English performance and non-English constrained generation. Its construction combines a translated-and-localized subset derived from IFEval with a native, synthetically generated subset grounded in Indic contexts, and its evaluation emphasizes prompt-level accuracy, instruction-level accuracy, and cross-lingual parity (Jayakumar et al., 25 Feb 2026).

1. Definition and benchmark lineage

IndicIFEval is introduced as a multilingual extension of the English IFEval framework for objective instruction-following evaluation. The underlying premise is that instruction-following benchmarks have been predominantly English-centric, even though instruction adherence in non-English settings is frequently weaker and more difficult to assess reliably. IndicIFEval preserves the IFEval emphasis on verifiable instructions and rule-based evaluation, thereby avoiding the expense, noise, and potential bias associated with human evaluation or LLM-as-a-judge pipelines (Jayakumar et al., 25 Feb 2026).

Within the broader benchmark lineage, IFEval is positioned as a benchmark that evaluates instruction following using deterministic, programmatic evaluation rather than subjective judgments (Dussolle et al., 7 Feb 2025). IndicIFEval inherits that design principle but adapts it to languages with substantial variation in script, morphology, and resource availability. This makes the benchmark not merely a translation exercise, but an attempt to probe whether models can follow explicit constraints in linguistically heterogeneous environments (Jayakumar et al., 25 Feb 2026).

A plausible implication is that the term “IFEval-Hi” can be misleading if interpreted narrowly as a Hindi-only benchmark. In the cited source, the term is associated with IndicIFEval as a whole, while Hindi is one of the 14 covered languages rather than the sole target language (Jayakumar et al., 25 Feb 2026).

2. Dataset composition and construction

IndicIFEval is organized into two complementary subsets. The first, IndicIFEval-Trans, is a translated-and-localized version of IFEval prompts. The authors do not simply translate source prompts mechanically. They inspect the original IFEval prompts, localize culturally specific references where needed, explicitly handle English-specific constraints such as capitalization by adding “Respond in English.” when necessary, extract keywords before translation so that prompt metadata and prompt text remain aligned, and then apply regex-based automatic checks followed by manual fixes for formatting problems. Because some original prompts are too fragile or unsuitable for clean translation, the source pool is reduced from 541 to 490 translated examples per language before verification (Jayakumar et al., 25 Feb 2026).

The second subset, IndicIFEval-Ground, is generated from native Indic contexts rather than English templates. It focuses on the constraint types that are hardest to translate faithfully: Keyword Inclusion, Keyword Prohibition, Keyword Frequency, Paragraph Count, Sentence Count, and First Word. To build this subset, the authors mine keywords from the Sangraha corpus using TF-IDF ranking, retrieve text segments that naturally satisfy a given constraint, and use Gemini-2.5 to generate a prompt that would plausibly elicit that response. They diversify prompt style through one of sixteen generation task types inspired by UltraChat, including news articles, essays, dialogues, summaries, formal rewrites, and creative stories (Jayakumar et al., 25 Feb 2026).

The benchmark covers 14 Indic languages: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Sanskrit, Tamil, Telugu, and Urdu. Across both subsets, the final dataset contains around 800 human-verified examples per language. For the translated subset, there are 321 parallel prompts per language that are correct in all languages, enabling cross-lingual comparison. Human verification is performed by one qualified native speaker per language, who checks semantic correctness, naturalness, and constraint validity, while the paper explicitly notes that single-annotator verification is a limitation (Jayakumar et al., 25 Feb 2026).

3. Instruction taxonomy and evaluation protocol

IndicIFEval retains the original IFEval style of automatically verifiable constraints, but adapts the evaluation stack to Indic scripts and linguistic structures. The paper groups the original 25 IFEval instruction templates into five higher-level categories for analysis: English, Keyword, Format, Length, and Language. The English category includes capitalization-related instructions; Keyword includes inclusion, exclusion, frequency, punctuation-related keyword constraints, and related detectable-content instructions; Format includes JSON formatting, numbered bullet lists, highlighted sections, and multiple responses; Length includes sentence, paragraph, and word counts; and Language specifies the required response language (Jayakumar et al., 25 Feb 2026).

This category structure makes clear that the benchmark is not limited to general semantic compliance. It also tests precise surface realization. In the grounded subset, the emphasis on keyword constraints, sentence and paragraph counts, and first-word constraints reflects the authors’ view that these are especially sensitive to translation artifacts and therefore require native prompt construction (Jayakumar et al., 25 Feb 2026).

Scoring follows the original IFEval setup and reports both prompt-level accuracy and instruction-level accuracy. Prompt-level accuracy is the percentage of prompts for which all constraints are satisfied, while instruction-level accuracy is the percentage of individual instructions satisfied across prompts. The paper emphasizes loose scoring, which tolerates minor formatting discrepancies. For category-wise analysis, it computes accuracy per instruction and then averages across instructions, so the prompt-level versus instruction-level distinction does not affect those category scores (Jayakumar et al., 25 Feb 2026).

To make structural verification reliable for Indic scripts, the benchmark replaces the original NLTK-based sentence and word checks with the Indic NLP Library, which provides better tokenization and sentence segmentation for the target languages. Evaluations are run through the LLM Evaluation Harness and use greedy decoding (Jayakumar et al., 25 Feb 2026).

4. Models evaluated and empirical findings

The benchmark evaluates both open-weight and proprietary models. The open-weight families are Llama (3.1, 3.2, 3.3, and 4 variants), Gemma 3 (1B, 4B, 12B, 27B), Aya-Expanse (8B and 32B), and Qwen 3 (0.6B through 32B). Proprietary models include GPT-5, GPT-5-mini, Gemini-3-Pro, and Gemini-3-Flash. The evaluated systems span roughly 0.5B to 70B+ parameters and include both reasoning and non-reasoning configurations (Jayakumar et al., 25 Feb 2026).

The main empirical pattern is that proprietary models lead overall on IndicIFEval-Trans, with Gemini-3-Pro/Flash and GPT-5 at the top, while Gemma-3-27B-IT is the strongest open-weight model overall. Performance generally improves with model scale, though gains begin to diminish around the 12B–14B range. The smallest models, particularly Llama-3.2-1B and Gemma-3-1B, perform poorly, suggesting a minimum capacity threshold for stable verifiable instruction adherence (Jayakumar et al., 25 Feb 2026).

A notable result is that Gemma shows the smallest gap between English and Indic performance, often substantially smaller than Llama or Aya. Qwen and Llama form a middle tier, while Aya-Expanse tends to show the largest degradation from English, especially in lower-resource languages. This suggests that multilingual grounding quality varies substantially even among high-capacity systems (Jayakumar et al., 25 Feb 2026).

This broader multilingual pattern is consistent with findings from M-IFEval, which showed that model rankings can vary across languages, that Japanese is consistently the hardest language in that benchmark, and that English-only evaluation can mis-rank models for non-English use cases (Dussolle et al., 7 Feb 2025). The connection is methodological rather than dataset-identical: both benchmarks argue that objective, programmatic instruction-following evaluation must extend beyond English.

5. Difficulty structure across languages and constraint types

IndicIFEval reports that task difficulty is highly non-uniform. Models are relatively strong on formatting constraints such as JSON and bullet-list style structure, but they struggle much more with lexical constraints, especially keyword inclusion. The paper explicitly notes that models are much better at excluding forbidden words than at including required ones. This asymmetry indicates that suppressing concepts is easier than generating exact required surface forms (Jayakumar et al., 25 Feb 2026).

The hardest category is cross-lingual tasks, where a prompt is in an Indic language but the required output is English. Performance drops sharply in this setting. Even for high-resource languages such as Hindi, models may continue answering in the prompt language or answer in English while violating capitalization or related English-specific constraints. The paper highlights Gemma as the standout model here, nearly closing the cross-lingual gap in some cases, while also observing that strict evaluation can penalize otherwise reader-helpful additions such as parenthetical translations or transliterations (Jayakumar et al., 25 Feb 2026).

The benchmark also finds substantial variation across languages. Hindi is repeatedly the strongest Indic language, often showing the smallest gap to English and sometimes nearing parity, especially for structural constraints such as format, length, and language control. By contrast, lower-resource or more divergent languages, including Sanskrit, Odia, or Assamese in some settings, show larger disparities, particularly under English-output requirements (Jayakumar et al., 25 Feb 2026).

This suggests that “IFEval-Hi” is important not because Hindi alone defines the benchmark, but because Hindi serves as one of the clearest cases where high-resource Indic instruction-following approaches English performance more closely than other Indic languages. A plausible implication is that Hindi can function as a relatively favorable probe of Indic instruction-following, while the full benchmark reveals how poorly that generalizes to the broader language family (Jayakumar et al., 25 Feb 2026).

6. Relation to adjacent instruction-following benchmarks

IndicIFEval belongs to a rapidly expanding family of benchmarks that decompose instruction-following into distinct, objectively testable dimensions. M-IFEval extends IFEval to French, Japanese, and Spanish, adding language-specific instructions rather than merely translating prompts, and shows that models are often weak at character-, script-, and orthography-level control (Dussolle et al., 7 Feb 2025). IndicIFEval is aligned with this perspective but addresses a larger, script-diverse regional family and places greater emphasis on localization and grounded native prompts (Jayakumar et al., 25 Feb 2026).

IHEval addresses a different axis: the ability of models to follow the instruction hierarchy among system messages, user messages, conversation history, and tool outputs. Its central finding is that models degrade sharply when instructions conflict across priority levels, with the most competitive open-source model achieving only 48% accuracy in resolving such conflicts (Zhang et al., 12 Feb 2025). IndicIFEval does not test hierarchy resolution; instead, it isolates whether models can satisfy verifiable constraints in multilingual generation. The contrast underscores that “instruction following” is not a single capability, but a bundle of separable behaviors.

Speech-IFEval similarly separates speech perception from instruction following in speech-aware LLMs, measuring whether models obey explicit output constraints independently of speech-task correctness (Lu et al., 25 May 2025). IFEval-FC applies the same verifiable-instruction philosophy to function calling, where constraints are embedded in JSON schema descriptions and evaluation is fully algorithmic (Skripko, 22 Sep 2025). Across these variants, the common design principle is the replacement of subjective evaluation with deterministic compliance checks, while each benchmark probes a different failure mode or deployment context.

7. Significance, limitations, and future directions

The central significance of IndicIFEval is that it demonstrates that multilingual constrained generation remains substantially harder than English constrained generation, even when the instructions are simple and automatically checkable. It shows that translation-only multilingual evaluation can miss important failures, because localization matters and grounded native prompts expose weaknesses that translated prompts may hide. In that sense, IndicIFEval is not merely “IFEval in Indic scripts,” but a benchmark for studying instruction following under conditions of script diversity, morphological complexity, and uneven resource availability (Jayakumar et al., 25 Feb 2026).

The benchmark also has explicit limitations. IndicIFEval-Ground covers only a subset of possible constraint types. Human verification relies on a single annotator per language, without inter-annotator agreement. Reasoning-mode experiments are limited to the Qwen family. The authors also identify possible future extensions, including romanization constraints, digit-versus-word numeral formatting, and unit-system constraints (Jayakumar et al., 25 Feb 2026).

More broadly, the benchmark reinforces a recurring conclusion across recent instruction-following research: English-only benchmarks are incomplete, and surface-level task success can obscure failures in exact compliance. IndicIFEval therefore serves both as a multilingual diagnostic instrument and as evidence that progress in instruction following should be measured not only by semantic adequacy, but also by precise, automatically verifiable adherence to explicit constraints in the languages where models are actually deployed (Jayakumar et al., 25 Feb 2026).