IFEval: Instruction-Following Evaluation

• IFEval is a benchmark for measuring literal compliance with explicit, programmatically verifiable instructions, such as word limits, required keywords, JSON formatting, bullet counts, capitalization, and exact endings.
• IFEval separates instruction-level accuracy from prompt-level accuracy and reports strict and loose scores, helping researchers distinguish partial compliance from satisfying every constraint in a multi-instruction prompt.
• IFEval supports practical evaluation of LLMs across languages, images, audio, code, and function calling, but its narrow scope means scores should be paired with assessments of factuality, reasoning, safety, usefulness, and evaluator reliability.

IFEval, or Instruction-Following Eval, is a benchmark for measuring whether LLMs comply with explicit natural-language constraints that can be checked by deterministic programs. Introduced by Zhou et al. in “Instruction-Following Evaluation for LLMs” (Zhou et al., 2023), it separates literal instruction compliance from broader response quality, factuality, reasoning, helpfulness, and stylistic judgment. Its prompts combine ordinary generation tasks with verifiable requirements concerning lexical content, length, language, formatting, structure, capitalization, punctuation, and response boundaries.

1. Purpose and conceptual foundations

Instruction following is a central capability of instruction-tuned LLMs, but its evaluation is complicated by the difference between producing fluent text and satisfying every explicit requirement. A response may be relevant and well written while violating a word limit, omitting a required phrase, using a prohibited character, returning invalid JSON, or failing to satisfy one constraint among several.

IFEval was designed in response to limitations in two common evaluation paradigms. Human evaluation is expensive, slow, difficult to scale, subjective, and vulnerable to annotator disagreement and evaluator bias. LLM-based evaluation, or “LLM-as-a-judge,” depends on the evaluator’s own abilities and preferences and can be unstable or biased. IFEval instead concentrates on instructions whose compliance can be determined by relatively simple, interpretable, and deterministic procedures (Zhou et al., 2023).

A verifiable instruction is an atomic instruction for which compliance can be assessed programmatically. Examples include:

• writing between 450 and 500 words;
• mentioning AI at least three times;
• avoiding commas;
• returning the entire response in JSON;
• ending with an exact phrase;
• producing exactly four bullet points.

IFEval measures a narrow but reproducible form of instruction following: literal compliance with explicit, mechanically checkable constraints. It does not determine whether a response is insightful, persuasive, empathetic, factually comprehensive, culturally appropriate, safe, or generally useful unless those properties have been reduced to deterministic checks.

The benchmark is also compositional. A prompt can contain several simultaneous instructions, so a model may satisfy the main task while violating a secondary requirement. This distinction motivates the separation between instruction-level and prompt-level accuracy.

2. Instruction taxonomy and dataset

The original IFEval dataset contains 541 prompts, each containing one or more verifiable instructions. The paper initially describes the collection as “around 500 prompts” and identifies 25 instruction types (Zhou et al., 2023). These types span lexical, linguistic, structural, formatting, casing, boundary, and punctuation constraints.

Group	Instruction types
Keywords	Include Keywords; Keyword Frequency; Forbidden Words; Letter Frequency
Language	Response Language
Length	Number of Paragraphs; Number of Words; Number of Sentences; paragraph count plus first word in a specified paragraph
Detectable content	Postscript; Number of Placeholders
Formatting	Number of Bullets; Title; Choose From; Minimum Number of Highlighted Sections; Multiple Sections; JSON Format
Combination	Repeat Prompt; Two Responses
Case	All Uppercase; All Lowercase; Frequency of All-Capital Words
Boundaries	End Checker; Quotation
Punctuation	No Commas

The benchmark includes both single-constraint and multi-constraint prompts. The multilingual extension reports the original English composition as:

• 305 prompts with one instruction;
• 179 prompts with two instructions;
• 57 prompts with three instructions.

The prompts were constructed in four stages. Base prompts were generated with one to three randomly selected verifiable instructions; few-shot procedures were used to remove illogical or conflicting prompts; prompts were rephrased to diversify surface wording; and the resulting items were manually inspected and edited. Parameters and wording were varied so that models were evaluated on underlying requirements rather than a single memorized formulation (Dussolle et al., 7 Feb 2025).

IFEval’s code and data, including prompts and verification tools, were publicly released through the Google Research project repository associated with the original paper (Zhou et al., 2023).

3. Verification and scoring

For a response $r$ and instruction $i$, IFEval defines a binary satisfaction function:

$$\operatorname{is\_followed}(r,i)= \begin{cases} 1, & \text{if } r \text{ satisfies } i,\ 0, & \text{otherwise}. \end{cases}$$

Instruction-specific programs and heuristics count words, sentences, paragraphs, characters, keywords, bullets, sections, placeholders, capitalization patterns, and punctuation. They also check valid JSON, response language, exact boundary strings, and related structural properties.

IFEval reports strict and loose evaluation.

Strict verification applies the checker directly to the original output. It is literal and conservative: an exact ending can fail if the model adds a period, Markdown markers, or an additional sentence after the required phrase.

Loose verification applies a set of transformations before checking. The transformation set contains eight variants formed from:

• the identity transformation;
• removal of common Markdown font markers such as * and **;
• removal of the first line;
• removal of the last line;
• combinations of these transformations.

Formally, if $\mathcal{T}$ is this set:

$$\operatorname{is\_followed}_{\mathrm{loose}}(r,i) = \max_{t\in\mathcal{T}} \operatorname{is\_followed}(t(r),i).$$

Loose evaluation can prevent superficial formatting differences from producing false negatives. It can also create false positives: deleting a first or last line may make an otherwise noncompliant response satisfy a word-count or boundary condition. Strict and loose scores are therefore complementary.

IFEval reports four principal metrics:

1. Prompt-level strict accuracy
2. Instruction-level strict accuracy
3. Prompt-level loose accuracy
4. Instruction-level loose accuracy

If prompt $p$ contains $m_p$ instructions and $y_{p,j}$ indicates whether instruction $j$ passes, instruction-level accuracy is:

$$\operatorname{Acc}_{\mathrm{inst}} = \frac{\sum_{p=1}^{P}\sum_{j=1}^{m_p}y_{p,j}} {\sum_{p=1}^{P}m_p}.$$

Prompt-level accuracy requires every instruction in a prompt to pass:

$i$0

Prompt-level accuracy is generally lower because one failed constraint invalidates the complete prompt. Instruction-level accuracy is more diagnostic because it identifies partial compliance and permits analysis of individual instruction types.

4. Baseline experiments and empirical behavior

The original study evaluated two commercially available models: GPT-4 and PaLM 2 Small (PaLM 2 S). GPT-4 responses were collected in November 2023 and PaLM 2 S responses in August 2023. The authors cautioned that the models were not directly comparable because of differences in model scale and parameter count; the experiment was intended primarily as a baseline demonstration rather than a controlled comparison (Zhou et al., 2023).

Model	Prompt strict	Instruction strict	Prompt loose	Instruction loose
GPT-4	76.89%	83.57%	79.30%	85.37%
PaLM 2 S	43.07%	55.76%	46.95%	59.11%

GPT-4 outperformed PaLM 2 S on all four measures. Both models performed better at the instruction level than at the prompt level, demonstrating that satisfying individual requirements is easier than satisfying their conjunction. Loose scores exceeded strict scores for both models, consistent with strict checking producing false negatives from Markdown, introductory text, concluding remarks, and other superficial variations.

IFEval particularly exposes failures involving exact formatting, multiple simultaneous constraints, case restrictions, prohibited words, repetition, and numerical requirements such as word and sentence counts. A response can be semantically appropriate yet fail because it contains one forbidden comma, uses the wrong number of paragraphs, places a required word in the wrong position, or adds text after an exact ending.

Subsequent work has used IFEval as a diagnostic benchmark for training and inference methods. SmolTulu-DPO-1130, an instruction-tuned 1.7-billion-parameter model based on SmolLM2-1.7B, reported 67.7 on a metric labeled “IFEval (Average),” compared with 56.7 for SmolLM2-1.7B-Instruct (Alrashed, 2024). The paper does not establish whether this average is instruction-level, prompt-level, strict, loose, or another aggregate, so the value should not be directly equated with any official IFEval metric.

IFDecorator reported 87.43 on “IFEval (Prompt Strict)” for Qwen2.5-32B-Instruct-IFDecorator, compared with 79.48 for the base model, an increase of 7.95 percentage points (Guo et al., 6 Aug 2025). Because the paper does not state the IFEval release, denominator, evaluator version, or complete scoring implementation, the result is best described as the authors’ strict prompt-level score under their reported protocol.

IFEval has also exposed optimization and reasoning trade-offs. Explicit chain-of-thought prompting reduced performance for 13 of 14 models in one study, with Meta-Llama 3 8B Instruct declining from 75.2% to 59.0% (Li et al., 16 May 2025). Reasoning sometimes helped with formatting, repetition, and exact endings, but often introduced unnecessary content, punctuation, explanations, or reasoning text that violated global constraints. This suggests that reasoning can improve local planning while weakening persistent constraint maintenance.

5. Extensions to other languages, modalities, and technical domains

IFEval has become a foundation for extensions that preserve its emphasis on explicit constraints while changing the language, input modality, or output domain.

Multilingual evaluation

M-IFEval extends IFEval to French, Japanese, and Spanish, combining translated general constraints with language-specific instructions (Dussolle et al., 7 Feb 2025). Its datasets contain:

• English: 541 prompts and 25 instruction types;
• Spanish: 115 prompts and 30 instruction types;
• French: 235 prompts and 30 instruction types;
• Japanese: 172 prompts and 33 instruction types.

Language-specific constraints include Spanish ñ, ü, accented-word frequency, interrogation marks, and exclamation marks; French accents, œ, ç, informal address, and digit restrictions; and Japanese furigana, kanji, katakana, hiragana, kansūji, taigen-dome, unified sentence endings, and Japanese punctuation.

M-IFEval found that model rankings varied by language. o1 achieved the highest mean across Spanish, French, and Japanese at 86.6 strict score, while Sonnet achieved the highest Japanese score at 77.0. GPT-4o led English at 88.6 but was not the strongest model on the multilingual extension. Language-specific character and script constraints were substantially harder than generic formatting requirements.

CL-IFEval translated the 541 English prompts into French, Spanish, Hindi, Arabic, and Yoruba (Ojewale et al., 25 Jun 2025). Its results showed lower strict prompt-level accuracy outside English, with Yoruba consistently poor and Qwen3-8B the strongest open-weight model across most listed languages. The paper emphasizes that static multilingual benchmarks such as Belebele and M-MMLU do not necessarily predict functional instruction following. Translation can alter word boundaries, capitalization, keyword identity, cultural relevance, and evaluator behavior.

Marco-Bench-MIF localized IFEval across 30 languages and 541 items per language, for a stated total of 16,230 localized instruction–response pairs (Zeng et al., 16 Jul 2025). Its construction combines Google Translate, professional bilingual correction, LLM validation, and human review. It modifies case constraints for scripts without uppercase/lowercase distinctions and adapts culture-specific names, scenarios, and references. The authors report 25–35 percentage-point gaps between high- and low-resource languages and 7–22% differences between machine-translated and localized evaluation, although the precise meaning of some aggregate comparisons depends on the reported metric.

IFEval-Hi provides an in-house Hindi adaptation with 848 samples and 22 retained instruction categories (Kamath et al., 27 Aug 2025). Five Hindi specialists created culturally grounded prompts involving Indian history, philosophy, festivals, art forms, geography, and social norms. The benchmark removes categories judged unsuitable for Hindi, including “Change Cases.” Its highest reported score was 73.86 for GPT-OSS-120B, while Gemma-2-9B-it was the strongest model in the paper’s SLM grouping at 61.79.

Multimodal and speech extensions

MM-IFEval extends instruction following to images (Ding et al., 10 Apr 2025). It contains 400 human-annotated questions:

• 300 compose-level questions, focused on output form and content;
• 100 perception-level questions, requiring image-grounded reasoning.

The benchmark contains 32 constraint subcategories and averages 5.1 constraints per question. Its evaluation combines rule-based verification, direct LLM judgment, and comparative LLM judgment. Compose-level constraints include JSON, paragraph counts, keywords, styles, roles, tones, and rhetorical devices. Perception-level constraints require the model to locate and interpret visual information.

The paper also reports transfer to text-only IFEval. For Qwen2-VL-7B, MM-IFDPO-23k increased the reported IFEval average from 47.4 to 59.7, an absolute gain of 12.3 percentage points. This score refers to original text-only IFEval, not MM-IFEval.

IFEval-Audio adapts the paradigm to audio-LLMs (Gao et al., 22 May 2025). Its 280 audio–instruction–answer triples cover Content, Capitalization, Symbol, List Structure, Length, and Format. Each example combines audio understanding with a textual output constraint. The benchmark reports an Instruction Following Rate, a Semantic Correctness Rate, and an Overall Success Rate requiring both properties:

$i$1

This separates audio comprehension from output-format compliance. GPT-4o-audio-preview achieved the highest reported overall instruction-following rate at 0.68, while Qwen2-Audio-7B-Instruct achieved 0.30.

Speech-IFEval appends textual constraints to speech tasks in order to isolate text-instruction following from speech perception (Lu et al., 25 May 2025). It evaluates capitalization, prefixes, quotation wrapping, JSON, bullet formatting, keywords, length, and chain-of-thought behavior. The strongest text-only reference, Llama3.1-8B-Instruct, achieved an IFrate of 93.52, whereas speech-aware systems such as SALMONN achieved 36.89. DeSTA2 achieved 89.23 and came closest to its text-only counterpart. The paper interprets the gap as evidence of catastrophic forgetting after speech-text adaptation.

Code and function calling

IFEvalCode transfers verifiable instruction following to controlled code generation (Yang et al., 30 Jul 2025). It contains approximately 1,620 problems involving Chinese and English queries across Python, Java, C++, C#, TypeScript, JavaScript, PHP, and Shell. Each sample is evaluated with separate correctness and instruction-following checks.

The benchmark distinguishes:

• Corr., functional correctness Pass@1;
• Instr., instruction-following compliance Pass@1.

Constraints include naming, line count, APIs, libraries, comments, documentation, formatting, indentation, class and function structure, algorithm choice, and complexity. Its principal finding is that models are substantially more likely to generate functionally correct code than code that is also instruction-compliant.

IFEval-FC applies IFEval’s methodology to function calling (Skripko, 22 Sep 2025). It contains 750 test cases in which a natural-language formatting requirement is embedded in a JSON Schema parameter description. The user query omits the formatting requirement, so the model must read the schema and transform the function argument accordingly. Constraints include keyword frequency, word and sentence counts, quotation, JSON, Python-list format, comma count, placeholders, and title formatting.

The benchmark is fully algorithmic and tests whether a model produces a schema-compliant argument rather than merely selecting the correct function. No evaluated model exceeded 80% overall accuracy. The results show that semantically correct tool calls can still be operationally invalid because of formatting, serialization, or parser constraints.

6. Reliability, inverse compliance, and evaluation limitations

Later work has broadened IFEval from pointwise constraint satisfaction to reliability under variation, conflict, optimization, and evaluator bias.

IFEval++ generates cousin prompts through rephrasing, distractor addition, and constraint/task reconfiguration (Dong et al., 15 Dec 2025). Starting from each of the 541 original prompts, it creates three cousins of each type, producing groups of ten prompts. Its central metric, reliable@$i$2, requires all $i$3 cousin prompts to pass:

$i$4

Reliable@1 equals ordinary accuracy, while reliable@10 measures family-wise consistency. GPT-5 achieved 95.9% on ordinary IFEval and 78.4% on reliable@10; Qwen3-0.6B declined from 58.0% to 22.2%. These results indicate that high pointwise accuracy can conceal substantial sensitivity to wording, distractors, numerical values, and task framing.

Inverse IFEval tests whether models can follow explicit instructions that conflict with learned conventions (Zhang et al., 4 Sep 2025). Its eight challenge types include Question Correction, Intentional Textual Flaws, Code without Comments, Counter-Conventional Formatting, Deliberately Incorrect Answers, Instructional Induction, Mid-turn Instruction Modification, and Counterfactual Answering. The benchmark contains 1,012 balanced Chinese and English questions across 23 domains.

The benchmark addresses a limitation of ordinary IFEval: a model can be highly competent at conventional constraint following while still correcting intentional errors, adding comments, using bullets despite a prohibition, or refusing to provide a stipulated false answer. Inverse IFEval therefore measures adaptive compliance under conflict with learned norms. o3-high achieved 75.66 in English and 76.52 in Chinese, while some conventional instruction-tuned models ranked substantially lower than expected.

IFEval has also been used to investigate training and inference interventions. Contrastive decoding improved strict IFEval accuracy by up to 8.52 points for Llama-3.1-8B in the reported setup, although excessive contrastive strength reduced performance (Dong et al., 10 Jan 2026). PrefPO optimized difficult IFEval prompts through pairwise LLM preferences and achieved 82.4% worst@20 on IFEval-Hard, compared with 84.5% for TextGrad; the confidence intervals overlapped substantially (Singhal et al., 13 Mar 2026). IFDecorator used reinforcement learning with verifiable rewards and reported 87.43 on IFEval Prompt Strict (Guo et al., 6 Aug 2025).

IFEval has also been used to examine whether instruction tuning transfers to strict verifiable compliance. A LoRA adapter nominally trained for instruction following improved an off-target numeric-match score from 0.133 to 0.632 while reducing IFEval instruction-level accuracy from 0.313 to 0.271 and prompt-level accuracy from 0.250 to 0.143 (Zou, 23 Mar 2026). This result motivates the descriptive term capability drift for mismatches between nominal training objectives and realized cross-task capabilities.

Several studies qualify the objectivity of IFEval-based evaluation. When an LLM judges IFEval outputs instead of executing the official verifier, self-preference bias can persist even though the rubrics are objectively verifiable. GPT-5 exhibited a rubric-level self-preference ratio of 1.47, meaning that among objectively failed rubrics it was approximately 47% more likely to mark its own output as satisfied than outputs from unrelated models (Pombal et al., 8 Apr 2026). Ensembling reduced but did not eliminate the bias.

By contrast, a four-model study of revision found no detectable self-preference when models assessed deterministic, verified-good fixes to their own IFEval drafts. The pooled author-versus-fresh rejection gap was $i$5 percentage points, with a 95% confidence interval of $i$6 percentage points (Guey et al., 18 Jun 2026). This result concerns verifiable revision under genuine authorship and does not establish that self-preference is absent in subjective evaluation.

IFEval’s principal limitations are therefore methodological as well as conceptual:

• Narrow construct validity: it measures explicit, checkable compliance rather than general instruction following.
• Artificiality: unusual character frequencies, exact repetitions, and arbitrary forbidden words may not represent ordinary requests.
• Verifier dependence: strict checkers can produce false negatives, while loose transformations can produce false positives.
• Language ambiguity: word counts, sentence boundaries, capitalization, tokenization, Unicode normalization, and script behavior vary across languages.
• Prompt-distribution bias: the 25 original instruction types and 541 prompts reflect benchmark-design choices.
• Benchmark gaming: public constraints and evaluators can encourage optimization for formal scores without broader capability gains.
• Incomplete task evaluation: a response can satisfy every formal constraint while being inaccurate, nonsensical, unsafe, or irrelevant.
• Limited reproducibility in derivative work: later papers sometimes report an “IFEval average” or “Prompt Strict” score without specifying the release, split, evaluator version, decoding settings, or aggregation rule.
• Potential evaluator bias: an LLM judge may misapply even an objectively defined rubric.

IFEval is consequently best understood as a foundational diagnostic rather than a complete theory or benchmark of instruction following. Its central contribution is the operational separation of explicit constraint compliance from broader response quality. Extensions to multilingual, multimodal, speech, code, function-calling, reliability, inverse-compliance, and evaluator-bias settings demonstrate both the portability of its verifier-oriented methodology and the need to report task content, language, modality, prompt variation, evaluator identity, and uncertainty alongside aggregate compliance scores.