Multi-IF: Multilingual Multi-turn Benchmark

• Multi-IF Benchmark is a comprehensive evaluation framework designed to measure LLMs' ability to follow multi-turn instructions across diverse languages.
• It leverages a unique multi-turn conversation structure with automated and human-verified translations to test instruction fidelity and error correction.
• Empirical results reveal significant performance degradation over turns, especially in non-Latin scripts, highlighting challenges in realistic LLM deployments.

Multi-IF Benchmark

The Multi-IF benchmark is a prominent evaluation framework designed to systematically measure LLMs’ (LLMs) capacity to follow complex, multi-turn instructions across multilingual scenarios (He et al., 2024). Developed to address limitations in prior single-turn, monolingual benchmarks, Multi-IF exposes LLMs to concatenated instruction sequences and multi-lingual interactions, thereby simulating realistic conversational settings observed in actual deployments. The benchmark evaluates models on rigorous metrics capturing instruction-following fidelity, error correction, and degradation across turns and languages.

1. Motivation and Scope

The motivation behind Multi-IF lies in the inadequacies of single-turn, English-centric benchmarks such as IFEval, which fail to capture two central aspects of real-world usage: multi-turn dialogue and instruction expression in various languages, including those with non-Latin scripts. Multi-IF thus explicitly targets (a) the retention and execution of instructions over multiple dialogue turns, and (b) robust multi-lingual instruction adherence, offering a more challenging landscape for LLM benchmarking (He et al., 2024).

2. Benchmark Construction and Dataset Composition

Multi-IF’s dataset consists of 4,501 conversations, each spanning three turns. The construction process employs a hybrid annotation framework: initial expansion of IFEval’s single-turn English prompts using LLMs (Llama 3.1 405B) to generate additional instructions and multi-turn prompts, followed by human review for contradiction and naturalness in each language. English prompts are translated into French, Italian, Portuguese, Spanish (Latin script), and Russian, Hindi, Chinese (non-Latin script) using LLMs augmented with human audits—around 15% of translations are substantially revised for fidelity. The final dataset removes all prompts deemed politically, religiously, or culturally sensitive.

Each conversation follows a strictly three-turn structure, with every turn introducing a new, independently verifiable instruction. Distribution is nearly balanced across languages, although exact counts vary due to some instructions defying direct translation in certain languages.

3. Evaluation Protocol and Metrics

Multi-IF adopts a multi-turn inference protocol: for turn $t$, the input consists of all prior prompts and responses concatenated, plus the current prompt. Fourteen models are evaluated under identical prompt-concatenation conditions (temperature=0.6, top-$p$=0.9, max tokens up to 25,000 depending on model). Metrics are:

• Instruction-level strict accuracy: proportion of instructions correctly followed at turn $t$.
• Conversation-level strict accuracy: fraction of conversations with all instructions up to turn $t$ correct.
• Instruction-level loose accuracy: variant allowing removal of boilerplate for partial credit.
• Conversation-level loose accuracy: as above but for conversations.
• FinalMetricₜ: mean of the four accuracy metrics per model, turn, and language.

The protocol enables evaluation of degradation and recovery as instruction sequences lengthen, capturing both strict and loose compliance.

4. Empirical Findings: Degradation Patterns and Error Taxonomies

Results reveal a consistent, monotonic drop in accuracy as the number of turns increases. For instance, o1-preview’s FinalMetric drops from 0.877 (turn 1) to 0.707 (turn 3); GPT-4’s accuracy falls more sharply, from 0.815 to 0.609. All models show substantial increased error rates on non-Latin languages—Russian is particularly challenging (third-turn average accuracy: 0.531 vs. ≈0.75 for Latin scripts).

Multi-IF measures the Instruction Forgetting Ratio (IFR) and Error Correction Ratio (ECR):

• $$\mathrm{IFR} = \frac{\#\text{instr followed in turn}\ t-1\ \text{but violated in turn}\ t}{\#\text{instr followed in turn}\ t-1} \times 100\%$$
• $$\mathrm{ECR} = \frac{\#\text{instr not followed in turn}\ t-1\ \text{but followed in turn}\ t}{\#\text{instr not followed in turn}\ t-1} \times 100\%$$

High-performing models (o1-preview, o1-mini, Llama 3.1 405B) exhibit IFRs of 15–20% across turns and ECRs around 25%; smaller models exceed 30% IFR and lower ECR.

Error rates are highest for length_constraint and combination instruction types, even in English. In non-Latin scripts, startend and keywords instructions present additional failure modes. This suggests notable cross-linguistic variance in instruction adherence and parsing.

5. Comparative Model Performance

Fourteen models representing contemporary architecture families—OpenAI o1 series, Meta Llama 3.1 suite, Google Gemini, Anthropic Claude 3, Qwen 2.5, Mistral Large 2—are evaluated under uniform protocols. Notably, all models degrade linearly or worse across turns, with performance spread magnified in non-Latin scripts. Stronger chain-of-thought inference (o1-preview/mini) yields improved error correction but does not fully arrest degradation.

6. Analysis, Limitations, and Recommendations

Key findings from Multi-IF include:

• Universality of multi-turn degradation, with accuracy gaps exceeding 0.2 over three turns in leading models.
• Pronounced shortfall in non-Latin languages, highlighting pretraining limitations—Russian is particularly problematic.
• Instruction forgetting constitutes the predominant multi-turn failure mode; only a minority of models display significant recovery.
• Strict instruction-matching in multi-lingual multi-turn settings exceeds the difficulty of any prior instruction-following benchmark.

Limitations include three-turn cap per conversation, exclusion of open-ended or subjective instructions, and limited language coverage. Extending turn count (5–10+) and incorporating additional non-European languages would expose further model weaknesses. There is also opportunity to supplement strict rule-based metrics with learned or human-adjudicated criteria.

Recommendations for model development derived from benchmark results include explicit memory or instruction-tracking modules to counteract forgetting, augmented multilingual pretraining, and hybrid inference schemes (e.g., chain-of-thought self-correction). Prompt-engineering strategies that reinforce historical instructions are strongly indicated.

7. Impact and Resource Availability

Multi-IF establishes a reproducible standard and open-source testbed (https://github.com/facebookresearch/Multi-IF) for evaluating conversational, multilingual instruction-following in LLMs. Its rigorous combination of multi-turn, multi-lingual scenarios, automated and manual quality controls, and nuanced error analytics provides clear direction for both benchmarking research and model improvement (He et al., 2024). The findings elucidate core problems for LLM alignment in realistic interactive use-cases, where both memory and robust cross-lingual parsing are paramount.