FollowEval: Evaluating Instruction-Following LLMs

• FollowEval is a comprehensive evaluation framework that incorporates benchmarks, metrics, and methodologies for assessing instruction-following in LLMs and autonomous systems.
• It employs multi-dimensional tests—spanning string manipulation, commonsense, logical, spatial reasoning, and response constraints—to rigorously stress-test composite instructions.
• Empirical analyses reveal significant performance gaps, highlighting challenges such as format precision and constraint satisfaction compared to human baselines.

FollowEval refers to a family of benchmarks, metrics, and methodologies for the rigorous evaluation of instruction-following, adherence to constraints, or behavior prediction in agents and LLMs. The term “FollowEval” appears in literature across instruction-following LLM assessment, open-domain dialog quality, and vehicle trajectory prediction settings. The paradigmatic use is the multi-dimensional benchmark "FollowEval" for LLMs, but the term is also used in dialog evaluation frameworks and in autonomous driving contexts for vehicle-following analysis.

1. Motivation and Context

Instruction-following is central to LLM deployment, especially as these models become increasingly embedded in real-world agent systems. Effective evaluation of a model’s ability to follow instructions (including multiple simultaneous constraints) is critical for reliability in human-facing or tool-using applications. Early benchmarks for LLMs focused on semantic correctness or API selection but generally neglected verifiable adherence to explicit format requirements or composite user constraints (Jing et al., 2023, Lior et al., 9 Mar 2025, Zhou et al., 2023, Skripko, 22 Sep 2025). In open-domain dialogue and in autonomous driving, the “follow” aspect extends to measuring qualitatively successful interactions or agent behavior fidelity (Bruyn et al., 2022, Mohamed et al., 2023).

2. Multi-Dimensional FollowEval Benchmarks for LLMs

The original "FollowEval" benchmark (Jing et al., 2023) is a human-authored, multi-lingual (English/Chinese) test suite for LLMs, designed to assess performance on tasks that require the simultaneous application of several instruction-following “dimensions” within a single response. The five critical dimensions are:

• String Manipulation: Character-level operations (identification, insertion, deletion).
• Commonsense Reasoning: Leveraging background knowledge or idiomatic understanding.
• Logical Reasoning: Arithmetic, inequalities, character counting.
• Spatial Reasoning: Textual analogs of spatial tasks (e.g., flipping or rotating characters).
• Response Constraints: Enforcement of explicit format, content, or length requirements.

Each prompt combines at least two of these, increasing evaluation complexity and moving beyond single-dimension or synthetic tasks. Human curation ensures high ambiguity and edge-case coverage. Prompts are paired with regex-based checkers for automatic validation, ensuring reproducibility and minimizing subjective annotation.

Empirical results indicate a substantial capability gap. On the 200-prompt benchmark, humans score 1.00, while GPT-4 reaches only ~0.78, and open-source models trail further (often below 0.40). Instructions involving strict output formatting or spatial/logical challenges cause the most frequent failures, demonstrating that even top models are brittle on complex, compositional constraints.

3. Instruction Following Benchmarks in Real-World Contexts

Extending the FollowEval paradigm, WildIFEval (Lior et al., 9 Mar 2025) introduces a large-scale, naturalistic dataset (∼12,000 prompts, 29,874 constraints) harvested from genuine user interactions. Each instruction is decomposed into atomic constraints spanning eight principal categories:

• Include/Avoid
• Editing
• Ensure Quality
• Length
• Format and Structure
• Focus/Emphasis
• Persona and Role
• Style and Tone

Performance evaluation leverages LLM-based judges that assess, for each constituent constraint, whether it is satisfied in the output. Models’ mean fulfillment decreases rapidly with constraint count, dropping from ≈0.75 (single-constraint) to ≈0.35 (eight constraints). Satisfying length, quality, and quantitative constraints remains especially difficult across all model scales (even 405B-parameter models do not exceed ≈0.62 mean fulfillment).

Error analysis reveals that while content-focused constraints are more tractable, quantitative requirements (word count, length) and ambiguous quality specifications (e.g., coherence) are persistent weaknesses. This naturally leads to research directions combining LLM- and tool-based checking, constraint decomposition, and iterative correction cycles.

4. Verifiable Instruction-Following Evaluation (IFEval and Variants)

IFEval (Zhou et al., 2023) and its derivatives (IFEval-FC (Skripko, 22 Sep 2025)) target the subdomain of "verifiable" or "format-constrained" instruction following by constructing benchmarks anchored in deterministic, algorithmically checkable rules. Typical instruction types include imposing minimum/maximum counts (keywords, bullets, words), forbidden tokens, required output formats (JSON, regular expression match), and specific case or punctuation restrictions.

IFEval comprises atomic instruction types and multi-instruction composites, each with explicit Python-check scripts. Metrics include per-instruction and per-prompt (all instructions correct) accuracy, reported both strictly and under lenient (preprocessing-tolerant) settings. Models are evaluated zero-shot via API calls, with strict objectivity and full reproducibility.

IFEval-FC specializes this paradigm for the function-calling scenario ubiquitous in API-driven agents. It uses formal JSON schemas to encode parameter-level formatting constraints and exposes 750 function calls, each with an associated verifiable requirement (e.g., dates in ISO format, quoted strings, no punctuation). Even advanced proprietary LLMs do not reach 80% overall adherence, highlighting structural deficiencies in LLM function-call fidelity critical for agentic tool use.

5. Dialog and Human Alignment FollowEval Approaches

A variant of FollowEval targets open-domain dialog evaluation by operationalizing the intuition that “good” conversations are less likely to elicit prototypically negative follow-ups (Bruyn et al., 2022). A LLM computes the average log-probability of a set of negative follow-up utterances given the previous conversation. Higher (less negative-surprisal) scores predict higher human-rated quality. This reference-free, unsupervised approach outperforms numerous existing metrics on standard datasets, albeit with the limitation that it is tuned to detecting undesirable turns rather than rewarding positive creativity or depth.

The HREF methodology (Lyu et al., 2024) further addresses the alignment of model outputs and human preferences by constructing a composite evaluation system that blends LLM-judged pairwise comparison, human-written references, and embedding-based similarity scoring. The system selects, per task category, the evaluation method best matched to human ratings, and recommends collection/curation of high-quality human exemplars as references to further reduce annotation bias and inter-model drift.

6. Evaluation Metrics and Analysis

FollowEval benchmarks typically define accuracy as the strict fraction of test cases for which all constraints, or all required dimensions, are met:

$$\mathrm{Accuracy} = \frac{\text{Number of correct responses}}{\text{Total number of examples}}$$

For function-calling and multi-field scenarios, per-argument precision, recall, and F1-score are additionally measured across all constrained parameters:

$$\mathrm{Precision} = \frac{\sum \mathrm{TP}}{\sum(\mathrm{TP}+\mathrm{FP})}, \quad \mathrm{Recall} = \frac{\sum \mathrm{TP}}{\sum(\mathrm{TP}+\mathrm{FN})}$$

Dialog-focused FollowEval computes the mean log-likelihood of negative follow-ups, and benchmarks correlation (Spearman ρ, Pearson r) versus human quality judgments. WildIFEval and HREF further dissect constraint-type recall, inter-task agreement, and embedding-based semantic match, which reveals the multi-faceted nature of following instructions and the varying difficulty posed by different constraint types.

Empirical findings consistently demonstrate a substantial performance deficit relative to human baselines and a pronounced degradation as the number and complexity of constraints increase.

7. Applications, Limitations, and Future Directions

FollowEval-style frameworks are deployed for:

• Comparative benchmarking of LLM instruction-following capabilities, especially for multi-constraint, cross-lingual, or tool-augmented tasks.
• Stress-testing API-agent LLMs where format adherence is critical for successful function execution (Skripko, 22 Sep 2025).
• Analyzing dialog quality and agent alignment via “negative follow-up” probabilities or human-guided composite judgment pipelines (Bruyn et al., 2022, Lyu et al., 2024).
• Identifying failure modes, including strict output formatting, spatial/logical task misalignment, and inability to handle quantitative instruction constraints (Jing et al., 2023, Lior et al., 9 Mar 2025).

Key limitations include restricted instruction type coverage (verifiable constraints are easier to check than semantic or creative ones), sensitivity to prompt phrasing, and limited robustness to adversarial or paraphrased instructions. LLM-based auto-evaluators may introduce drift or bias relative to human raters unless actively corrected using high-quality reference responses or hybrid pipelines (LLM plus rule-based).

Suggested future developments encompass hybrid evaluation schemes, curriculum-based model fine-tuning on multi-constraint tasks, enhanced automatic task decomposition, and multi-modal extension (including code or tabular formats). Expanding linguistic and domain diversity, as well as algorithmic and model transparency, remain open research priorities.

---

References:

• "FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of LLMs" (Jing et al., 2023)
• "WildIFEval: Instruction Following in the Wild" (Lior et al., 9 Mar 2025)
• "Instruction-Following Eval for LLMs" (Zhou et al., 2023)
• "Instruction-Following Evaluation in Function Calling for LLMs" (Skripko, 22 Sep 2025)
• "Automatic Dialog Evaluation using Follow-Ups Likelihood" (Bruyn et al., 2022)
• "HREF: Human Response-Guided Evaluation" (Lyu et al., 2024)