IFEval-FC: Schema-Compliant Function Calls
- The paper introduces IFEval-FC as a benchmark that assesses whether LLMs produce valid, schema-compliant tool calls with embedded JSON-formatting instructions.
- It emphasizes methodology by embedding verifiable formatting constraints such as lowercase, punctuation rules, ISO dates, and Python list syntax into parameter descriptions.
- Empirical results show that even advanced LLMs struggle with deterministic format compliance, uncovering critical gaps in reliable tool invocation.
IFEval-FC is a benchmark for instruction-following in function calling that tests whether a LLM can obey format constraints written inside JSON-schema parameter descriptions when producing tool calls. Introduced in "Instruction-Following Evaluation in Function Calling for LLMs" (Skripko, 22 Sep 2025), it addresses a gap left by function-calling benchmarks that primarily evaluate whether a model selects the correct function and supplies semantically correct arguments, but do not directly test adherence to verifiable formatting instructions embedded in parameter descriptions. Its central premise is that real agent failures often occur at the final step of producing valid, schema-compliant tool calls, even when the underlying reasoning and tool selection are correct (Skripko, 22 Sep 2025).
1. Problem formulation and motivation
IFEval-FC was introduced to evaluate a dimension of tool use that conventional function-calling benchmarks largely leave implicit: whether the argument values emitted by the model satisfy explicit formatting rules stated in the schema itself. The paper contrasts this objective with benchmarks such as BFCL, -Bench, and ACEBench, which evaluate tool selection, argument correctness, conversational agent performance, and broader agentic behavior, but do not directly test compliance with such embedded formatting instructions (Skripko, 22 Sep 2025).
The motivating examples are operationally simple but practically consequential. A parameter description may require that a value must be lowercase, do not include punctuation, wrap the value in double quotes, use ISO 8601 date format, or format as a Python list. In the benchmark’s framing, these are not post hoc validators; they are part of the function specification that the model is expected to follow. The authors argue that semantically correct but syntactically noncompliant arguments can still yield invalid API requests, failed tool execution, brittle downstream automation, and degraded agent reliability (Skripko, 22 Sep 2025).
Conceptually, IFEval-FC adapts the IFEval idea of verifiable instructions to function calling. Standard IFEval evaluates whether free-form responses satisfy verifiable constraints; IFEval-FC relocates that principle into the tool-use setting by requiring the model to produce a tool call argument that obeys a constraint encoded in the schema description (Skripko, 22 Sep 2025). This design choice makes the benchmark a targeted probe of schema compliance rather than a general benchmark of agent competence.
2. Benchmark construction and curation pipeline
The benchmark encodes the instruction directly in the description field of one parameter in a function’s JSON schema. This placement is central to the task definition: the model sees the format rule as part of the function interface rather than as an auxiliary instruction. The benchmark uses a mix of functions sourced from BFCL, described as real-world schemas, and synthetic functions generated with GPT-5 (Skripko, 22 Sep 2025).
The synthetic generation pipeline has three main steps. First, 80 diverse domains were curated. Second, the authors generated schemas such that each function had at least one free-form string parameter with no enum or format field. Third, they performed instruction injection, embedding one verifiable instruction into that parameter’s description. The reliance on a free-form string parameter is deliberate: it allows arbitrary formatting rules in a manner analogous to the text-generation setting of IFEval, whereas fixed-format parameters such as IDs do not readily support this type of manipulation (Skripko, 22 Sep 2025).
For each function, the authors generated five diverse natural-language user queries using GPT-5. These queries are conversational, specify values for all required parameters, and are generated from the original function, without embedded formatting, so the user does not pre-bake the required format into the prompt. The transformation into the required format must therefore be performed by the model during tool calling rather than copied from the user query (Skripko, 22 Sep 2025).
The final benchmark contains 750 test cases. Its organization follows directly from the construction procedure: each function is paired with five user queries, and each function/query pair requires satisfaction of a single verifiable instruction in the emitted call (Skripko, 22 Sep 2025).
Curation included filtering and validation. If an LLM failed to call the target function on all five queries, the task was considered ill-posed or too hard and removed. If all five queries were answered correctly with ease, the task was also removed. An ensemble of LLMs was used for robust task selection. The stated goal was to retain tasks that were neither trivial nor impossible (Skripko, 22 Sep 2025).
A notable practical issue was function-call avoidance. The paper reports that Anthropic models, especially Claude Opus 4.1, sometimes refused to call a function and instead asked for clarification when formatting requirements were ambiguous from the user’s perspective. To eliminate that confound, the evaluation added the following system message (Skripko, 22 Sep 2025):
1 2 3 4 5 |
SYSTEM_MESSAGE = '''
YOU MUST CALL A FUNCTION NO MATTER WHAT.
NEVER ASK A USER TO SPECIFY OR CLARIFY ANYTHING.
ALWAYS CALL A FUNCTION.
'''.strip() |
3. Instruction taxonomy and schema-level constraints
IFEval-FC evaluates 19 instruction types, grouped into 7 major categories. Many are adapted from the original IFEval benchmark, while Cyrillic Greek and Python List Format are introduced here for the first time (Skripko, 22 Sep 2025).
| Category | Representative instruction types |
|---|---|
| Keywords | Keywords Presence; Keyword Frequency; Letter Frequency |
| Language | Cyrillic Greek |
| Length Constraints | Word Count; Sentence Count |
| Detectable Content | Postscript; Placeholder Count |
| Detectable Format | Spaces In Between; Title Format; Highlighted Sections Count; Json Format; Python List Format |
| Case | All Uppercase; All Lowercase; N All Capital Words |
| Start/End | End Phrase; Quotation |
| Punctuation | N Commas |
The design criterion for these instruction types is that they are algorithmically checkable. That requirement shapes the benchmark’s scope. Constraints such as Quotation, Json Format, Python List Format, or All Lowercase admit deterministic validation logic; more open-ended stylistic constraints do not. The benchmark therefore operationalizes instruction-following as compliance with formalizable output contracts rather than with underspecified preferences (Skripko, 22 Sep 2025).
This schema-centric formulation changes the locus of instruction-following. In ordinary prompt engineering, formatting requests typically appear in the natural-language prompt. In IFEval-FC, the critical instruction resides in the parameter description, which means the model must treat the schema as an active control surface rather than as passive documentation. A plausible implication is that the benchmark probes whether function-calling models genuinely parse and operationalize JSON-schema semantics, rather than merely extracting argument values from the user query.
4. Scoring methodology and reproducibility
IFEval-FC uses a deliberately simple binary scoring rule. For a model response and a verifiable instruction , the paper defines (Skripko, 22 Sep 2025):
Each response is therefore evaluated solely on whether the specified instruction was followed. Because the instructions are designed to be verifiable by code, the evaluation is described as fully algorithmic, reproducible, and scalable, and it does not require a human judge or an LLM-as-a-judge (Skripko, 22 Sep 2025).
This choice is methodologically significant. In many instruction-following benchmarks, especially those involving open-ended outputs, evaluation depends on subjective interpretation or model-based adjudication. IFEval-FC explicitly positions deterministic verification as a strength relative to such approaches. The benchmark is consequently narrower than a holistic tool-use evaluation, but it gains objectivity by restricting itself to constraints that admit unambiguous programmatic checks (Skripko, 22 Sep 2025).
The scoring abstraction also clarifies what is and is not being measured. IFEval-FC does not primarily ask whether the function call is semantically useful in a broad agentic sense. It asks whether one designated aspect of the output conforms exactly to a verifiable contract embedded in the schema. This makes the benchmark particularly suited to diagnosing last-mile failures in structured tool invocation.
5. Empirical results and characteristic failure modes
The paper’s headline empirical result is that no evaluated model surpassed 80% accuracy overall on the benchmark (Skripko, 22 Sep 2025). The authors present this as evidence that precise instruction-following in function calling remains an open problem even for strong modern systems.
The reported failure pattern is heterogeneous rather than uniform. Models may succeed on some schema constraints and fail sharply on others. The benchmark attributes typical weaknesses to failures in preserving punctuation constraints, producing valid JSON, obeying quotation requirements, satisfying exact word or sentence counts, generating the correct casing or list syntax, and following language or script constraints (Skripko, 22 Sep 2025).
Among Anthropic models, Opus 4.1 achieved 100% on Highlighted Sections Count, 100% on Title Format, 68% on Json Format, 90% on Keyword Frequency, 80% on Keywords Presence, 78% on N Commas, and 94% on Placeholder Count. Opus 4.1 Thinking achieved 100% on Highlighted Sections Count, 68% on Json Format, 90% on Keywords Presence, 92% on Placeholder Count, 96% on Word Count, and 86% on Sentence Count. Yet the same family still showed conspicuous weaknesses: Python List Format was only 90% for both Opus 4.1 and Opus 4.1 Thinking, Letter Frequency was 28% and 54%, Spaces In Between dropped to 8% on Opus 4.1, and Json Format remained at 68% rather than approaching perfect reliability (Skripko, 22 Sep 2025).
Among OpenAI models, GPT-5 minimal reached 98% on Keyword Frequency, 98% on Python List Format, 88% on Quotation, 94% on Title Format, 82% on Sentence Count, and 58% on Json Format. GPT-4.1 reached 90% on Word Count, 94% on Keyword Frequency, and 94% on Python List Format, but only 14% on Json Format. GPT-4o was strong on some formatting tasks but low on others, including 18% on Quotation and 12% on Placeholder Count (Skripko, 22 Sep 2025).
The paper also reports GigaChat 2, GigaChat 2 Pro, and GigaChat 2 Max. For GigaChat 2 Max, the appendix reports 50% on Cyrillic Greek, 72% on Highlighted Sections Count, 64% on Keyword Frequency, 58% on Placeholder Count, 24% on Python List Format, 36% on Quotation, and 18% on Word Count. Json Format was 0% for all three GigaChat variants (Skripko, 22 Sep 2025).
These results support a specific qualitative conclusion emphasized by the paper: even top models frequently fail at formatting rules that would ordinarily be treated as basic. The issue is not that models are uniformly incapable of tool use, but that they are unreliable in the exact output constraints that real function-calling systems require (Skripko, 22 Sep 2025). A common misconception is that good semantic tool selection is sufficient for dependable agent behavior; IFEval-FC shows that correct tool choice and semantically plausible arguments do not guarantee schema-compliant execution.
6. Position within the IFEval family, terminological clarification, and extensions
IFEval-FC belongs to a broader line of benchmarks that transplant the IFEval emphasis on verifiable instruction-following into domain-specific settings. In speech-aware LLMs, Speech-IFEval isolates instruction-following from speech perception and quantifies catastrophic forgetting relative to a text-only reference system (Lu et al., 25 May 2025). In code generation, IFEvalCode separates correctness from instruction-following in controlled code generation and reports them as distinct metrics, Corr. and Instr. (Yang et al., 30 Jul 2025). This suggests a broader methodological movement toward decomposing task success into semantic competence and explicit instruction compliance.
At the same time, terminological precision matters. Inverse IFEval does not define a benchmark officially named IFEval-FC. In that paper, the relevant internal label is CCF, standing for Counter-Conventional Formatting; the paper explicitly states that FC is not used as a standalone named variant (Zhang et al., 4 Sep 2025). Conflating IFEval-FC with the formatting-related subset of Inverse IFEval is therefore inaccurate.
The practical significance of IFEval-FC lies in its role as a diagnostic tool for tool-use robustness and as a complement to existing function-calling benchmarks. Its findings imply that agent evaluation should cover not only whether a model picked the right tool and supplied the right content, but also whether it obeyed the exact output contract specified by the schema (Skripko, 22 Sep 2025).
The authors outline several future directions: expanding the number of functions, selecting harder samples, adding multi-function selection scenarios, and extending the benchmark to multilingual settings, inspired by M-IFEval (Skripko, 22 Sep 2025). These proposals preserve the benchmark’s core commitment to verifiable instructions while widening its coverage of realistic agentic settings.
IFEval-FC therefore occupies a specific niche in the evaluation landscape. It does not attempt to be a complete theory of function calling or agent behavior. Rather, it provides an algorithmically checkable measure of whether a model can honor schema-embedded formatting requirements at the point where downstream systems are most brittle: the production of the actual tool call (Skripko, 22 Sep 2025).