---
title: 'IFEval-FC: Schema-Compliant Function Calls'
url: https://www.emergentmind.com/topics/ifeval-fc
type: topic
---

# IFEval-FC: Schema-Compliant Function Calls

IFEval-FC is a benchmark for **instruction-following in function calling** that tests whether a large language model can obey **format constraints written inside JSON-schema parameter descriptions** when producing tool calls. Introduced in "Instruction-Following Evaluation in Function Calling for Large Language Models" [2509.18420], it addresses a gap left by function-calling benchmarks that primarily evaluate whether a model selects the correct function and supplies semantically correct arguments, but do not directly test adherence to **verifiable formatting instructions embedded in parameter descriptions**. Its central premise is that real agent failures often occur at the final step of producing **valid, schema-compliant tool calls**, even when the underlying reasoning and tool selection are correct [2509.18420].

## 1. Problem formulation and motivation

IFEval-FC was introduced to evaluate a dimension of tool use that conventional function-calling benchmarks largely leave implicit: whether the argument values emitted by the model satisfy **explicit formatting rules** stated in the schema itself. The paper contrasts this objective with benchmarks such as **BFCL**, **\(\tau^2\)-Bench**, and **ACEBench**, which evaluate tool selection, argument correctness, conversational agent performance, and broader agentic behavior, but do not directly test compliance with such embedded formatting instructions [2509.18420].

The motivating examples are operationally simple but practically consequential. A parameter description may require that a value **must be lowercase**, **do not include punctuation**, **wrap the value in double quotes**, **use ISO 8601 date format**, or **format as a Python list**. In the benchmark’s framing, these are not post hoc validators; they are part of the function specification that the model is expected to follow. The authors argue that semantically correct but syntactically noncompliant arguments can still yield invalid API requests, failed tool execution, brittle downstream automation, and degraded agent reliability [2509.18420].

Conceptually, IFEval-FC adapts the IFEval idea of **verifiable instructions** to function calling. Standard IFEval evaluates whether free-form responses satisfy verifiable constraints; IFEval-FC relocates that principle into the tool-use setting by requiring the model to produce a **tool call argument** that obeys a constraint encoded in the schema description [2509.18420]. This design choice makes the benchmark a targeted probe of schema compliance rather than a general benchmark of agent competence.

## 2. Benchmark construction and curation pipeline

The benchmark encodes the instruction directly in the `description` field of one parameter in a function’s JSON schema. This placement is central to the task definition: the model sees the format rule as part of the function interface rather than as an auxiliary instruction. The benchmark uses a mix of **functions sourced from BFCL**, described as real-world schemas, and **synthetic functions generated with GPT-5** [2509.18420].

The synthetic generation pipeline has **three main steps**. First, **80 diverse domains were curated**. Second, the authors generated schemas such that each function had at least one **free-form string parameter** with no enum or format field. Third, they performed **instruction injection**, embedding one verifiable instruction into that parameter’s description. The reliance on a free-form string parameter is deliberate: it allows arbitrary formatting rules in a manner analogous to the text-generation setting of IFEval, whereas fixed-format parameters such as IDs do not readily support this type of manipulation [2509.18420].

For each function, the authors generated **five diverse natural-language user queries** using GPT-5. These queries are conversational, specify values for all required parameters, and are generated from the **original function, without embedded formatting**, so the user does not pre-bake the required format into the prompt. The transformation into the required format must therefore be performed by the model during tool calling rather than copied from the user query [2509.18420].

The final benchmark contains **750 test cases**. Its organization follows directly from the construction procedure: each function is paired with five user queries, and each function/query pair requires satisfaction of a **single verifiable instruction** in the emitted call [2509.18420].

Curation included filtering and validation. If an LLM failed to call the target function on all five queries, the task was considered ill-posed or too hard and removed. If all five queries were answered correctly with ease, the task was also removed. An ensemble of LLMs was used for robust task selection. The stated goal was to retain tasks that were neither trivial nor impossible [2509.18420].

A notable practical issue was function-call avoidance. The paper reports that Anthropic models, especially **Claude Opus 4.1**, sometimes refused to call a function and instead asked for clarification when formatting requirements were ambiguous from the user’s perspective. To eliminate that confound, the evaluation added the following system message [2509.18420]:

```python
SYSTEM_MESSAGE = '''
YOU MUST CALL A FUNCTION NO MATTER WHAT.
NEVER ASK A USER TO SPECIFY OR CLARIFY ANYTHING.
ALWAYS CALL A FUNCTION.
'''.strip()
```

## 3. Instruction taxonomy and schema-level constraints

IFEval-FC evaluates **19 instruction types**, grouped into **7 major categories**. Many are adapted from the original IFEval benchmark, while **Cyrillic Greek** and **Python List Format** are introduced here for the first time [2509.18420].

| Category | Representative instruction types |
|---|---|
| Keywords | Keywords Presence; Keyword Frequency; Letter Frequency |
| Language | Cyrillic Greek |
| Length Constraints | Word Count; Sentence Count |
| Detectable Content | Postscript; Placeholder Count |
| Detectable Format | Spaces In Between; Title Format; Highlighted Sections Count; Json Format; Python List Format |
| Case | All Uppercase; All Lowercase; N All Capital Words |
| Start/End | End Phrase; Quotation |
| Punctuation | N Commas |

The design criterion for these instruction types is that they are **algorithmically checkable**. That requirement shapes the benchmark’s scope. Constraints such as **Quotation**, **Json Format**, **Python List Format**, or **All Lowercase** admit deterministic validation logic; more open-ended stylistic constraints do not. The benchmark therefore operationalizes instruction-following as compliance with formalizable output contracts rather than with underspecified preferences [2509.18420].

This schema-centric formulation changes the locus of instruction-following. In ordinary prompt engineering, formatting requests typically appear in the natural-language prompt. In IFEval-FC, the critical instruction resides in the parameter description, which means the model must treat the schema as an active control surface rather than as passive documentation. A plausible implication is that the benchmark probes whether function-calling models genuinely parse and operationalize JSON-schema semantics, rather than merely extracting argument values from the user query.

## 4. Scoring methodology and reproducibility

IFEval-FC uses a deliberately simple binary scoring rule. For a model response \(r\) and a verifiable instruction \(i\), the paper defines [2509.18420]:

$$
\text{score}(r, i) =
\begin{cases}
1, & \text{if instruction is followed} \\
0, & \text{otherwise}
\end{cases}
$$

Each response is therefore evaluated solely on whether the specified instruction was followed. Because the instructions are designed to be **verifiable by code**, the evaluation is described as **fully algorithmic**, **reproducible**, and **scalable**, and it does not require a human judge or an LLM-as-a-judge [2509.18420].

This choice is methodologically significant. In many instruction-following benchmarks, especially those involving open-ended outputs, evaluation depends on subjective interpretation or model-based adjudication. IFEval-FC explicitly positions deterministic verification as a strength relative to such approaches. The benchmark is consequently narrower than a holistic tool-use evaluation, but it gains objectivity by restricting itself to constraints that admit unambiguous programmatic checks [2509.18420].

The scoring abstraction also clarifies what is and is not being measured. IFEval-FC does not primarily ask whether the function call is semantically useful in a broad agentic sense. It asks whether one designated aspect of the output conforms exactly to a verifiable contract embedded in the schema. This makes the benchmark particularly suited to diagnosing **last-mile** failures in structured tool invocation.

## 5. Empirical results and characteristic failure modes

The paper’s headline empirical result is that **no evaluated model surpassed 80% accuracy overall** on the benchmark [2509.18420]. The authors present this as evidence that precise instruction-following in function calling remains an open problem even for strong modern systems.

The reported failure pattern is heterogeneous rather than uniform. Models may succeed on some schema constraints and fail sharply on others. The benchmark attributes typical weaknesses to failures in preserving punctuation constraints, producing valid JSON, obeying quotation requirements, satisfying exact word or sentence counts, generating the correct casing or list syntax, and following language or script constraints [2509.18420].

Among **Anthropic models**, **Opus 4.1** achieved **100%** on **Highlighted Sections Count**, **100%** on **Title Format**, **68%** on **Json Format**, **90%** on **Keyword Frequency**, **80%** on **Keywords Presence**, **78%** on **N Commas**, and **94%** on **Placeholder Count**. **Opus 4.1 Thinking** achieved **100%** on **Highlighted Sections Count**, **68%** on **Json Format**, **90%** on **Keywords Presence**, **92%** on **Placeholder Count**, **96%** on **Word Count**, and **86%** on **Sentence Count**. Yet the same family still showed conspicuous weaknesses: **Python List Format** was only **90%** for both Opus 4.1 and Opus 4.1 Thinking, **Letter Frequency** was **28%** and **54%**, **Spaces In Between** dropped to **8%** on Opus 4.1, and **Json Format** remained at **68%** rather than approaching perfect reliability [2509.18420].

Among **OpenAI models**, **GPT-5 minimal** reached **98%** on **Keyword Frequency**, **98%** on **Python List Format**, **88%** on **Quotation**, **94%** on **Title Format**, **82%** on **Sentence Count**, and **58%** on **Json Format**. **GPT-4.1** reached **90%** on **Word Count**, **94%** on **Keyword Frequency**, and **94%** on **Python List Format**, but only **14%** on **Json Format**. **GPT-4o** was strong on some formatting tasks but low on others, including **18%** on **Quotation** and **12%** on **Placeholder Count** [2509.18420].

The paper also reports **GigaChat 2**, **GigaChat 2 Pro**, and **GigaChat 2 Max**. For **GigaChat 2 Max**, the appendix reports **50%** on **Cyrillic Greek**, **72%** on **Highlighted Sections Count**, **64%** on **Keyword Frequency**, **58%** on **Placeholder Count**, **24%** on **Python List Format**, **36%** on **Quotation**, and **18%** on **Word Count**. **Json Format** was **0%** for all three GigaChat variants [2509.18420].

These results support a specific qualitative conclusion emphasized by the paper: even top models frequently fail at formatting rules that would ordinarily be treated as basic. The issue is not that models are uniformly incapable of tool use, but that they are **unreliable** in the exact output constraints that real function-calling systems require [2509.18420]. A common misconception is that good semantic tool selection is sufficient for dependable agent behavior; IFEval-FC shows that correct tool choice and semantically plausible arguments do not guarantee **schema-compliant execution**.

## 6. Position within the IFEval family, terminological clarification, and extensions

IFEval-FC belongs to a broader line of benchmarks that transplant the IFEval emphasis on verifiable instruction-following into domain-specific settings. In speech-aware language models, **Speech-IFEval** isolates instruction-following from speech perception and quantifies catastrophic forgetting relative to a text-only reference system [2505.19037]. In code generation, **IFEvalCode** separates **correctness** from **instruction-following** in controlled code generation and reports them as distinct metrics, **Corr.** and **Instr.** [2507.22462]. This suggests a broader methodological movement toward decomposing task success into semantic competence and explicit instruction compliance.

At the same time, terminological precision matters. **Inverse IFEval** does **not** define a benchmark officially named **IFEval-FC**. In that paper, the relevant internal label is **CCF**, standing for **Counter-Conventional Formatting**; the paper explicitly states that **FC is not used as a standalone named variant** [2509.04292]. Conflating IFEval-FC with the formatting-related subset of Inverse IFEval is therefore inaccurate.

The practical significance of IFEval-FC lies in its role as a **diagnostic tool** for tool-use robustness and as a **complement** to existing function-calling benchmarks. Its findings imply that agent evaluation should cover not only whether a model picked the right tool and supplied the right content, but also whether it obeyed the exact output contract specified by the schema [2509.18420].

The authors outline several future directions: **expanding the number of functions**, **selecting harder samples**, **adding multi-function selection scenarios**, and **extending the benchmark to multilingual settings**, inspired by **M-IFEval** [2509.18420]. These proposals preserve the benchmark’s core commitment to verifiable instructions while widening its coverage of realistic agentic settings.

IFEval-FC therefore occupies a specific niche in the evaluation landscape. It does not attempt to be a complete theory of function calling or agent behavior. Rather, it provides an algorithmically checkable measure of whether a model can honor schema-embedded formatting requirements at the point where downstream systems are most brittle: the production of the actual tool call [2509.18420].

Source: https://www.emergentmind.com/topics/ifeval-fc