FC-RewardBench: Tool-Calling Evaluation

• FC-RewardBench is a benchmark for assessing structured tool-calling correctness via pairwise preference judgments in a single-turn, schema-aware setup.
• It is built from BFCL v3 data with 1,500 examples comparing correct and incorrect tool call sequences to capture subtle semantic and parameter errors.
• The benchmark shows strong downstream correlation and validates specialized reward models like ToolRM over conventional natural-language reward models.

FC-RewardBench is a pairwise preference benchmark for tool calling introduced in "ToolRM: Outcome Reward Models for Tool-Calling LLMs" to evaluate whether reward models can judge tool-calling or function-calling outputs correctly rather than merely rank natural-language responses. It is built from BFCL v3 single-turn function-calling data, contains 1,500 unique examples, and presents each example as a comparison between a correct tool call sequence and an incorrect tool call sequence under a shared tool catalog and user query. The benchmark was proposed to fill a gap left by prior reward-model benchmarks, which emphasize chat quality, reasoning, safety, factuality, style, or math/code, but not whether a model produces a valid and useful sequence of function calls (Agarwal et al., 15 Sep 2025).

1. Origin and problem setting

FC-RewardBench was introduced in response to a benchmarking gap in reward modeling for tool use. The motivating claim is that existing reward models are trained primarily on natural language preference data and therefore often miss the signals that matter in tool-calling settings. In this formulation, the relevant judgment is not whether a response is broadly helpful or stylistically preferred, but whether a structured sequence of tool invocations is correct with respect to the user request and the available tool schema (Agarwal et al., 15 Sep 2025).

The benchmark operationalizes tool-calling evaluation through pairwise preference judgments. Each example contains a tool catalog, a user query, a correct tool call sequence, and an incorrect tool call sequence. A reward model succeeds when it assigns a higher score to the correct call than to the incorrect one. This design makes FC-RewardBench a benchmark for structured correctness in function calling rather than for free-form response quality.

The paper frames the need for such a benchmark in two ways. First, no dedicated benchmark had existed to test reward models on function calling. Second, a useful benchmark should correlate with downstream performance when reward models are used for Best-of-$n$ selection or data filtering. This emphasis on downstream validity is central to the benchmark’s purpose: FC-RewardBench is not presented as a generic preference dataset, but as an evaluation instrument for reward providers in tool-using LLM systems.

2. Dataset construction and instance format

FC-RewardBench is derived from the single-turn splits of BFCL-v3. For each source example, the benchmark inherits the tool catalog, the user query, and the correct tool call or calls directly from BFCL-v3. Incorrect candidates are then generated by prompting a pool of 25 permissively licensed models ranging from 0.5B to 685B parameters to produce a function call for the same query. Their outputs are compared against ground truth, and only incorrect generations are retained. To prevent overrepresentation of any one query, one incorrect call is randomly sampled per instance, after which 1,500 examples are randomly selected to form the benchmark (Agarwal et al., 15 Sep 2025).

This construction procedure matters because it produces negatives that are model-generated rather than hand-authored. The resulting errors are often subtle, semantically plausible, and schema-aware. The paper therefore distinguishes FC-RewardBench from earlier RewardBench-style datasets not only by task domain but also by the nature of its negative examples: the rejected outputs are generated within the same function-calling regime that deployment systems must handle.

The evaluation target is pairwise ranking. For a reward model $r(x,y)$, the benchmark follows the Bradley-Terry preference formulation:

$$p(y_{+} \succ y_{-} \mid x) = \frac{\exp(r(x, y_{+}))}{\exp(r(x, y_{+})) + \exp(r(x, y_{-}))}.$$

Training on pairwise preference data $D=\{(x,y_+,y_-): y_+ \succ y_-\}$ uses

$$J(r) = \max_{r_\theta} \mathbb{E}_{(x, y_+, y_-) \sim D} \left[\log\left(\sigma(r_\theta(x, y_+) - r_\theta(x, y_-))\right)\right],$$

with reward-centering regularization

$$J_{reg}(r) = J(r) + \eta \mathbb{E}_{(x, y_+, y_-) \sim D} \left[(r_{\theta}(x, y_+) + r_{\theta}(x, y_-))^2\right].$$

At evaluation time, the rule is simple: the prediction is counted as correct if the correct call receives a higher score than the incorrect one.

3. Error taxonomy and judgment criteria

A defining feature of FC-RewardBench is its explicit emphasis on fine-grained tool-call failure modes. The benchmark is designed to test whether a judge can detect wrong function names, wrong parameter values, wrong numbers of tool calls, missing required parameters, missing optional parameters, incorrect parameter types, unexpected parameters, and incorrect output format. The paper stresses that many of these failures are not syntactic; they are semantic and schema-aware, which makes the benchmark more discriminative than simple format checks (Agarwal et al., 15 Sep 2025).

The error distribution reported for the 1,500-example benchmark is as follows:

Error type	Count
Incorrect parameter value	650
Incorrect function name	403
Incorrect number of functions	245
Missing optional parameter	78
Missing required parameter	45
Incorrect parameter type	43
Unexpected parameter	21
Incorrect output format	15

The largest category is incorrect parameter value, indicating that many benchmark examples hinge on semantically precise argument selection rather than coarse structural validity. The representative example highlighted in the paper changes player_count from 6 to 5, illustrating the intended granularity of judgment: the call can appear structurally well formed while still being incorrect in a way that matters for execution.

For LLM-as-judge evaluation, the two candidate tool-call sequences are presented in randomized order, and the judge is instructed to inspect tool existence, correct order, correct parameters, parameter validity and format, completeness with respect to the query, and the absence of unnecessary tools or parameters. This prompt design reflects the benchmark’s assumption that tool-calling competence requires schema-level and task-level scrutiny simultaneously.

4. Empirical findings and benchmark validity

The main FC-RewardBench experiments compare 8 reward models from RewardBench, spanning roughly 3B to 70B parameters, 6 LLM judges spanning 70B to 685B parameters, and the proposed ToolRM models. The paper reports three headline findings. First, general-purpose reward models underperform: models that do well on RewardBench do not transfer well to tool calling, and their performance is inconsistent. Second, LLM judges do better but are expensive: the judge LLMs exceed 80% accuracy across all tested models, but incur high computational cost due to their size. Third, ToolRM performs best: ToolRM-7B and ToolRM-14B achieve the highest accuracy on FC-RewardBench, and even ToolRM-1.5B beats the very large gpt-oss-120B judge while approaching larger Llama-4 models (Agarwal et al., 15 Sep 2025).

A central validation claim is that FC-RewardBench is a good proxy for downstream tool-use quality rather than an isolated synthetic test. To test this, the paper computes Pearson correlation between FC-RewardBench accuracy and Best-of-$n$ downstream accuracy with $n=32$, across six generator models, 11 reward models, and five benchmarks. The reported average correlation is 0.84, with generator-family-specific correlations ranging from 0.62 to 0.94. This result is presented as evidence that benchmark performance tracks practical reward-model utility in tool-calling workflows.

The article’s interpretation of failure modes is equally important. General-purpose reward models appear to miss the key signals of effective tool use, especially when errors are semantically subtle rather than obviously malformed. This suggests that standard natural-language reward modeling does not automatically transfer to structured function-calling evaluation.

5. ToolRM and the benchmark’s downstream use

FC-RewardBench was introduced together with ToolRM, a family of outcome reward models specialized for tool calling. The training data for ToolRM is synthesized from permissively licensed, open-weight LLMs using public function-calling datasets such as API-Gen, Schema-Guided Dialogue (SGD), and xlam-irrelevance. To avoid regurgitation, the pipeline obfuscates function and parameter names by replacing them with random strings and reordering schema keys. The final training set contains about 180,000 samples, including 85,000 single/multi-turn examples and 10,000 irrelevance examples, with the remainder drawn from the generated tool-use corpus described in the paper. ToolRM is built on Qwen2.5-Instruct backbones at 1.5B, 7B, and 14B parameters, with the final LM head replaced by a linear scalar reward head; training uses 1 epoch, learning rate $1\times10^{-6}$, a cosine schedule with 3% warmup, and reward centering coefficient 0.01 (Agarwal et al., 15 Sep 2025).

The benchmark is then linked to two practical deployment settings. In Best-of-32 inference, ToolRM improves performance across seven out-of-domain benchmarks, with gains especially strong for small LLMs. One reported example is Qwen3-0.6B + ToolRM-14B, which improves from 39.5% to 64.38%, a gain of 24.9 points, surpassing Qwen3-32B greedy at 63.8% and Llama-xLAM-2-70b-fc-r greedy at 63.6%. On BFCL-v3, xLAM-2-1B-fc-r + ToolRM-14B improves overall, Non-Live AST, and Live AST by 3.2, 6.5, and 6.2 points, while Qwen3-1.7B gains 5.3, 9.6, and 8.7 points on the same metrics.

The paper also evaluates ToolRM as a data-filtering mechanism. A Llama-3.1-8B-Instruct model is fine-tuned with LoRA on a 16K corpus assembled from APIGen-MT, SealTools, Glaive V2, and the Granite function-calling dataset, under three conditions: full 16K data, a random 8K subset, and the top 8K examples selected by ToolRM-14B. The reported average performances are 54.0% for the base model, 61.0% for full 16K fine-tuning, 58.4% for random 8K, and 62.5% for the ToolRM-filtered top 8K. The result indicates that reward-guided filtering can outperform both full-data training and random subsampling while using only half the data.

The benchmark therefore functions as more than a leaderboard. It also serves as an empirical validator for specialized outcome reward models, Best-of-$n$ reranking, and reward-guided training-data selection in tool-calling systems.

6. Scope, limitations, and relation to adjacent RewardBench variants

FC-RewardBench is explicitly scoped to single-turn BFCL-v3-derived examples, so multi-turn and interactive tool-use dynamics are only partially covered. The ToolRM paper also notes several open directions beyond the benchmark’s present scope: classification-based reward models are limited, future work could use generative verifiers with chain-of-thought reasoning, tool or environment state is not modeled, outcome and process reward models remain to be unified, and irrelevance errors remain a persistent issue because ToolRM can still prefer a tool call when the correct behavior is not to call a tool (Agarwal et al., 15 Sep 2025).

Later work has clarified FC-RewardBench’s position within a broader family of RewardBench-like datasets. "Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling" characterizes FC-RewardBench as a prior tool-call-level benchmark mainly about correct versus incorrect tool calls, and contrasts it with Plan-RewardBench, whose evaluation unit is the full multi-turn agent trajectory rather than a single-turn function-call output (Wang et al., 9 Apr 2026). This makes FC-RewardBench a benchmark for structured call correctness, whereas Plan-RewardBench targets planning quality, tool grounding, recovery, and refusal quality over entire trajectories.

The broader literature also contains similarly named but substantively different benchmarks. "Multimodal RewardBench" is a human-annotated benchmark for evaluating reward models or judges for vision-LLMs over 5,211 annotated triplets spanning general correctness, preference, knowledge, reasoning, safety, and VQA, rather than tool-call sequences (Yasunaga et al., 20 Feb 2025). "Long-form RewardBench" is a testbed for long-form generation across QA, RAG, Chat, Writing, and Reasoning, focusing on long responses rather than structured function invocation (Huang et al., 13 Mar 2026). In "Multi-Agent Collaborative Reward Design for Enhancing Reasoning in Reinforcement Learning," the paper’s rewardBench is the benchmark and training suite aligned with a collaborative multi-agent reward framework, again differing from the function-calling setting (Yang et al., 20 Nov 2025). "BaseReward" evaluates MM-RLHF-Reward Bench, VL-Reward Bench, and Multimodal RewardBench, and does not mention FC-RewardBench explicitly, so its results are adjacent to but not directly about the benchmark (Zhang et al., 19 Sep 2025).

A common misconception is therefore to treat FC-RewardBench as interchangeable with other RewardBench variants. The literature instead presents it as a benchmark with a specific technical object: pairwise reward-model evaluation for single-turn tool-calling correctness. Within that niche, its main significance is that it exposes a failure mode of conventional reward modeling—poor transfer from natural-language preference learning to schema-sensitive function-calling evaluation—and provides a benchmark whose scores correlate strongly with downstream Best-of-$r(x,y)$0 tool-use performance.