---
title: 'FC-RewardBench: Tool-Calling Evaluation'
url: https://www.emergentmind.com/topics/fc-rewardbench
type: topic
---

# FC-RewardBench: Tool-Calling Evaluation

FC-RewardBench is a pairwise preference benchmark for tool calling introduced in "ToolRM: Outcome Reward Models for Tool-Calling Large Language Models" to evaluate whether reward models can judge tool-calling or function-calling outputs correctly rather than merely rank natural-language responses. It is built from BFCL v3 single-turn function-calling data, contains 1,500 unique examples, and presents each example as a comparison between a correct tool call sequence and an incorrect tool call sequence under a shared tool catalog and user query. The benchmark was proposed to fill a gap left by prior reward-model benchmarks, which emphasize chat quality, reasoning, safety, factuality, style, or math/code, but not whether a model produces a valid and useful sequence of function calls [2509.11963].

## 1. Origin and problem setting

FC-RewardBench was introduced in response to a benchmarking gap in reward modeling for tool use. The motivating claim is that existing reward models are trained primarily on natural language preference data and therefore often miss the signals that matter in tool-calling settings. In this formulation, the relevant judgment is not whether a response is broadly helpful or stylistically preferred, but whether a structured sequence of tool invocations is correct with respect to the user request and the available tool schema [2509.11963].

The benchmark operationalizes tool-calling evaluation through pairwise preference judgments. Each example contains a tool catalog, a user query, a correct tool call sequence, and an incorrect tool call sequence. A reward model succeeds when it assigns a higher score to the correct call than to the incorrect one. This design makes FC-RewardBench a benchmark for structured correctness in function calling rather than for free-form response quality.

The paper frames the need for such a benchmark in two ways. First, no dedicated benchmark had existed to test reward models on function calling. Second, a useful benchmark should correlate with downstream performance when reward models are used for Best-of-\(n\) selection or data filtering. This emphasis on downstream validity is central to the benchmark’s purpose: FC-RewardBench is not presented as a generic preference dataset, but as an evaluation instrument for reward providers in tool-using LLM systems.

## 2. Dataset construction and instance format

FC-RewardBench is derived from the single-turn splits of BFCL-v3. For each source example, the benchmark inherits the tool catalog, the user query, and the correct tool call or calls directly from BFCL-v3. Incorrect candidates are then generated by prompting a pool of 25 permissively licensed models ranging from 0.5B to 685B parameters to produce a function call for the same query. Their outputs are compared against ground truth, and only incorrect generations are retained. To prevent overrepresentation of any one query, one incorrect call is randomly sampled per instance, after which 1,500 examples are randomly selected to form the benchmark [2509.11963].

This construction procedure matters because it produces negatives that are model-generated rather than hand-authored. The resulting errors are often subtle, semantically plausible, and schema-aware. The paper therefore distinguishes FC-RewardBench from earlier RewardBench-style datasets not only by task domain but also by the nature of its negative examples: the rejected outputs are generated within the same function-calling regime that deployment systems must handle.

The evaluation target is pairwise ranking. For a reward model \(r(x,y)\), the benchmark follows the Bradley-Terry preference formulation:
$$
p(y_{+} \succ y_{-} \mid x) = \frac{\exp(r(x, y_{+}))}{\exp(r(x, y_{+})) + \exp(r(x, y_{-}))}.
$$
Training on pairwise preference data \(D=\{(x,y_+,y_-): y_+ \succ y_-\}\) uses
$$
J(r) = \max_{r_\theta} \mathbb{E}_{(x, y_+, y_-) \sim D} \left[\log\left(\sigma(r_\theta(x, y_+) - r_\theta(x, y_-))\right)\right],
$$
with reward-centering regularization
$$
J_{reg}(r) = J(r) + \eta \mathbb{E}_{(x, y_+, y_-) \sim D} \left[(r_{\theta}(x, y_+) + r_{\theta}(x, y_-))^2\right].
$$
At evaluation time, the rule is simple: the prediction is counted as correct if the correct call receives a higher score than the incorrect one.

## 3. Error taxonomy and judgment criteria

A defining feature of FC-RewardBench is its explicit emphasis on fine-grained tool-call failure modes. The benchmark is designed to test whether a judge can detect wrong function names, wrong parameter values, wrong numbers of tool calls, missing required parameters, missing optional parameters, incorrect parameter types, unexpected parameters, and incorrect output format. The paper stresses that many of these failures are not syntactic; they are semantic and schema-aware, which makes the benchmark more discriminative than simple format checks [2509.11963].

The error distribution reported for the 1,500-example benchmark is as follows:

| Error type | Count |
|---|---:|
| Incorrect parameter value | 650 |
| Incorrect function name | 403 |
| Incorrect number of functions | 245 |
| Missing optional parameter | 78 |
| Missing required parameter | 45 |
| Incorrect parameter type | 43 |
| Unexpected parameter | 21 |
| Incorrect output format | 15 |

The largest category is incorrect parameter value, indicating that many benchmark examples hinge on semantically precise argument selection rather than coarse structural validity. The representative example highlighted in the paper changes `player_count` from 6 to 5, illustrating the intended granularity of judgment: the call can appear structurally well formed while still being incorrect in a way that matters for execution.

For LLM-as-judge evaluation, the two candidate tool-call sequences are presented in randomized order, and the judge is instructed to inspect tool existence, correct order, correct parameters, parameter validity and format, completeness with respect to the query, and the absence of unnecessary tools or parameters. This prompt design reflects the benchmark’s assumption that tool-calling competence requires schema-level and task-level scrutiny simultaneously.

## 4. Empirical findings and benchmark validity

The main FC-RewardBench experiments compare 8 reward models from RewardBench, spanning roughly 3B to 70B parameters, 6 LLM judges spanning 70B to 685B parameters, and the proposed ToolRM models. The paper reports three headline findings. First, general-purpose reward models underperform: models that do well on RewardBench do not transfer well to tool calling, and their performance is inconsistent. Second, LLM judges do better but are expensive: the judge LLMs exceed 80% accuracy across all tested models, but incur high computational cost due to their size. Third, ToolRM performs best: ToolRM-7B and ToolRM-14B achieve the highest accuracy on FC-RewardBench, and even ToolRM-1.5B beats the very large gpt-oss-120B judge while approaching larger Llama-4 models [2509.11963].

A central validation claim is that FC-RewardBench is a good proxy for downstream tool-use quality rather than an isolated synthetic test. To test this, the paper computes Pearson correlation between FC-RewardBench accuracy and Best-of-\(n\) downstream accuracy with \(n=32\), across six generator models, 11 reward models, and five benchmarks. The reported average correlation is 0.84, with generator-family-specific correlations ranging from 0.62 to 0.94. This result is presented as evidence that benchmark performance tracks practical reward-model utility in tool-calling workflows.

The article’s interpretation of failure modes is equally important. General-purpose reward models appear to miss the key signals of effective tool use, especially when errors are semantically subtle rather than obviously malformed. This suggests that standard natural-language reward modeling does not automatically transfer to structured function-calling evaluation.

## 5. ToolRM and the benchmark’s downstream use

FC-RewardBench was introduced together with ToolRM, a family of outcome reward models specialized for tool calling. The training data for ToolRM is synthesized from permissively licensed, open-weight LLMs using public function-calling datasets such as API-Gen, Schema-Guided Dialogue (SGD), and xlam-irrelevance. To avoid regurgitation, the pipeline obfuscates function and parameter names by replacing them with random strings and reordering schema keys. The final training set contains about 180,000 samples, including 85,000 single/multi-turn examples and 10,000 irrelevance examples, with the remainder drawn from the generated tool-use corpus described in the paper. ToolRM is built on Qwen2.5-Instruct backbones at 1.5B, 7B, and 14B parameters, with the final LM head replaced by a linear scalar reward head; training uses 1 epoch, learning rate \(1\times10^{-6}\), a cosine schedule with 3% warmup, and reward centering coefficient 0.01 [2509.11963].

The benchmark is then linked to two practical deployment settings. In Best-of-32 inference, ToolRM improves performance across seven out-of-domain benchmarks, with gains especially strong for small language models. One reported example is Qwen3-0.6B + ToolRM-14B, which improves from 39.5% to 64.38%, a gain of 24.9 points, surpassing Qwen3-32B greedy at 63.8% and Llama-xLAM-2-70b-fc-r greedy at 63.6%. On BFCL-v3, xLAM-2-1B-fc-r + ToolRM-14B improves overall, Non-Live AST, and Live AST by 3.2, 6.5, and 6.2 points, while Qwen3-1.7B gains 5.3, 9.6, and 8.7 points on the same metrics.

The paper also evaluates ToolRM as a data-filtering mechanism. A Llama-3.1-8B-Instruct model is fine-tuned with LoRA on a 16K corpus assembled from APIGen-MT, SealTools, Glaive V2, and the Granite function-calling dataset, under three conditions: full 16K data, a random 8K subset, and the top 8K examples selected by ToolRM-14B. The reported average performances are 54.0% for the base model, 61.0% for full 16K fine-tuning, 58.4% for random 8K, and 62.5% for the ToolRM-filtered top 8K. The result indicates that reward-guided filtering can outperform both full-data training and random subsampling while using only half the data.

The benchmark therefore functions as more than a leaderboard. It also serves as an empirical validator for specialized outcome reward models, Best-of-\(n\) reranking, and reward-guided training-data selection in tool-calling systems.

## 6. Scope, limitations, and relation to adjacent RewardBench variants

FC-RewardBench is explicitly scoped to single-turn BFCL-v3-derived examples, so multi-turn and interactive tool-use dynamics are only partially covered. The ToolRM paper also notes several open directions beyond the benchmark’s present scope: classification-based reward models are limited, future work could use generative verifiers with chain-of-thought reasoning, tool or environment state is not modeled, outcome and process reward models remain to be unified, and irrelevance errors remain a persistent issue because ToolRM can still prefer a tool call when the correct behavior is not to call a tool [2509.11963].

Later work has clarified FC-RewardBench’s position within a broader family of RewardBench-like datasets. "Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling" characterizes FC-RewardBench as a prior tool-call-level benchmark mainly about correct versus incorrect tool calls, and contrasts it with Plan-RewardBench, whose evaluation unit is the full multi-turn agent trajectory rather than a single-turn function-call output [2604.08178]. This makes FC-RewardBench a benchmark for structured call correctness, whereas Plan-RewardBench targets planning quality, tool grounding, recovery, and refusal quality over entire trajectories.

The broader literature also contains similarly named but substantively different benchmarks. "Multimodal RewardBench" is a human-annotated benchmark for evaluating reward models or judges for vision-language models over 5,211 annotated triplets spanning general correctness, preference, knowledge, reasoning, safety, and VQA, rather than tool-call sequences [2502.14191]. "Long-form RewardBench" is a testbed for long-form generation across QA, RAG, Chat, Writing, and Reasoning, focusing on long responses rather than structured function invocation [2603.12963]. In "Multi-Agent Collaborative Reward Design for Enhancing Reasoning in Reinforcement Learning," the paper’s rewardBench is the benchmark and training suite aligned with a collaborative multi-agent reward framework, again differing from the function-calling setting [2511.16202]. "BaseReward" evaluates MM-RLHF-Reward Bench, VL-Reward Bench, and Multimodal RewardBench, and does not mention FC-RewardBench explicitly, so its results are adjacent to but not directly about the benchmark [2509.16127].

A common misconception is therefore to treat FC-RewardBench as interchangeable with other RewardBench variants. The literature instead presents it as a benchmark with a specific technical object: pairwise reward-model evaluation for single-turn tool-calling correctness. Within that niche, its main significance is that it exposes a failure mode of conventional reward modeling—poor transfer from natural-language preference learning to schema-sensitive function-calling evaluation—and provides a benchmark whose scores correlate strongly with downstream Best-of-\(n\) tool-use performance.

Source: https://www.emergentmind.com/topics/fc-rewardbench