BFCL-Hi: Hindi Benchmark for Function Calling

• BFCL-Hi is the Hindi adaptation of BFCL V2 that assesses LLMs’ ability to generate correct English function calls from Hindi conversational prompts.
• It evaluates structured output fidelity using AST matching and tests various scenarios including single, multiple, and parallel function calls.
• Benchmark results highlight that targeted Hindi post-training improves tool use, even when larger models do not always lead in accuracy.

BFCL-Hi is the Hindi adaptation of BFCL V2, a benchmark for evaluating whether instruction-following LLMs can correctly use tools or functions via structured function calls. In the Hindi benchmark suite introduced for evaluating instruction-tuned LLMs in Hindi, BFCL-Hi targets a specific cross-lingual setting: a model reads a Hindi conversational prompt or history and must generate correct function calls defined in English, including function name selection, argument structure, and argument values under the BFCL evaluation protocol (Kamath et al., 27 Aug 2025). This makes BFCL-Hi a benchmark for structured tool use rather than open-ended dialogue quality, and it is positioned around a production-relevant scenario in which Hindi-speaking users interact with systems whose back-end tools and APIs remain English-specified.

1. Definition, scope, and rationale

BFCL-Hi stands for Berkeley Function-Calling Leaderboard—Hindi. It is designed to test whether a model can read a Hindi conversational prompt or history, decide which tool or tools to call, and produce the correct function call or calls under the BFCL evaluation protocol (Kamath et al., 27 Aug 2025).

The benchmark assesses three closely related capabilities. First, it evaluates tool or function use across single, multiple, and parallel function calls. Second, it evaluates tool relevance, namely whether a model attempts a function call when tools are relevant and refrains when they are irrelevant. Third, it evaluates structured output fidelity through adherence to the prescribed function signature and argument schema, measured by AST matching.

The rationale for a Hindi-specific benchmark is not merely linguistic translation. BFCL-Hi deliberately probes cross-lingual grounding from Hindi prompts to English tool definitions. In realistic Indian deployments, end users often converse in Hindi while back-end tools and APIs remain defined in English. The benchmark therefore asks whether a Hindi LLM can understand Hindi queries and generate accurate, English-specified function calls. This is directly relevant to production agent systems, customer-support assistants, and government-service chatbots serving Hindi speakers.

A common misconception is that strong general instruction following in Hindi is sufficient for strong function-calling behavior. BFCL-Hi is constructed around a narrower and more operationally constrained competence: producing schema-conformant structured calls, not merely generating fluent or instruction-compliant Hindi text. The suite explicitly treats this as a complementary rather than redundant capability.

2. Construction methodology and dataset format

BFCL-Hi is derived from BFCL V2, an English benchmark containing diverse function-calling scenarios, including simple, multiple, and parallel calls, together with relevance and irrelevance diagnostics (Kamath et al., 27 Aug 2025). Its construction follows a translate-and-verify versus from-scratch distinction used across the larger Hindi suite, but BFCL-Hi itself is purely translation-based: the conversational history is machine-translated into Hindi using Google Cloud Platform, while the function or tool schemas are retained in English.

The curation pipeline is defined as follows. Each BFCL V2 example is taken in its original JSON form with conversation and function specifications. Only the conversation history is translated to Hindi via GCP. The function array is kept exactly as in English, including names, descriptions, and parameter schema. The original ground truth is preserved for categories with a defined answer, namely simple, multiple, and parallel calls. For relevance and irrelevance scenarios, the entries are used to check whether the model correctly attempts a function call when appropriate; those entries do not include ground-truth function outputs.

This design intentionally preserves English function names, descriptions, and parameter keys or types in order to test cross-lingual grounding. Function parameters were not localized into Hindi by design, although the paper notes that fully localized parameter schemas may be desirable in future work. A plausible implication is that BFCL-Hi evaluates a practically important but not exhaustive slice of Hindi tool use: it is aligned to mixed-language deployments rather than fully localized Hindi tool ecosystems.

BFCL-Hi contains 2,251 items. It includes BFCL V2 scenario types—simple, multiple, parallel calls, plus relevance and irrelevance diagnostics—but the paper does not report per-category counts or splits. The benchmark is test-only; training and validation splits are not applicable. Difficulty is not explicitly stratified, although the paper notes that difficulty naturally varies by scenario type, with multiple and parallel calls tending to be harder than single calls.

The data format mirrors BFCL V2 and uses JSON. Each entry contains Hindi conversation messages and a functions array in English. Each function includes a name, description, and parameters represented as a JSON Schema-like object with keys, types, and constraints. Ground truth is present for simple, multiple, and parallel calls; relevance and irrelevance cases are diagnostic rather than exact-argument supervision.

3. Evaluation protocol and scoring semantics

The expected model behavior is to produce one or more function calls conforming to the provided schemas, including names, argument keys and values, and structural format (Kamath et al., 27 Aug 2025). Multiple calls may require ordering or parallel semantics as in BFCL V2, and the evaluator is designed to assess structural correctness rather than superficial text overlap.

Scoring uses the BFCL AST methodology. The model’s function-call output is parsed and compared with ground truth through AST matching, covering function names, required arguments, value equivalence or validity, and structural equivalence, including order-invariance where relevant. Relevance and irrelevance entries are diagnostic: they verify whether the model attempts a call or refrains from doing so, and they do not include ground-truth function outputs.

The reported metric is overall accuracy on a 1–100 scale:

$$\text{Acc} = \frac{\sum_{i=1}^{N} \mathbf{1}\{\text{AST}(\hat{y}_i) = \text{AST}(y_i)\}}{N} \times 100$$

Here, $N$ is the number of scored examples, meaning those with ground-truth calls; $\hat{y}_i$ is the model’s predicted function-call output, and $y_i$ is the ground truth. The AST comparator returns 1 for a structurally correct match and 0 otherwise.

The paper reports a single overall accuracy for BFCL-Hi. It does not provide micro or macro breakdowns, nor additional metrics such as F1 or ROUGE. No human evaluation is used for BFCL-Hi scoring; reproducibility is determined by the BFCL AST checker and the released dataset. The paper does not specify BFCL-Hi prompt templates, decoding parameters, seeds, hardware, or runtime.

A schematic illustration in the paper describes a Hindi request to book a train ticket to Jaipur for Saturday evening, with English tools such as book_train_ticket(origin, destination, date, time, seat_class) and get_weather(city, date). The expected output is a single function call to book_train_ticket with correctly filled arguments, and the AST checker validates structural correctness and parameters. The illustration is explicitly schematic rather than verbatim.

4. Benchmark results and comparative patterns

Benchmarking on BFCL-Hi reports the following accuracies (Kamath et al., 27 Aug 2025).

Model	Class	Accuracy
Nemotron-4-Mini-Hindi-4B-Instruct	SLM	52.82
Gemma-2-9b-it	SLM	50.51
Aya-expanse-8b	SLM	36.56
Llama-3.2-3B-Instruct	SLM	33.81
Gemma-2-2b-it	SLM	32.96
Llama-3.1-8B-Instruct	SLM	31.23
Krutrim-2-instruct	SLM	26.88
Gemma-3-27b-it	LLM	62.42
GPT-OSS-120B (reasoning low)	LLM	61.26
Qwen3-235B-A22B-FP8 (reasoning off)	LLM	59.88
GPT-OSS-20B (reasoning low)	LLM	54.60
Llama-3.1-405B	LLM	49.53
Sarvam-M (reasoning off)	LLM	48.60
Mistral-Small-3.2-24B-Instruct-2506	LLM	41.45

Among small LLMs, Nemotron-4-Mini-Hindi-4B-Instruct is the top SLM on BFCL-Hi with 52.82. Among LLMs, Gemma-3-27b-it obtains the highest BFCL-Hi score at 62.42. Nemotron-Mini-4B-Instruct was not reported on BFCL-Hi.

The comparative interpretation supplied in the paper is limited but specific. Language-specific post-training appears to help tool use: Nemotron-4-Mini-Hindi-4B-Instruct, despite being a 4B model, strongly leads among SLMs, indicating that targeted Hindi alignment can improve function-calling accuracy. The results also show that bigger is not always better: several very large LLMs trail Gemma-3-27b-it on BFCL-Hi. This suggests that cross-lingual tool grounding depends on post-training and interface alignment, not solely on parameter count.

The paper does not report statistical significance tests or detailed error analysis for BFCL-Hi. It notes only general failure modes typical of function-calling tasks: selecting an irrelevant function, producing natural language instead of the required structured call, missing required arguments, wrong argument types or values, and incorrect multi-call or parallel-call structure. The AST-based scorer is intended to surface these failure types.

5. Position within the Hindi evaluation suite

BFCL-Hi is one of five datasets in the Hindi LLM evaluation suite, alongside IFEval-Hi, MT-Bench-Hi, GSM8K-Hi, and ChatRAG-Hi (Kamath et al., 27 Aug 2025). Its role within the suite is orthogonal rather than overlapping.

IFEval-Hi tests verifiable instruction-following behaviors in Hindi, including formatting and content constraints, with human-created Indian cultural themes. BFCL-Hi complements this by testing tool use via structured calls, and strong IFEval-Hi performance does not guarantee function-calling accuracy.

MT-Bench-Hi evaluates multi-turn conversational ability judged by an LLM, using Hindi prompts with cultural adaptation. BFCL-Hi instead focuses on correctness of structured tool calls rather than conversational quality. Taken together, the two datasets assess agentic competence as the combination of dialogue and tool execution.

GSM8K-Hi measures Hindi math reasoning via translate-and-verify and emphasizes step-by-step problem solving. BFCL-Hi measures agent or tool execution, which the paper characterizes as an orthogonal capability. The paper explicitly notes that some models excel in math, such as GPT-OSS-120B, but are not necessarily best in function calling.

ChatRAG-Hi evaluates conversational question answering with retrieved Hindi context and uses F1 scoring. BFCL-Hi evaluates tool calling without external retrieval, focusing on API-like execution. Cross-dataset results exhibit specialization: Llama-3.1-405B leads on ChatRAG-Hi but not on BFCL-Hi, while Gemma-3-27b-it leads BFCL-Hi.

6. Limitations, ethics, and practical use

BFCL-Hi has several explicit limitations (Kamath et al., 27 Aug 2025). The conversational content is machine-translated without human verification, so residual translation artifacts may affect clarity or intent. No inter-annotator agreement statistics apply because there was no human annotation. The evaluation relies on BFCL’s AST-based automatic checker rather than multi-stage human validation.

A second limitation is that function names and parameters remain in English. This reflects many real deployments, but it constrains the benchmark’s ability to evaluate fully localized Hindi tool ecosystems. The paper suggests translating parameter schemas in future work. A plausible implication is that BFCL-Hi should not be read as a complete test of Hindi-language agent tooling; it is a test of Hindi-to-English tool mediation.

A third limitation concerns relevance categories. These entries do not have ground-truth function outputs and are used only to verify whether models attempt calls appropriately. Accordingly, BFCL-Hi’s headline accuracy is computed on scored examples with ground-truth calls, while relevance and irrelevance entries function as diagnostics.

The ethical framing is narrow. BFCL-Hi does not target safety or harm directly. The paper states, however, that any tool-using agent must be vetted for safe tool execution. Structured correctness is presented as a step toward reliable agent behavior, not a substitute for safety audits.

In practical terms, the paper does not provide licensing details, dataset URLs, or repository links for BFCL-Hi. The described usage procedure is straightforward: load the JSON entries, render the Hindi conversation into the model prompt, include the English function schemas as tool definitions, capture the model’s tool-call outputs, and evaluate them with the BFCL AST checker. Overall accuracy is then computed on categories with ground-truth calls, while relevance and irrelevance examples are treated as diagnostics. In the absence of a provided repository, the setup can be replicated by using BFCL V2 structure, replacing the conversation text with Hindi GCP translation, retaining English tool schemas, and applying AST-based matching for scoring.

BFCL-Hi therefore occupies a specific place in Hindi LLM evaluation: it is a test-only, machine-translated benchmark for cross-lingual function calling, centered on the ability to read Hindi conversational context and emit correct English-defined tool calls. Its empirical results highlight strong gains from Hindi-specific post-training in structured tool use and underscore that general model scale does not by itself determine function-calling accuracy.