DiaFORGE: Enterprise API Disambiguation

• DiaFORGE is a dialogue framework that employs synthetic dialogue generation, supervised fine-tuning, and dynamic evaluation to address API disambiguation challenges.
• It utilizes a three-stage pipeline—dialogue synthesis, LoRA-based fine-tuning, and a dynamic DiaBENCH loop—to optimize tool-call accuracy and mitigate input underspecification.
• Empirical results show significant performance gains, with up to 27 percentage point improvements over leading LLM baselines in enterprise API integration.

DiaFORGE (Dialogue Framework for Organic Response Generation & Evaluation) is a disambiguation-centric pipeline for robust fine-tuning and evaluation of LLMs designed to invoke enterprise APIs. Its architecture and methodology specifically address failures in LLM-based tool-calling where highly similar APIs compete for user intent and inputs are frequently underspecified. DiaFORGE integrates synthetic dialogue generation, supervised fine-tuning on reasoning traces, and agentic benchmarking within a unified framework, supported by an open corpus of rigorously validated, disambiguation-focused enterprise dialogues (Hathidara et al., 4 Jul 2025).

1. Three-Stage Pipeline Architecture

DiaFORGE is organized as a three-stage pipeline:

1. Dialogue Synthesis (UTC-Gen Engine): A simulator generates multi-turn, persona-conditioned dialogues centering on tool-disambiguation challenges. Given an enterprise tool set $$\mathcal{T} = \{\tau_1, ..., \tau_{|\mathcal{T}|}\}$$, each tool $\tau_i$ is specified as $(\text{name}_i, \text{desc}_i, \text{params}_i)$, with arguments encoded per JSON-Schema. A “gold” tool $\tau^*$ is selected, and the top $k$ closest distractor APIs $\mathcal{D}_k(\tau^*)$ are retrieved via an embedding $\phi(\tau)\in\mathbb{R}^d$. The distractor pool $$\mathcal{C}_k(\tau^*) = \{\tau^*\} \cup \mathcal{D}_k(\tau^*)$$ forms a competitive candidate set. User personae and concrete slot values are drawn from conditional distributions ($p \sim \pi^{(k)}$, $g \sim P_\text{goal}$, argument values $\tau_i$0), then a user-proxy and assistant interact under a policy pair $\tau_i$1 in two phases: (i) tool selection via iterative clarification, and (ii) argument filling for required slots. Dialogue ends upon emission of a valid, schema-conformant tool-call or a turn cap.
2. Supervised Fine-Tuning (SFT): The synthesized dialogues, after multi-stage functional and LLM validation as well as random human spot-checks, provide a dataset $\tau_i$2 of $\tau_i$3 annotated dialogues. For SFT, each assistant turn $\tau_i$4 is split into a context $\tau_i$5 and target $\tau_i$6, with loss masking so the context tokens are not penalized. Cross-entropy loss $\tau_i$7 is minimized, optionally with an auxiliary disambiguation loss $\tau_i$8 (set to zero in the published runs). Optimization uses LoRA adapter tuning (rank $\tau_i$9, scale $(\text{name}_i, \text{desc}_i, \text{params}_i)$0), AdamW with a peak learning rate of $(\text{name}_i, \text{desc}_i, \text{params}_i)$1, cosine schedule, and a single epoch in 8-bit precision.
3. Dynamic Evaluation (DiaBENCH Agentic Loop): A dynamic benchmark, DiaBENCH, comprises 119 held-out tools with their own dialogues. Assistants are deployed in an on-policy agentic “live” loop where user-proxy actions are generated interactively. Metrics include tool-call accuracy (Acc), false-positive rate (FTR), and abstention rate (TAR), alongside precision/recall at tool and parameter-key levels. Next-turn static evaluation is supplemented by these fully dynamic rollouts to assess goal completion (Hathidara et al., 4 Jul 2025).

2. Technical Methodology and Mathematical Formalism

DiaFORGE’s data generation, fine-tuning, and evaluation stages are all mathematically formalized:

• Dialogue Generation:
  • Distractor selection uses nearest-neighbor search in the tool-embedding space.
  • Persona and goal are sampled from conditional distributions tailored to the task.
  • User and assistant policies $(\text{name}_i, \text{desc}_i, \text{params}_i)$2 and $(\text{name}_i, \text{desc}_i, \text{params}_i)$3 govern dialogue rollout, respecting a two-phase structure for disambiguation and argument filling.
• Fine-Tuning Objective:
  • Main loss: $(\text{name}_i, \text{desc}_i, \text{params}_i)$4 where $(\text{name}_i, \text{desc}_i, \text{params}_i)$5.
  • Turn-slicing produces per-turn supervision, and only target tokens contribute to the loss.
• Dynamic Evaluation Metrics:
  • First tool-invocation in each test dialogue $(\text{name}_i, \text{desc}_i, \text{params}_i)$6: $(\text{name}_i, \text{desc}_i, \text{params}_i)$7; gold reference $(\text{name}_i, \text{desc}_i, \text{params}_i)$8.
  • Tool-call accuracy:

  $(\text{name}_i, \text{desc}_i, \text{params}_i)$9 - False-positive rate (FTR) and abstention rate (TAR) are similarly defined.

• Pseudocode:

  • SFT iterates over masked turn slices, applying cross-entropy loss.
  • The evaluation loop simulates full multi-agent interactions, dynamically sampling user turns until correct tool-call or turn cap.

3. Quantitative Performance and Empirical Outcomes

On the DiaBENCH dynamic benchmark, DiaFORGE-tuned models exhibit substantial improvements over leading LLM tool-calling baselines:

Model	Accuracy (Acc)	FTR	TAR	Δ Accuracy (vs GPT-4o / Claude)
Llama-3.3-Nemotron-DiaFORGE-49B	0.89	0.06	0.03	+27 pp / +50 pp
GPT-4o-20241120 (CAPO-optimized)	0.62	0.02	0.36	baseline / baseline
Claude-3.5-Sonnet-20241022 (CAPO-op)	0.39	0.03	0.55
Llama-3.2-DiaFORGE-3B	0.80			+18 pp / +41 pp

Even the smallest DiaFORGE-tuned model surpasses GPT-4o by 18 percentage points (pp) and Claude-3.5-Sonnet by 41 pp in accuracy. Larger models achieve up to 0.89 accuracy, a 27 pp gain over GPT-4o and 50 pp over Claude, with low false-positive and abstention rates (Hathidara et al., 4 Jul 2025).

4. Corpus and Annotation Schema

DiaFORGE releases a corpus pairing approximately 5,000 enterprise API specifications (in JSON-Schema format) with synthetic, persona-conditioned multistep dialogues. Each corpus entry contains:

• Persona and seed tool
• Sampled distractors and canonical goal
• A sequence of dialogue turns, including for each assistant turn:
  • Private reasoning trace (> ...)
  • Public response
  • Tool-call stub if present

Annotations cover user and assistant utterances, validator results, and human spot-check status. Dialogue length distributions show 60% of dialogues fit within five turns, with tools averaging $\tau^*$0 parameters (mean ≈ 2.3), and a typical disambiguation:filling turn ratio of 2:1.

5. Flowcharts and Integration Diagrams

The pipeline is represented in detailed workflow diagrams:

• UTC-Gen Engine: Orchestrates transition from metadata to dialogue synthesis and cascaded multi-view validation, feeding into the training set.
• Turn-Slicing SFT Flow: Converts dialogues to per-turn (context, target) pairs with token-level loss masking.
• Dynamic User Simulation Diagram: Depicts sampling and multi-way voting (n=3 sampling, m=3 voting) for dynamic evaluation robustness.

The integration sequence is:

1. Data generation via UTC-Gen, yielding validated training dialogues
2. Supervised fine-tuning of open LLMs with LoRA and masked cross-entropy
3. Static and dynamic evaluation with DiaBENCH agentic loop and reporting of all critical metrics

6. Reproducibility and Research Utility

All key definitions, formulas, data schemas, and pseudocode required to reconstruct the entire DiaFORGE procedure are provided (Hathidara et al., 4 Jul 2025). The released corpus and evaluation suite offer a practical blueprint for enterprise-ready tool-calling LLMs and facilitate direct benchmarking and ablation studies for future research. The framework’s explicit focus on disambiguation under enterprise constraints distinguishes its approach and empirical results from prior general-purpose LLM fine-tuning schemes.