FocusCal Loss in iMAD Systems

• FocusCal Loss is a novel asymmetric loss function that combines focal loss, confidence penalty, and calibration error to optimize debate decision boundaries in iMAD systems.
• It selectively triggers multi-agent debate only when necessary, reducing token usage by up to 92% and preventing unnecessary compute overhead.
• Empirical results demonstrate up to 13.5% accuracy gains and robust zero-shot generalization across diverse QA and VQA tasks.

FocusCal Loss is a novel asymmetric objective function introduced for efficient and robust debate-decision classification in Intelligent Multi-Agent Debate (iMAD) systems. Its design addresses the challenge of selectively triggering multi-agent debate for LLM inference—the goal being to minimize token cost and avoid situations where debate either wastes compute on instances already solved by the base agent or flips a correct answer to an incorrect one. The FocusCal loss is a composition of asymmetric focal loss, confidence penalty, and expected calibration error terms. It is specifically crafted to optimize for critical decision boundaries relevant to the debate-triggering task and to provide reliable zero-shot generalization across heterogeneous question types and domains (Fan et al., 14 Nov 2025).

1. Background: Selective Debate Triggering in iMAD

In classic Multi-Agent Debate frameworks, every query is routed through a fixed debate protocol involving multiple agents and rounds, incurring 3×–5× compute costs compared to single-agent inference. Empirical analysis shows only 5%–19% of samples actually benefit from debate ("error → corrected"), with the remainder being either non-recoverable or already correct in the base agent. Furthermore, 3%–14% of debates can overturn a correct base answer—degrading end accuracy. Thus, there is a strong need for a selective, reliable gating mechanism to trigger debating only on "uncertain" or "recoverable" cases, maximizing both efficiency and accuracy (Fan et al., 14 Nov 2025).

2. Architecture: Structured Self-Critique and Hesitation Cues

iMAD first queries a single agent for a structured self-critique: (1) chain-of-thought justification for its answer $A_1$, (2) forced counter-argument for an alternative answer $A_2$, and (3) explicit verbalized confidences for both options (e.g., "I'm 0.85 confident in $A_1$ and 0.60 in $A_2$"). From this output, a feature extractor builds a 41-dimensional vector $z \in \mathbb{R}^{41}$ (surface cues, readability, syntax, POS statistics, and uncertainty markers). The debate-decision classifier $C$ receives $z$ plus the LLM confidence $p_\mathrm{LLM}$ and outputs $p$—the confidence that the single-agent answer is correct or irrecoverable—and $u$—a modelled hesitation or uncertainty score. The classifier is a 6-layer MLP encoder with parallel heads for $p$ and $u$.

3. Definition of FocusCal Loss

The FocusCal loss $L_\mathrm{FC}$, specific to debate-decision classification, is defined as

$$L_\mathrm{FC}(y,p,u) = L_\mathrm{AF}(y,p) + \lambda\,L_\mathrm{CP}(y,p,u) + \mu\,\mathrm{ECE}(y,p)$$

where

• $y \in \{0,1\}$ is the "single agent correct" label.
• $p$ is the classifier’s predicted probability for skipping debate.
• $u$ is the hesitation score.
• $\lambda, \mu > 0$ are hyperparameters.
• $L_\mathrm{AF}$ is the asymmetric focal loss, $L_\mathrm{CP}$ the confidence penalty, $\mathrm{ECE}$ the expected calibration error.

Details:

3.1 Asymmetric Focal Loss ($L_\mathrm{AF}$)

$$L_\mathrm{AF}(y,p) = \begin{cases} -\alpha_1 (1-p)^\gamma \ln p, & y=1 \ -\alpha_0 p^\gamma \ln(1-p), & y=0 \end{cases}$$

• $\gamma>0$ modulates the focus on hard examples.
• $\alpha_0 > \alpha_1$ penalizes highly confident skips on misclassified cases more than on correctly skipped ones.
• This controls the tradeoff between false positive (skipping when $y=0$) and false negative errors.

3.2 Confidence Penalty ($L_\mathrm{CP}$)

$$L_\mathrm{CP}(y,p,u) = \begin{cases} u^2, & y=0,\, p>\tau \ (1-u)^2, & y=1,\, p<\tau \ 0, & \text{otherwise} \end{cases}$$

• Encourages alignment between the auxiliary hesitation score $u$ and the debate-trigger decision.
• Reinforces high hesitation for false positive skips, and low hesitation for false negatives.

3.3 Calibration Error (ECE)

$$\mathrm{ECE} = \sum_{b=1}^B \frac{|I_b|}{N} \left| \frac{1}{|I_b|} \sum_{i \in I_b} p^{(i)} - \frac{1}{|I_b|} \sum_{i \in I_b} y^{(i)} \right|$$

• $B$ is the number of bins, $I_b$ is the set of samples in bin $b$.
• Encourages probabilistic calibration of the classifier.

The skip threshold $\tau$ is set globally (e.g., $\tau=0.7$), not optimized per dataset, enabling strong zero-shot generalization.

4. Training and Zero-Shot Generalization

The classifier is trained on ~10,000 examples from auxiliary QA/VQA datasets using only the self-critique outputs of single LLM agents. No tuning is done per downstream task. By focusing on interpretable hesitation cues and robust calibration, FocusCal-trained debate decision models generalize across domain shifts without per-task engineering or retraining (Fan et al., 14 Nov 2025).

5. Empirical Results and Ablations

FocusCal-optimized iMAD systems deliver multiple advantages over naive MAD or greedy gating:

• Up to 92% reduction in token usage compared to always-trigger MAD.
• Absolute accuracy gains of up to 13.5% over standard single-agent chain-of-thought and up to 5% over always-trigger MAD on diverse QA and VQA datasets (Table 3).
• The addition of self-critique in the prompt provides a 2–7% accuracy gain with only 5% additional token overhead.
• Ablation confirms that all three components of FocusCal are necessary—accuracy degrades by over 1% when any loss term is dropped.
• SHAP and PCA analyses identify linguistic markers of uncertainty as the most discriminative features for robust classification.

6. Practical Implications and Limitations

FocusCal enables deployment-scale iMAD by making MAD cost-effective and robust to spurious debate triggers. Its focus on asymmetric error penalties and calibration addresses the high cost of both false positives and false negatives in debate gating. The classifier can be interpreted and audited for hesitation features in deployment. Nonetheless, the system may still face challenges for short, unambiguous factual queries lacking strong uncertainty cues and in domains with few linguistic signals of hesitancy (Fan et al., 14 Nov 2025).

7. Extensions and Impact on iMAD Systems

By introducing a generalizable, asymmetric loss specifically structured for selective debate gating, FocusCal sets a precedent for other meta-debate classifiers and policy-learning objectives in agentic systems. The loss can in principle be adapted to streaming generation, online learning, or non-QA modalities (e.g., code generation), facilitating highly efficient, interpretable, and contextually aware debate policies in large-scale LLM deployments (Fan et al., 14 Nov 2025).