MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights

Abstract: Multilingual and multicultural benchmarks now cover dozens of languages and model families, but the resulting score landscapes remain metric-rich and insight-poor, necessitating fine-grained multilingual post-evaluation diagnosis. However, single LLMs and open-ended agents are easily swamped by the long, noisy diagnostic input, and no reusable taxonomy exists for it. To address this, we propose MADE, a Multilingual Agentic Diagnosing Engine that decomposes post-evaluation analysis into planning, aggregate analysis, instance-level case inspection, multilingual and cultural reflection, and grounded report synthesis. MADE is paired with an expert-led 54-query and 15-language diagnostic set, evaluated on top of a large-scale multilingual evaluation substrate (33 model families, 11 benchmarks, 26 languages, 34 cultures, 8.66M evaluation records). Experiments show that MADE outperforms the strongest shared baseline by 47% in diagnosis report quality and is preferred by human multilingual experts in 87.9% of pairwise comparisons. Applied with multilingual experts, MADE further surfaces four actionable findings on deployment, iteration, and cross-cultural pitfalls, turning benchmark score tables into model-selection and remediation guidance.

• The paper introduces a five-role agent architecture that automates fine-grained multilingual evaluation and provides actionable diagnostics.
• The paper leverages a reusable 54-query taxonomy and deterministic tool access to ground evidence and reduce hallucinations in model evaluation.
• The paper demonstrates significant performance gains over baselines in evidence grounding, uncertainty calibration, and diagnostic actionability.

MADE: A Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights

Motivation and Problem Formulation

The expansion of LLMs into diverse linguistic and cultural contexts has exposed several deficiencies in current benchmark-driven evaluation protocols. While multilanguage benchmarks provide extensive quantitative metrics, these aggregates yield limited actionable insight, failing to inform the why behind model failures or actionable how for remediation. Furthermore, the vast quantity and heterogeneity of raw evaluation records introduce scale and noise that defeat single LLMs and under-constrain general agentic baselines, and no reusable taxonomy systematically structures the space of post-evaluation diagnosis. This context motivates MADE, a Multilingual Agentic Diagnosing Engine, which formalizes and automates expert-driven post-evaluation diagnosis as a first-class task.

MADE Architecture and Workflow

MADE operationalizes large-scale multilingual evaluation through an explicit five-role agent division of labor: Planner, Evidence Analyst, Case Analyst, Language Reflector, and Reporter, each instantiated with a structured expert context and deterministic tool access.

Figure 1: The MADE pipeline routes multilingual queries through coordinated specialist agents to yield evidence-grounded diagnosis and actionable deployment maps.

The diagnostic workflow follows the observed practice of multilingual LLM specialists: the Planner triages and plans diagnostic scope and axes, the Evidence Analyst produces aggregate slice-level findings via a bounded ReAct loop, the Case Analyst retrieves and labels instance-level errors, the Language Reflector enforces cross-cutting multilingual and cultural sensitivity and evidence-grounding, and the Reporter synthesizes a final, evidence-tagged user-facing report. Grounding constraints enforce that every quantitative or sample-level claim be auditable to a deterministic tool invocation, thereby eliminating hallucinated conclusions and overgeneralized inference.

Diagnostic Taxonomy and Multilingual Expansion

Central to MADE is a reusable, expert-curated 54-query $\times$ 15-language diagnostic set covering dataset, instance, and iteration evidence levels, six fine-grained diagnostic categories (Task/Language, Capability, Compliance, Behavior, Culture, Improvement), and six query templates (weakness localization, comparison, evolution, correlation, advice, selection). The taxonomy is intentionally saturated along axes most relevant to model selection and real-world deployment, guaranteeing extensive and balanced coverage of actionable diagnostic needs.

Figure 2: Marginal distribution of the 54-query diagnostic set by evidence, category, and template axes for coverage.

Each query and its translations are professionally authored, back-translated, and audited for substrate alignment, ensuring cross-lingual validity and diagnostic executability.

Empirical Results

MADE is evaluated on a comprehensive multilingual and multicultural substrate: 33 model families, 11 benchmarks, 26 languages, and 34 cultures, constituting over 8.66M raw evaluation records. Performance is assessed through both automatic LLM-as-judge and human expert evaluation, covering seven rigorous dimensions: Requirement Fulfillment, Evidence Quality, Evidence Grounding, Structure, Multilingual Sensitivity, Diagnostic Actionability, and Uncertainty Calibration.

MADE achieves an aggregate judge score of 8.02 (SD 1.04), outperforming the strongest agentic baseline (opencode at 5.45) by 2.57 points and the best single-LLM baseline by 3.70 points. It is robust across all languages ($100\%$ valid completion) and consistently wins human expert preference in pairwise comparisons ($87.9\%$ over baselines across 12 languages). Margins are especially pronounced in evidence grounding ($+$3.83), uncertainty calibration ($+$3.43), and diagnostic actionability ($+$3.35), directly reflecting the impact of tool-grounding and role specialization.

Ablation studies confirm the additive benefit of each component and validate that performance gains originate from agentic structure and not backbone engineering alone. MADE’s runtime cost is higher than compact baselines ($7.26$ min/report vs. $2-3$ min), but well within offline diagnosis budgets, and justified by quality gains.

Diagnostic Insights and Deployment Implications

MADE transforms benchmarks into actionable guidance, surfacing deployment-relevant findings inaccessible to leaderboard-driven analysis:

• Contradictory Local vs. Global Leadership: Leaderboard superiority does not predict in-context or slice-level performance; MADE detects that overall winning models frequently lose on specific languages or capabilities relevant for deployment, flagging $12.0\%$ of slices with local reversals.
• Iteration Churn Auditing: MADE reveals that minimal aggregate deltas across model versions often conceal significant internal churn (sample-level fix/regress flips) or domain-specific regressions, enabling actionable regression auditing beyond headline metrics.
• Cultural Behavioral Divergence: Models tied on aggregate cultural scores diverge in behavior—to the extent of opposing failure modalities (output format vs. stereotype compliance)—a distinction critical for risk auditing, and undetectable from scalar scores.
• Decomposition of Low-Resource Weaknesses: Rather than attributing underperformance to a monolithic ‘low-resource’ label, MADE partitions failure modes into shared substrate bottlenecks, task-conditioned imbalances, and model-specific issues, mapping each to a tailored intervention policy.

These findings are structurally unavailable to naive leaderboard analysis and represent a shift from leaderboards to reproducible, slice-aware, culturally accountable diagnostic pipelines.

Theoretical and Practical Implications

MADE formalizes post-evaluation diagnosis as a first-class research and engineering task, introducing a reusable taxonomy, deterministic evidence tools, and a workflow reflecting expert specialization. This fundamentally addresses the "insight-poor" condition of prior aggregate benchmarking by rendering failure analysis auditable and actionable across both linguistic and cultural boundaries.

In the practical AI deployment cycle, MADE supports more robust model selection, release governance, failure remediation triage, and policy compliance audits. The explicit grounding and cultural reflection reduce hallucination and overclaiming in diagnosis, thus constraining risk of unintentional bias propagation or misalignment across regions and stakeholder groups.

Theoretically, MADE opens a modular architecture for extending post-evaluation diagnosis to future benchmarks, modalities (e.g., multimodal, code), or agent architectures. The decoupled toolchain and structured query expansion are amenable to both automated scaling and further specialization (e.g., lightweight analysts, closed-loop diagnose-then-improve cycles).

Conclusion

MADE establishes multilingual, evidence-grounded, and actionable post-evaluation diagnosis as a scalable, reproducible paradigm. Its multi-agent orchestration, structured taxonomy, and deterministic tool grounding deliver actionable insights unavailable to baseline scoring or unconstrained agentic systems. As the LLM field matures towards deployment at linguistic, cultural, and regulatory frontier, MADE’s workflow and design principles will be increasingly central—shifting the field from measuring to understanding and improving models (2606.07020).