Correlation between LLM‑as‑a‑judge scores and ground‑truth benchmark outcomes

Determine the correlation between Mimosa’s LLM‑as‑a‑judge aggregate score and ground‑truth benchmark metrics to assess the reliability and effectiveness of judge‑guided workflow optimization.

Background

Mimosa uses an LLM‑as‑a‑judge to guide iterative refinement of multi‑agent workflows, but this signal is only approximate. Establishing how well judge scores align with ground‑truth task success is necessary to quantify the sample efficiency and validity of the optimization process.

References

This judge provides approximate rather than exact quality signal; its directional accuracy is sufficient to drive convergence, and its correlation with benchmark ground-truth metrics is left to future work.

— Mimosa Framework: Toward Evolving Multi-Agent Systems for Scientific Research  (2603.28986 - Legrand et al., 30 Mar 2026) in Section 3.1.2 (Workflow Discovery Problem)

Whether a judge that clears the screen survives sustained optimization---under drift and Goodhart pressure---is the question our necessary condition leaves open, and the natural next step.

— Competence, Not Accuracy: A Diagnostic for Reference-Free Judge Gates in Skill Optimization  (2608.18719 - Chen et al., 19 Aug 2026) in Section Conclusion

He et al. survey LLM-as-a-Judge practice in software engineering and identify a lack of validation against ground truth as one of the paradigm's central open problems.

— Can LLMs Fix It Without Code? Toward Automated Verification of No-Code Bug Fixes  (2610.11963 - Torun et al., 8 Oct 2026) in Section 2.3, “LLM Evaluation in Software Engineering” (Related Work)