Validation of LLM-as-a-judge alignment with human judgment for taxonomy evaluation

Establish whether the LLM-based evaluation used to score the Sparse Autoencoder-derived reasoning taxonomy—specifically the consistency, completeness, and independence metrics—aligns with true human judgment, thereby validating the reliability of LLM-as-a-judge for this task.

Background

To assess unsupervised taxonomies of reasoning mechanisms, the paper employs LLM-as-a-judge to compute consistency (F1), completeness (confidence), and independence (semantic orthogonality) scores. While cost-effective and scalable, the authors note that these LLM-generated evaluations should be checked against human judgments to ensure validity.

Validating this alignment would strengthen confidence in taxonomy comparisons across models and configurations and inform the broader use of LLM-as-a-judge in interpretability research.

References

Note that the consistency, completeness, and independence scores are generated by prompting an LLM. Although LLM-as-a-judge is a common practice in current literature, the alignment between our evaluation pipeline and true human judgment remains to be validated.

— Base Models Know How to Reason, Thinking Models Learn When  (2510.07364 - Venhoff et al., 8 Oct 2025) in Section 2.2 (Taxonomy Evaluation)

Although the verifier achieves 94\% agreement with human annotations on a random sample of 100 instances and is complemented by deterministic schema and AST checks, this evaluation does not rule out verifier-specific or correlated semantic biases.

— ToolLoop: Closed-Loop Tool-Use Data Synthesis via Decomposed Generation and Dynamic Self-Feedback  (2609.09072 - Zeng et al., 8 Sep 2026) in Section 'Limitations'

A human-agreement study would test the stronger claim, that the panel tracks shopper preference at all, and we leave it as the main open item.

— ZooWork-ShopRanker: An Open, Preference-Aligned E-Commerce Reranker  (2609.31002 - Xue et al., 25 Sep 2026) in Appendix C, Section 'Does the margin depend on the training judges?' (Section label app:circularity)

However, as noted in \S\ref{sec:analysis_precrecf1}, we do not know how many of the novel items would be adjudicated as has if they were adjudicated as instances of disagreement.

— Beyond Majority Vote: Multi-Perspective Adjudication for Medical Hallucination Detection  (2609.03953 - Cecil et al., 3 Sep 2026) in Section Appendix, subsection "Sensitivity of the Constructed Reference" (Section \ref{sec:appendix_sensitivity})

Our error counts are defined relative to human labels collected under a single criterion: the false positives reported for faithfulness-style sources are errors with respect to $y_{\text{fact}$, not in an absolute sense, and comparison against human annotations collected under a reference-faithfulness criterion is left to future work.

— The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation  (2610.08026 - Valjakka et al., 6 Oct 2026) in Section 4.4, “Implications and Limitations”