Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement

Abstract: Evaluating LLM-generated business ideas is often harder to scale than generating them. Unlike standard NLP benchmarks, business idea evaluation relies on multi-dimensional criteria such as feasibility, novelty, differentiation, user need, and market size, and expert judgments often disagree. This paper studies a methodological question raised by such disagreement: should an automatic judge approximate an aggregate consensus, or model evaluators individually? We introduce PBIG-DATA, a dataset of approximately 3,000 individual scores across 300 patent-grounded product ideas, provided by domain experts on six business-oriented dimensions: specificity, technical validity, innovativeness, competitive advantage, need validity, and market size. Analyses show substantial expert disagreement on fine-grained ordinal scores, while agreement is higher under coarse selection, suggesting structured heterogeneity rather than random noise. We then compare three judge configurations: a rubric-only zero-shot judge, an aggregate judge conditioned on mixed evaluator histories, and a personalized judge conditioned on the target evaluator's scoring history. Across dimensions and model sizes, personalized judges align more closely with the corresponding evaluator than aggregate judges, and evaluator agreement correlates with similarity of judge-generated reasoning only under personalized conditioning. These results indicate that pooled labels can be a fragile target in pluralistic evaluation settings and motivate evaluator-conditioned judge designs for business idea assessment.

• The paper demonstrates that personalized judges better align with individual expert evaluations than aggregated judgments.
• It employs PBIG-DATA of 3,000 expert scores across 300 ideas to reveal significant inter-evaluator variation and structured disagreements.
• Results indicate personalized judges outperform aggregate configurations in both score replication and capturing evaluator-specific reasoning.

Comparative Analysis of Aggregate and Personalized Judges in Business Idea Evaluation

Introduction and Motivation

Evaluating LLM-generated business ideas poses unique challenges differing substantially from established NLP benchmarks. Business idea assessment necessitates multidimensional evaluation along axes such as specificity, technical validity, innovativeness, competitive advantage, need validity, and market size. Crucially, such evaluations rely on domain experts whose judgments frequently diverge, even when guided by identical rubrics. The fundamental methodological question addressed is whether automatic judging should target an aggregate consensus or reproduce individual evaluator standards. This paper introduces PBIG-DATA, a substantial dataset of 3,000 expert scores across 300 patent-based product ideas, enabling systematic exploration of expert disagreement and its implications for judge modeling (2604.22517).

Data and Annotation Protocol

PBIG-DATA comprises product ideas generated by diverse LLM ideation systems, each grounded in distinct patents from three technical domains: NLP, CS, and materials chemistry. Each idea is evaluated by domain experts possessing significant technical or business background. The annotation protocol utilizes staged screening, mandating minimum thresholds for upstream dimensions before downstream dimensions are assessed, intentionally resulting in missing scores as a reflection of real screening decisions.

Figure 1: Staged screening protocol for expert scoring.

The rubric operationalizes relevant business frameworks (NABC, Stage-Gate), assigning scales to dimensions according to their practical granularity. This approach and the diversity of evaluators ensure that disagreement patterns are a property of business idea evaluation rather than a generation artifact.

Expert Disagreement Characterization

Substantial inter-evaluator variation in scoring scales is observed across all six dimensions, especially for need validity and market size, where implicit assumptions about user base and adoption context dominate scoring behavior.

Figure 2: Distribution of per-evaluator mean scores for each dimension.

Fine-grained agreement, measured using Krippendorff's $\alpha$, is generally close to zero and sometimes negative, supporting the claim that experts do not employ a shared ordinal scale. However, agreement is significantly higher when scores are converted to coarse selection sets (above-median ideas), indicating that disagreement is structurally driven rather than noise. This pluralistic landscape underpins the central claim: pooled ordinal scores are a fragile target and may misrepresent the diversity of expert standards.

Judge Configurations and Alignment Metrics

Three LLM-as-a-Judge configurations are evaluated:

• Zero-shot: scoring based only on rubric and instructions,
• Aggregate: conditioned on scoring histories from mixed evaluators,
• Personalized: conditioned on the target evaluator's own scoring history.

The experimental setup uses Qwen3 models across four scales, supplemented by results from GPT-5 mini. Prediction agreement with expert annotations is measured via Krippendorff's $\alpha$.

Figure 3: Alignment between automatic judges and expert annotations, measured by Krippendorff's $\alpha$ as a function of few-shot examples.

Across all dimensions, personalized judges achieve higher alignment with their corresponding evaluator than aggregate or zero-shot judges. The gap is more pronounced for larger models, and aggregate conditioning consistently converges toward an averaged standard that fails to capture individual evaluator calibration. Zero-shot judges rarely achieve non-trivial alignment, confirming that rubric-only prompts are insufficient.

Coarse Selection and Justification Behavior

The personalization advantage holds beyond ordinal agreement into coarse selection metrics. Personalized judges outperform aggregate judges in both above-median Jaccard similarity and top-50% overlap across most dimensions, demonstrating that alignment extends to practical decision-making—not just score replication.

Additionally, personalized judges better capture evaluator-specific reasoning patterns. There is a positive correlation ($r=0.31$) between evaluator agreement and cosine similarity of judge-generated reasoning texts only under personalized conditioning.

Figure 4: Relationship between evaluator agreement and similarity of judge-generated reasoning texts, with positive trend under personalized conditioning.

This supports the assertion that personalized conditioning inherits not only score thresholds but also underlying evaluation policies, allowing the judge to mirror qualitative justification behavior associated with each expert.

Implications and Future Directions

The findings demonstrate that aggregate label modeling, common in LLM evaluation protocols, may be fundamentally misaligned with pluralistic expert standards in business ideation contexts. Personalized conditioning produces more faithful replications of expert behavior, both quantitatively and in reasoning structure. Practically, this calls for evaluator-conditioned judge systems to surface structured disagreement rather than enforce artificial consensus.

From a theoretical standpoint, the results underscore the need for evaluation protocols where heterogeneous standards are the norm. Current benchmarks that rely on pooled labels lose important information about stakeholder-specific judgments. For real-world ideation workflows—where investment decisions hinge on assessments from diverse business and technical stakeholders—systems that preserve and expose pluralism will offer stronger decision support.

This research prompts future exploration of calibration mechanisms, multi-perspective scoring, and protocols explicitly designed to support evaluator diversity in automatic assessment. Extensions could focus on interpretability, interplay between judge reasoning and organizational outcomes, and scaling personalized judge constructions within enterprise deployment workflows.

Conclusion

In summary, the paper establishes that LLM-based business idea evaluation is inherently pluralistic: expert scores and justifications neither converge on a single scale nor represent simple annotation noise. Personalized judges, conditioned on individual evaluator history, consistently outperform aggregate configurations in reproducing expert scores and reasoning, particularly for dimensions dominated by implicit standards. The implications span both practical LLM evaluation and methodological protocol design in creative, judgment-driven NLP tasks. Future ideation assessment systems should accommodate evaluator diversity, leveraging personalized conditioning to provide more nuanced support for organizational decision-making.