BiGGen Bench: Granular LLM Evaluation

• BiGGen Bench is a principled evaluation framework that uses instance-specific scoring rubrics to assess LLM performance across nine key capabilities.
• It structures tests with a four-level hierarchy and 77 distinct tasks to provide transparent, actionable diagnostics for various model types.
• Its evaluation protocol combines automated scoring and rigorous statistical analyses to reveal scaling trends and performance gaps among LLMs.

BiGGen Bench is a principled evaluation framework for LLMs that addresses the limitations of existing generation benchmarks by offering comprehensive, fine-grained, and human-aligned assessment. Unlike prevailing benchmarks that rely on generic, abstract criteria and focus narrowly on specific capabilities, BiGGen Bench quantifies the multifaceted performance of LMs using instance-specific scoring rubrics across a broad skills spectrum, facilitating actionable diagnoses and enabling consistent comparison across diverse models (Kim et al., 2024).

1. Motivation and Context

The proliferation of LLMs raises both capability and evaluation challenges, with standard benchmarks (e.g., MT-Bench, AlpacaEval, Chatbot Arena) primarily scoring models on vague qualifiers such as "helpfulness" or "harmlessness." These criteria often conflate distinct skills—such as factual correctness, relevance, or stylistic appropriateness—and offer little interpretability or diagnostic value. Most traditional benchmarks center on instruction following, neglecting areas like planning, tool use, or multilingual competence, thereby introducing coverage bias and masking systematic weaknesses.

BiGGen Bench is introduced in response to these deficits. It operationalizes the granularity and flexibility of human expert evaluation by imposing instance-specific criteria: each task is evaluated using a rubric precisely tailored to its requirements (e.g., logical validity for proofs, idiomaticity for translation), enabling more transparent, interpretable, and actionable measurement of capability.

2. Organizational Structure and Capabilities Assessed

BiGGen Bench is structured as a four-level hierarchy mapping nine high-level capabilities to tasks, instances, and evaluation criteria:

• Capabilities:

1. Instruction Following — open-ended, ambiguous, or creative directives.
2. Grounding — adherence to system contexts (messages, formats, and hierarchy).
3. Planning — generation of multi-step, goal-oriented action sequences.
4. Reasoning — inductive/deductive/abductive reasoning, formal proofs, and legal/hypothesis inference.
5. Refinement — output improvement (self-refinement, peer review, log-based edits).
6. Multilingualism — translation, cultural/idiomatic sensitivity, and nuanced generation in diverse languages.
7. Safety — ethical refusal, moral-dilemma handling, harmful concept unlearning.
8. Theory of Mind — modeling beliefs, intentions, and emotions, next-turn prediction, mind-state representation.
9. Tool Usage — API description parsing, tool invocation/sequencing, web/code generation, and dynamic tool synthesis.

Each capability is realized via 7–10 tasks, totaling 77 distinct tasks and 765 cross-validated instances. Evaluation criteria for each instance are encoded as a five-point Likert rubric, with explicit performance anchors tailored to the nature of the problem—for example, sequence completion accuracy in planning, or critical step correctness in mathematical derivations.

3. Evaluation Protocol and Metrics

BiGGen Bench employs a three-stage protocol:

• Response Generation:

103 models (28 base, 61 chat, 14 proprietary; parameter sizes from 1B to 141B) are evaluated. Base LMs adopt the URIAL prompt (a standardized 3-shot format providing cross-task context), while chat-tuned LMs operate zero-shot.

• Automated Scoring:

Five LM-based evaluators score each response against the instance-specific rubric, assigning Likert scores (1–5) in a direct-assessment regime. Evaluators include GPT-4-1106, GPT-4-Turbo-2024-04-09, Claude-3-Opus, and two Prometheus-2 configurations.

• Aggregation and Metrics:

For each instance $i$, score $S_i\in\{1,2,3,4,5\}$; average within-capability $C_k = \frac{1}{N_k} \sum_{i=1}^{N_k} S_i$; aggregate across capabilities as $B = \sum_{k=1}^9 w_k C_k$, typically with $w_k$ uniform.

Additional metrics: - Scaling Trends: Pearson $r$ and $R^2$ for log-parameter size vs. capability scores. - Evaluator Reliability: Correlation of LM-evaluator scores with human ratings, and inter-evaluator agreement. - Gap Analyses: Statistically controlled comparisons (Welch's $t$-test, Hedges's $g$ effect size, mixed-effect models) for model type (base/chat, open/proprietary) and scale.

4. Key Findings and Diagnostic Insights

• Scaling Laws: Base LMs show linear performance scaling with log-parameter size ($r=0.68$, $S_i\in\{1,2,3,4,5\}$0), with capability-specific slopes (e.g., tool use $S_i\in\{1,2,3,4,5\}$1, refinement $S_i\in\{1,2,3,4,5\}$2). Chat LMs scale more weakly ($S_i\in\{1,2,3,4,5\}$3, $S_i\in\{1,2,3,4,5\}$4), indicating a post-training effect beyond parameter count.
• Model Group Dynamics: The performance gap between base and chat models narrows with size, especially in instruction following, but persists in refinement and reasoning. Proprietary LMs consistently outperform open-source chat LMs, especially in safety ($S_i\in\{1,2,3,4,5\}$5) and multilingual tasks ($S_i\in\{1,2,3,4,5\}$6), with significance at $S_i\in\{1,2,3,4,5\}$7.
• Per-Capability Leaderboards: Phi-3-mini-4K-Instruct leads in reasoning/refinement under 20B parameters, whereas Llama-3-70B-Instruct and Qwen1.5-110B-Chat dominate open-source cohorts, and GPT-4/Claude-3-Opus top proprietary groupings.
• Surprising Behaviors:
  • Phi-3-mini outperforms larger peers in reasoning, suggesting data quality may supersede parameter count for certain skills.
  • Llama-2 chat models excel in safety but underperform in reasoning and tool use, evidencing an "alignment tax."
  • Evaluator LMs diverge from human judgments in Theory of Mind and Tool Usage, indicating the need for domain-adapted judgment protocols.
  • Verbosity bias is negligible: response length explains minimal variance in scores ($S_i\in\{1,2,3,4,5\}$8, $S_i\in\{1,2,3,4,5\}$9).

5. Resources, Reproducibility, and Benchmarking Ecosystem

All BiGGen Bench artifacts are public:

• Dataset and Rubrics: Instance-level items and scoring rubrics available via Hugging Face “prometheus-eval/BiGGen-Bench”.
• Evaluation Results: Full results matrix (103 models, 765 instances, 5 evaluators) at “prometheus-eval/BiGGen-Bench-Results”.
• Codebase: Evaluation scripts, prompts, and utilities at https://github.com/prometheus-eval/prometheus-eval/tree/main/BiGGen-Bench.
• Interactive Reports and Leaderboards: Available via ZenoML and Hugging Face Spaces.

This infrastructure enables both reproduction of canonical results and seamless integration of new LMs into the pipeline.

6. Comparative Context and Broader Significance

BiGGen Bench advances beyond prior generation benchmarks by formalizing coverage across a full spectrum of cognitive, linguistic, and interactive skills. Instance-specific, human-like rubrics circumvent the interpretability and conflation issues of abstract scoring. Compared to frameworks like FLASK and evaluation-by-arena, BiGGen Bench explicitly quantifies performance at resolution compatible with system-level audits and failure-mode discovery.

This suggests BiGGen Bench's approach of rubric-driven, granular, and scalable evaluation is essential for robust measurement as LMs acquire broader, more complex competencies. Its diagnostic power, reproducibility tools, and comprehensive scope have motivated its adoption as a new standard for LM evaluation (Kim et al., 2024).