GiantsBench: LM Benchmark & Stellar Calibration

• GiantsBench is a dual-purpose resource offering a benchmark for language models to anticipate core scientific insights and a photometric calibration recipe for red giants.
• The insight anticipation benchmark uses summaries of two parent papers to generate concise core insights, demonstrating robust cross-domain performance and high human-validated similarity scores.
• The stellar calibration employs SDSS photometry and metallicity from Galactic clusters to estimate red giants’ absolute magnitudes with quantified residuals and uncertainty metrics.

GiantsBench refers to two distinct, field-specific resources: (1) a comprehensive benchmark for evaluating LLMs' capacity for insight anticipation from scientific literature (He-Yueya et al., 10 Apr 2026), and (2) a photometric calibration recipe for estimating red giants' absolute magnitudes in the SDSS system (Karaali et al., 2012). Both resources are characterized by formal task definitions, carefully validated methodologies, and broad utility across their respective research domains.

1. Formal Definition and Scope

1.1. Insight Anticipation Benchmark

GiantsBench is a large-scale benchmark designed to evaluate whether a LLM (LM) can “anticipate” the core insight of a future scientific paper, given textual summaries of its two most synergistic parent papers (He-Yueya et al., 10 Apr 2026). The formal setting is as follows: Let $A$ and $B$ be parent papers summarized by $x_A$, $x_B$. For a downstream paper $C$ that cites $A$ and $B$, $y^*$ denotes a concise description of $C$'s core insight. The insight anticipation task is:

Given input $x = (x_A, x_B)$, learn a conditional distribution $B$0 such that generated insight $B$1 maximizes semantic similarity to $B$2.

1.2. Stellar Calibration Recipe

Independently, GiantsBench denotes a self-contained recipe for estimating the absolute $B$3-band magnitude ($B$4) of red-giant stars from SDSS photometry and metallicity (Karaali et al., 2012). For a dereddened color $B$5 and metallicity $B$6, absolute magnitude is parameterized as:

$B$7

Coefficients $B$8 are tabulated as a function of $B$9; typical application involves interpolation in color (step $x_A$0 mag).

2. Dataset Construction and Properties

2.1. Scientific Literature Benchmark

GiantsBench comprises 17,839 examples spanning eight core scientific macro-domains:

Domain	Notes
Computer Science (CS)	Includes subfields; split only in training
Economics
Electrical Engineering
Mathematics
Physics
Quantitative Biology
Quantitative Finance
Statistics

The construction pipeline involves:

1. Selecting all arXiv papers (2007–2026) with ≥2 citations.
2. Using a high-capacity LM (gemini-2.5-flash) to identify for each paper the two most synergistically relevant parents and a synergy explanation.
3. Producing concise summaries for both parents.
4. Rewriting the synergy explanation as an atomic target insight using another LM (gemini-3-pro).
5. Keeping the most cited downstream paper for shared parent pairs.
6. Splitting into training (pre-2023-07-01; domain-restricted) and test (post-cutoff, cross-domain; 7,504 examples) with an “unseen-parents” subset (N = 5,294) that minimizes overlap.

2.2. Stellar Calibration Sample

The GiantsBench calibration (Karaali et al., 2012) is derived from six Galactic clusters spanning $x_A$1 dex:

Cluster	$x_A$2
M92	–2.15
M13	–1.41
M3	–1.50
M71	–0.78
NGC 2158	–0.25
NGC 6791	+0.37

Photometry is in the SDSS $x_A$3 bands. Reddening corrections and ridge line fitting (4th–5th order polynomials) yield the calibration polynomials at each $x_A$4. Application is only valid for stars older than 2 Gyr.

3. Data Format and Encoding Strategies

3.1. Benchmark Input/Output (NLP)

Each GiantsBench instance consists of:

• Input: Concatenated LM-generated summaries $x_A$5, $x_A$6 ($x_A$71,000–2,000 tokens each).
• Target: Reference insight $x_A$8 (one or two sentences).
• Processing: Inputs are fed directly into an autoregressive LM; sequences are truncated or padded to fit model context windows (typically 4,096–8,192 tokens).

3.2. Photometric Calibration (Astrophysics)

Inputs for GiantsBench stellar application include:

• SDSS $x_A$9, $x_B$0 apparent magnitudes.
• Line-of-sight color excess $x_B$1.
• Metallicity $x_B$2.
• Age estimate ($x_B$3 required for applicability).

Outputs are computed via:

1. Reddening correction:

$x_B$4

$x_B$5

1. Interpolation of coefficients for $x_B$6 from GiantsBench tables.
2. Evaluation of the calibration polynomial.

4. Evaluation and Validation

4.1. LLM Benchmarking

For insight anticipation, GiantsBench employs a black-box “judge” function $x_B$7, implemented as a gemini-3-pro prompt rating semantic similarity. Human studies validate the LM-judge as highly correlated with expert assessment (Spearman’s $x_B$8, $x_B$9).

The GIANTS-4B model is fine-tuned via Group Relative Policy Optimization (GRPO), sampling 8 candidates per iteration, scoring with a reward model (gemini-2.5-flash), and enforcing KL-regularization. Evaluation always uses a distinct judge LM (gemini-3-pro) to avoid overfitting and reward hacking.

Zero-shot evaluation on held-out domains and “unseen-parents” test set demonstrates robust cross-domain generalization and resistance to memorization effects.

4.2. Photometric Fit Statistics

For test clusters, calibration residuals demonstrate:

• Range: $C$0 mag.
• 94% quantile: $C$1 mag.
• Mean residual: $C$2 mag; scatter: $C$3 mag.

Total uncertainty in practical application combines photometric errors (typically $C$4 mag), metallicity uncertainties ($C$5 dex induces $C$6 mag), and intrinsic calibration scatter.

5. Comparative Performance and Use Cases

5.1. Insight Anticipation

GIANTS-4B achieves a 35% higher average similarity score than the strongest proprietary baseline (gemini-3-pro) on the full test set, with a 34% advantage on the stringent “unseen-parents” subset (He-Yueya et al., 10 Apr 2026). Human raters and the SciJudge-30B model, which predicts citation impact, consistently prefer GIANTS-4B outputs (89.7% and 68% win rates, respectively). Conceptual clarity is measurably superior to baseline models.

Performance scales positively with best-of-$C$7 inference: the advantage over baselines increases up to 40% for $C$8 candidate generations.

5.2. Stellar Photometry

The GiantsBench calibration provides reliable absolute magnitude estimates for red-giant stars in Galactic populations, supporting population studies and distance estimation efforts. Its high $C$9 fits and well-characterized residuals facilitate robust error budgeting. Application to red-clump stars or to giants younger than 2 Gyr is not supported due to systematic deviation from the calibration locus.

6. Limitations and Caveats

• The language-modeling benchmark is restricted to synthesis over pairs of parent papers; generalization to larger parent sets or alternative insight types is untested.
• The reward model and evaluation are LM-based, not human-in-the-loop, though strong correlation with expert annotators is demonstrated.
• For the stellar calibration, extrapolation outside the specified color ($A$0), metallicity ($A$1), or to stars $A$2 Gyr old may yield substantial errors.
• Red-clump interlopers, alpha-enhanced populations, and non-SDSS photometric systems require dedicated treatment or transformation before applying the recipe.

7. Research Impact and Future Directions

GiantsBench, in both its language modeling and astronomical calibration contexts, provides high-precision, large-scale resources for methodological benchmarking and practical application. For automated scientific discovery, the release of GiantsBench (17k examples, code, and evaluation scripts) is intended to stimulate advances in literature-grounded synthesis, improved retrieval-generation pipelines, RL-tuned reward functions with greater novelty emphasis, and integrated frameworks coupling parent identification with insight generation. In astrophysics, calibration tables enable precise, reproducible red-giant magnitude estimation for broad population analyses, but further work is needed for younger populations, high-alpha stars, and novel photometric systems.

---

References:

• "GIANTS: Generative Insight Anticipation from Scientific Literature" (He-Yueya et al., 10 Apr 2026)
• "Absolute Magnitude Calibration for Giants based on the Colour-Magnitude Diagrams of Galactic Clusters. II-Calibration with SDSS" (Karaali et al., 2012)