---
title: STELLA Benchmark Overview
url: https://www.emergentmind.com/topics/stella-benchmark
type: topic
---

# STELLA Benchmark Overview

STELLA Benchmark

The term "STELLA Benchmark" refers to several distinct, rigorously constructed benchmark datasets, evaluation protocols, and code frameworks across domains such as information retrieval, astrophysical radiative transfer, stencil-based PDE solving, gyrokinetics, protein function prediction, and time series forecasting. Each instantiation of STELLA provides a rigorously defined methodology, publicly available corpus, and robust evaluation metrics designed to facilitate direct, reproducible comparison among models and algorithms.

## 1. STELLA for Living-Lab Academic Search Evaluation

STELLA, as described in [2210.13202], is a living-lab "wrapper" for online evaluation of academic search and recommendation systems. It enables direct integration of Dockerized, RESTful microservices as experimental retrieval or recommendation engines alongside production systems. The core architecture comprises:

- **Experimental Micro-services**: Each participant submits a container exposing REST endpoints for ranking (/rank?query=q) and recommendation (/recommend?docid=d), operating on the same fully disclosed corpus as the production system.
- **Multi-Container Application (MCA)**: At runtime, the MCA orchestrates container execution, merges baseline and experimental results via Team-Draft Interleaving, and ensures fully comparable, randomized A/B testing without session disruption.
- **Central Server & Analytics Dashboard**: Manages registration, logging of all requests and interactions, and provides real-time analytics (win/loss/tie, click distributions, session satisfaction, reward metrics) with fine-grained drill-down.

Experiments define a session as $S = \langle q_1, …, q_n;\ \textrm{clicks};\ \textrm{dwell times}\rangle$ and implement formal interleaving of ranked lists. Evaluation is based on:

- **Click Attribution & Win/Loss/Tie**: Each click is attributed to baseline or treatment via interleaving team assignment, yielding per-session outcome margins, $\Delta = C_B - C_A$.
- **Weighted Reward**: Differential SERP-element weighting, $R(S) = \sum_{\text{clicks } j \in S} w_{\text{element}(j)}$, is aggregated to yield system-level mean reward.
- **Key Metrics**:
    - Click-Through Rate: $\text{CTR}_X = \frac{\text{Total Clicks}_X}{\text{Total Impressions}_X}$
    - Mean Dwell Time: $\text{MDT}_X = \frac{1}{\text{Clicks}_X} \sum_{c} \text{dwell}_c$
    - Session Satisfaction: $\text{SessionSat}_X = \frac{1}{|\text{Sessions}_X|}\sum_S \text{SS}_S$, where $\text{SS}_S$ is an indicator for clicks
    - Reward and Interleaving-based statistics as above

Logging infrastructure records all HTTP events, interactions, and click metadata at scale, allowing for high-resolution, multi-corpus analytic slicing. The initial CLEF LiLAS 2021 evaluation involved nine systems, demonstrating that production academic search is highly optimized (no significant click-rate win by experimental systems), but the reward metric exposed systems yielding longer dwell times through more appealing result presentations. The design enforces strict reproducibility: containers and code can be reused directly by other labs for subsequent evaluation [2210.13202].

## 2. STELLA in the Aerospace Domain: Terminology-Aware IR Benchmark

The STELLA framework for aerospace IR, introduced in [2601.03496], addresses the lack of a public benchmark capturing terminology-driven query intent on aerospace documentation:

- **Data Construction Pipeline**:
    1. Document layout detection/extraction from NASA NTRS PDFs using DocLayout-YOLO.
    2. Recursive-Token-Chunking of detected text into 2.4 million 100-token passages.
    3. Terminology Dictionary extraction via structured regex, POS-tag and specificity filtering, yielding ~12,000 distinct aerospace terms.
    4. Synthetic Query Generation: Filtering passages with ≥5 terms, intent classification (Def, Num, Proc, Comp, Anom) via LLM, representative clustering, and dual-type query generation:
       - **Terminology Concordant Query (TCQ)**: Embeds technical term(s) verbatim for lexical-matching evaluation.
       - **Terminology Agnostic Query (TAQ)**: Substitute descriptions for terms for semantic-matching evaluation.
    5. Cross-lingual extension: Hybrid translation (preserve terms, translate context) generates 7×1,000 queries (languages: En, Ko, Id, Th, Fr, Zh, Ja) in BEIR format.

- **Formal Task Definition**:
    - $Q_\mathrm{TC}(P)$ targets lexical matching (BM25 relevance).
    - $Q_\mathrm{TA}(P)$ targets semantic matching (encoder cosine similarity).
    - Dual scoring functions:
        - $S_\mathrm{lex}(Q, P)$: BM25
        - $S_\mathrm{sem}(Q, P)$: embedding cosine

- **Query Generation**:
    - Chain-of-Density (CoD) yields progressively denser queries within passage context.
    - Self-Reflection: LLM-generated critiques enforce answerability, intent, and no external leakage at each generation stage.

- **Retrieval Metrics**:
    - Precision@k, Recall@k, MAP, nDCG@10 over defined query-passage pairs.

- **Empirical Findings**:
    - Decoder-only embedding models (Llama-Embed-Nemotron) minimize the gap between lexical and semantic matching, outperforming bi-encoders and BM25 on TAQ.
    - BM25 remains dominant for direct terminology matching (TCQ), but fails for TAQ.
    - Hybrid translation in cross-lingual settings preserves retrieval performance, justifying real-user query emulation [2601.03496].

Reproducible code, dataset, and pipeline documentation are provided for complete transparency and extensibility.

## 3. STELLA in Astrophysical Radiative Transfer

STELLA [2009.01566] is a time-dependent, multigroup radiation hydrodynamics code extensively benchmarked for SN Ia, SN II-peculiar, and SN II-P LCs, with particular focus on the role of the thermalisation parameter $\epsilon$ (fraction of line opacity treated as absorption):

- **Governing Equation**: Time-dependent RT equation with source function $S_\nu = (1-\epsilon)J_\nu + \epsilon B_\nu$, where $\epsilon = \kappa_\nu/(\kappa_\nu + \sigma_\nu)$ is imposed globally for all bound-bound transitions.
- **Benchmarks**:
    - Grid-scan in $\epsilon$ (1, 0.9, 0.8, 0.5, 0.1, 0) and comparison to ARTIS and observed LCs.
    - U, B sensitivities: blue-band LCs are excessively broadened for $\epsilon < 0.8$ in all SN types.
    - Best agreement in B/V (Type Ia), U/B (II-P/II-pec) consistently occurs for $0.8 \leq \epsilon \leq 0.9$.
    - Strongly supports $\epsilon \approx 0.9$ as default: mimics partial fluorescence without excessive blue trapping.

Correlation, least-squared, and linear regression analyses confirm this setting across nearly all bands and SN types, providing a prescription for future STELLA-based LC modeling [2009.01566].

## 4. STELLA in Stencil-Based PDE and Parareal Frameworks

In [1409.8563], STELLA (STEncil Loop LAnguage) is a C++ DSEL for high-performance stencil computations, explicitly benchmarked in the context of the Parareal time-parallelization scheme:

- **Problem**: 3D advection-diffusion with time-dependent $\nu(t)$.
- **Implementation**: Coarse (upwind+Euler) and fine (RK4+4th order FD) solvers as STELLA stencils; time parallelism by MPI, spatial within-node via OpenMP/CUDA.
- **Quantitative Results**:
    - Parareal converges in $K=3$ iterations (defect below fine-mesh error).
    - Speedup closely follows theoretical bound to $N_p=128$ time slices, with parallel efficiency $E_{meas}\approx$ 20—32%.
    - Energy-to-solution overhead matches $1/E$ bound; communication costs negligible to this scale.

This validates the Parareal-STELLA approach for large-scale PDEs, yielding energy-efficient, scalable time-parallel solutions [1409.8563].

## 5. STELLA as a Gyrokinetic Benchmark Suite

STELLA [1806.02162, 2107.06060, 2201.01506] supplies comprehensive benchmarks for collisional and collisionless $\delta f$ gyrokinetics in both axisymmetric and 3D stellarator geometry:

- **Physics**: Mixed implicit–explicit time stepping, VMEC/Miller geometry support, flux-tube and global approaches.
- **Benchmarks**:
    - Agreement with GS2 (axisymmetric) and GENE (stellarator, W7-X; see [2107.06060]) to within a few percent in linear growth rates, frequencies, eigenfunctions, and nonlinear saturated flux.
    - New flux-tube–based global scheme [2201.01506] reproduces radial profiles and fluxes accurately, eliminating Dirichlet boundary spectral artifacts and restoring quantitative agreement with local theory.
- **Parallelization**: Strong scaling up to $10^4$ cores is feasible with negligible communication overhead in explicit/semi-Lagrangian steps, and efficient Green's function solves for implicit parallel terms.

STELLA in this context sets a de facto standard for stellarator/tokamak code validation [1806.02162, 2107.06060, 2201.01506].

## 6. STELLA for Multimodal Protein Function Prediction

STELLA [2506.03800] represents a multimodal LLM framework, integrating sequence–structure embeddings (ESM3) with a Llama-3.1 backbone for protein function prediction via the OPI-Struc instruction dataset:

- **Model**: Frozen ESM3 encoder; trainable linear connector; Llama-3.1-8B-Instruct LLM; two-stage MMIT for alignment and instruction following.
- **Tasks**: Functional description generation (free-text and multiple-choice) and enzyme-catalyzed reaction (EC label) prediction.
- **Evaluation**:
    - BLEU-4, BERTScore, ROUGE, accuracy.
    - STELLA advances state-of-the-art by +3 BLEU-4 and +2 ROUGE-L over prior models, with top enzyme-naming accuracy (88.85%).
- **Ablations**: Show ESM3+Llama-3.1 yields optimal feature clustering and free-text generation; two-stage fine-tuning and dialogue augmentation further increase robustness.

This benchmark establishes the paradigm of LLM-augmented multimodal models for bioinformatics [2506.03800].

## 7. STELLA for Time Series Forecasting with Language Models

STELLA [2512.04871] introduces semantic–temporal alignment for time series forecasting by dynamically abstracting trend, seasonality, and residual, then mapping these via hierarchical semantic anchors into a frozen LLM context:

- **Architecture**: Neural STL decomposition, patch embeddings, CSP (Corpus-level) and FBP (Instance-specific) prompts as cross-attention prefix tokens for the transformer.
- **Datasets**: Eight multivariate time-series datasets (ETTm1, ETTm2, ETTh1, ETTh2, Weather, Exchange, Illness, M4).
- **Metrics**: MSE, MAE, SMAPE, MASE, OWA.
- **Performance**:
    - STELLA surpasses prior art in both zero-shot and few-shot transfer across all benchmarks (e.g., ETTm2 few-shot: 0.291 MSE vs. 0.303 for TiDE).
    - Ablations demonstrate explicit decomposition (N-STL) as critical; both CSP and FBP non-redundantly improve accuracy by 1–3%.

STELLA thereby systematizes prompt-driven LLM forecasting from explicit component analysis, merging statistical and language model paradigms [2512.04871].

---

**In summary, "STELLA Benchmark" denotes a set of state-of-the-art, domain-specific benchmark frameworks, each exemplifying formalized protocols, public reproducibility, and fine-grained analysis for robust, transparent evaluation across IR, astrophysics, computational physics, molecular biology, and ML-based forecasting.**

Source: https://www.emergentmind.com/topics/stella-benchmark