Temporal & Domain-Adaptive LLM Benchmarks

• The paper presents adaptive LLM benchmarks that measure temporal generalization and domain adaptability through continuous data ingestion and dynamic difficulty modeling.
• It details methodologies like TAM-Bench’s web-agent scraping and BenchHub’s modular pipeline for real-time, diverse, and balanced evaluation across multiple modalities.
• Empirical findings reveal challenges such as temporal bias, domain-dependent performance shifts, and the need for transparent, versioned evaluation protocols.

Temporal and Domain-Adaptive LLM Benchmarks

LLMs have driven forward the capability to automate diverse cognitive and engineering tasks. Benchmarking these models’ abilities, especially their adaptability over time and across evolving domains, is essential for credible evaluation and progress. Temporal and domain-adaptive benchmarks are designed to systematically measure both the shifting performance of LLMs as data and real-world contexts evolve, and their competence across a broad spectrum of subject domains and modalities. This article provides a comprehensive account of the principles, architectures, metrics, and representative benchmarks that enable temporally and domain-adaptive evaluation of LLMs and LLM-based agents.

1. Conceptual Foundations and Motivation

Temporal generalization is the ability of an LLM to maintain robust performance when evaluated on data or events outside its training cutoff—whether drawn from the past, present, or (critically) the future. Formally, given $D_t$ as a distribution of text/events at time $t$ and a model $M$ trained up to $t_0$, high temporal generalization implies that both $$\Delta_{\mathrm{future}} = P(M, D_{t_0}) - P(M, D_{t>t_0})$$ and $$\Delta_{\mathrm{past}} = P(M, D_{t<t_0}) - P(M, D_{t_0})$$ are small (Zhu et al., 2024).

Domain adaptability denotes an LLM’s capability to generalize and maintain performance across heterogeneous knowledge areas, data types, and task modalities, often spanning tabular, text, image, audio, graph, and their combinations (Jia et al., 11 Sep 2025, Kim et al., 31 May 2025).

LLMs suffer from temporal bias, including nostalgia bias (overweighting historical data) and neophilia bias (preferring recent data). Domain-adaptive evaluation is equally crucial, as models often demonstrate domain-dependent ranking reversals (e.g., ranking highly on social or cultural subjects and poorly on scientific or technical domains) (Kim et al., 31 May 2025).

2. Automated, Temporally Adaptive Benchmark Construction

Modern temporally adaptive benchmarks rely on automated, continuous data ingestion pipelines. These systems are engineered to discover, parse, and standardize new benchmark tasks and datasets as they are published, thus maintaining the relevance and freshness of evaluation data.

TAM-Bench implements a web-agent-driven pipeline, with an LLM controller generating web-interaction plans, a custom browser and DOM layer scraping structured data from competition sites, and post-processing with LLM-powered schema mapping (e.g., mapping extracted competition details into a normalized JSON schema with fields like task_type, metric, data_information) (Jia et al., 11 Sep 2025). Temporal refresh is achieved by frequent (e.g., weekly) scraping cycles that filter out data seen during model pretraining (e.g., by only including post-2023 tasks), append new candidates to an evaluation pool, and recalibrate task characteristics such as difficulty.

BenchHub employs a modular architecture: a Data Ingestion Agent (DIA) pipes new datasets through a schema mapper (with rule-based plus LLM-assisted key realignment), metadata extractor (prompt-based assignment of task/answer types), and a fine-tuned classifier (e.g., Qwen-2.5-7B) to label subject, skill, and target attributes. Each sample’s timestamp enables longitudinal repository versioning and time-sliced evaluation (Kim et al., 31 May 2025).

FreshBench isolates temporal effects using strict chronological splits: for each model with cutoff $t_0$, only evaluates on text/events issued after $t_0$ (e.g., BBC news articles, arXiv papers, or real-world prediction questions surfaced after the model’s pretraining window) (Zhu et al., 2024).

3. Domain Taxonomies, Expansion, and Adaptive Balancing

Comprehensive benchmarks implement granular, extensible taxonomies covering both high-level domains and fine-grained subdomains or modalities. BenchHub’s taxonomy, for example, assigns each sample a metadata tuple $m(q)$: task type, answer format, (possibly multi-label) subject, cognitive skill (knowledge, reasoning, alignment), and target language/region. Domain selection for custom evaluation suites is performed by set-theoretic filtering on these attributes—enabling, e.g., construction of “STEM-only,” culture-weighted, or non-English domain evaluations (Kim et al., 31 May 2025).

TAM-Bench enforces balancing heuristics to ensure domain and modality diversity. This includes modality quotas (e.g., Lite subset: 3 tasks per 6 modalities), difficulty-modality coupling (ensuring all modalities are represented across easy, medium, hard tiers), and frequency-aware weighting in evaluation metrics to prevent over-representation by high-frequency domains (Jia et al., 11 Sep 2025).

TemporalBench extends these principles to time-series domains, unifying retail, healthcare, energy, and physical-systems data and supporting cross-domain adaptation and transfer by providing pre-training/fine-tuning splits alongside multi-domain evaluation tasks (Weng et al., 5 Feb 2026).

4. Temporal Difficulty Modeling and Evaluation Dynamics

Adaptive difficulty modeling is required to maintain benchmark rigor as new data and models emerge. TAM-Bench employs leaderboard-driven difficulty estimation using normalized mean and best scores, participant counts, and empirical score ceilings, with parameterized weighting (e.g., $w_1=0.4$, $t$0, $t$1). Tasks are dynamically binned into Easy (≤0.6), Medium (0.6–0.85], and Hard (>0.85) based on up-to-date community performance statistics, with each updated scrape cycle recalibrating scores and generating versioned benchmark snapshots (Jia et al., 11 Sep 2025).

BenchHub enables drift analysis through per-domain, time-windowed accuracy and difference metrics:

• $t$2
• Drift: $t$3

Exponentially decaying time weights (e.g., $t$4) emphasize recency for dynamic reporting (Kim et al., 31 May 2025).

FreshBench formalizes temporal bias via quantifiable indices (TBI) and regression slopes from fine-grained monthly or quarterly bins. Time-weighted accuracy aggregates model performance preferentially over more recent slices, supporting model ranking by recency adaptation (Zhu et al., 2024).

5. Modality and Task-Type Adaptivity in Evaluation

Benchmarks now rigorously control for modality, dataset provenance, and task type in constructing evaluation sets and reporting aggregate metrics.

In multi-modal settings such as TAM-Bench, evaluation spans tabular, text, image, graph, audio, and multi-modal (e.g., vision+language) inputs, with explicit constraints that no single modality is allowed to overwhelmingly dominate evaluation (Jia et al., 11 Sep 2025). Modality-aware weighting in the main metrics (e.g., $t$5) ensures balanced coverage in aggregate performance reporting.

TemporalBench introduces a four-tier taxonomy for time-series reasoning and forecasting: (T1) historical structure interpretation, (T2) context-free forecasting, (T3) contextual temporal reasoning (with explicit ancillary semantic features), and (T4) event-conditioned prediction. Each tier is evaluated with discrete metrics—multiple-choice accuracy for reasoning, MAE/sMAPE for numerical extrapolation, and comparative diagnostics to detect context and event sensitivity (Weng et al., 5 Feb 2026).

VBenchComp (for video LLMs) explicitly partitions benchmark QA pairs into LLM-Answerable (language prior only), Semantic (static frame-shuffled invariant), and Temporal (strictly time-ordered essential), and reports the proportion of each alongside per-category performance. The VBenchComp Score focuses only on semantic and temporal questions, excluding those where language priors yield trivial answers (Feng et al., 20 May 2025).

6. Empirical Findings and Model Behavior Under Drift

Comprehensive evaluations across temporally and domain-adaptive benchmarks reveal systematic trends and open challenges:

• Models exhibit high variance in domain-dependent rankings; e.g., Llama-3.3 ranks first in Culture and Social-Intelligence but sixth in Technology (Kim et al., 31 May 2025).
• Sampling strategy (uniform, stratified by dataset, by subject distribution) significantly alters leaderboard positions, reinforcing the need for balanced and transparent evaluation suite assembly (Kim et al., 31 May 2025).
• Strong static forecasting performance does not guarantee event-aware or contextual temporal reasoning: numerical MAE often fails to correlate with qualitative or event-conditioned reasoning accuracy; many models fail in difference judgment, lag analysis, or context integration tasks (Weng et al., 5 Feb 2026).
• On fresh, post-cutoff data, powerful models demonstrate rapid decay, with ΔAcc and ΔBPC declining –10% to –40% within a year of training cutoff for many closed-source or over-parameterized models (faster for those with highly curated pretraining sets) (Zhu et al., 2024). Open-source models using broader or deduplicated pretraining display better long-term adaptability.
• VBenchComp analysis shows that even SOTA models often rely on language or static cues; temporal-reasoning-proper questions constitute only a minority, and models’ performance drops by up to 30 percentage points on true temporal subsets (Feng et al., 20 May 2025).

7. Benchmark Subsets, Temporal Versioning, and Best Practices

Benchmarks are versioned to support both longitudinal model assessment and real-time adaptability tracking. TAM-Bench provides Lite (18 tasks), Medium (54), and Full (150+) subsets, each balanced across domain/modality/difficulty, with version tags (e.g., Lite-v1.0, Lite-v1.1, etc.) corresponding to specific scrape snapshots and difficulty recalibrations (Jia et al., 11 Sep 2025). Researchers may compare models using “frozen” historical snapshots for stability or dynamically updated sets to probe ongoing generalization.

Best practices recommended across leading benchmarks include:

• Strict chronological splitting of held-out sets by timestamp to avoid data leakage and accurately measure forward generalization (Zhu et al., 2024).
• Ensuring balanced and under-represented domain coverage, often by modality quotas, targeted niche injection, and frequency-aware metric weighting (Jia et al., 11 Sep 2025, Kim et al., 31 May 2025).
• Transparent reporting of per-domain, per-task-type, and per-time-window performance, and public leaderboards for drift tracking (Weng et al., 5 Feb 2026).
• Automated pre-profiling for apparent leakage and domain imbalance, e.g., via VBenchComp’s language-prior and shuffling-invariance measures (Feng et al., 20 May 2025).

Table: Summary of Key Benchmarks and Design Features

Benchmark	Temporal Adaptivity	Domain/Modality Adaptivity
TAM-Bench (Jia et al., 11 Sep 2025)	Continuous scraping, difficulty recalibration, versioned subsets	6 modalities (tabular/text/image/graph/audio/multi-modal); balanced task pools; domain tagging
BenchHub (Kim et al., 31 May 2025)	Incremental dataset pipeline, time-stamped samples, drift metrics	Coarse/fine-grained subject taxonomy; custom filters for domain-specific evaluation
TemporalBench (Weng et al., 5 Feb 2026)	Event/context-aware, time-sliced multi-tier evaluation; cross-domain leaderboards	Retail, healthcare, energy, physical systems; supports domain transfer and multi-modal reasoning
FreshBench (Zhu et al., 2024)	Strict post-cutoff held-out sets, decay/drift tracking	Supports multi-domain corpora and event types (arXiv, BBC, Wikipedia, forecasting Qs)
VBenchComp (Feng et al., 20 May 2025)	Automated pipeline for temporal, semantic, and text-prior separation	Video; explicit categorization for true temporal reasoning evaluation

• (Jia et al., 11 Sep 2025) Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, and Metric Optimization.
• (Kim et al., 31 May 2025) BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM Evaluation.
• (Feng et al., 20 May 2025) Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?
• (Weng et al., 5 Feb 2026) TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Tasks.
• (Zhu et al., 2024) Is Your LLM Outdated? A Deep Look at Temporal Generalization.