OpenDataArena: Benchmarking LLM Datasets

• OpenDataArena (ODA) is an open, extensible platform that rigorously benchmarks post-training datasets for LLMs through automated, reproducible evaluation and transparent lineage tracing.
• It features a four-layer architecture—data input, evaluation, analysis, and visualization—that standardizes fine-tuning experiments and enables precise cross-dataset comparisons.
• ODA’s multi-dimensional scoring system quantifies dataset efficiency, complexity, and contamination risks, guiding principled data curation for enhanced LLM performance.

OpenDataArena (ODA) is an open, extensible platform designed to rigorously benchmark the value of post-training datasets used in the development of LLMs. Addressing the prevailing opacity surrounding data provenance, composition, and evaluation, ODA introduces an automated, reproducible ecosystem for dataset evaluation, scoring, and lineage tracing. By making both methodology and results public, ODA aims to transition the field from ad hoc, trial-and-error data curation practices to systematic, data-centric scientific inquiry (Cai et al., 16 Dec 2025).

1. Platform Architecture and Core Pillars

ODA is architected around four sequential operational layers that together form the backbone for dataset benchmarking:

1. Data Input Layer: Responsible for intake of user-submitted Supervised Fine-Tuning (SFT) datasets. Input instances are normalized to a uniform instruction–response schema. All datasets are categorically tagged by domain—General, Math, Code, Science, Reasoning.
2. Data Evaluation Layer: For each dataset $D_i$, the platform fine-tunes one or more fixed base models (e.g., Llama3.1-8B, Qwen2.5-7B, Qwen3-8B) using standardized hyperparameters via LLaMA-Factory. Model checkpoints are uniformly evaluated on 22 downstream benchmarks with OpenCompass and additional task-specific harnesses.
3. Data Analysis Layer: Aggregates model performance metrics (e.g., accuracy, pass@1), data scores, and computes cross-dataset/model comparisons. Analyses include data efficiency, feature correlation, and dataset lineage tracing.
4. Data Visualization Layer: Publishes interactive leaderboards, detailed trend charts, and dataset lineage graphs to the ODA web portal.

ODA guarantees that base models and all hyperparameters are held fixed, ensuring that the only independent variable in any pipeline run is the training dataset itself—a strict control for "apple-to-apple" comparisons across experiments. The orchestration pipeline supports scalability beyond 600 independent fine-tuning and over 10,000 evaluation runs (Cai et al., 16 Dec 2025).

Layer	Function	Key Technology
Data Input	Normalize, tag datasets	Python/YAML CLI
Data Evaluation	SFT & benchmarking on fixed models	LLaMA-Factory, OpenCompass
Data Analysis	Metrics aggregation, correlation, lineage	Custom scoring modules
Data Visualization	Leaderboards, trend charts, lineage graphs	Web frontend

2. Multi-Dimensional Dataset Scoring Framework

ODA implements a comprehensive scoring system, profiling each dataset along approximately 15–20 axes, grouped into three methodological classes:

• Model-Based Evaluation: Quantified using learned predictors and automated verifiers. Metrics include:
  • Deita Complexity (Q): Predictor of prompt difficulty.
  • Thinking Probability (Q): Estimated likelihood that the prompt requires multi-step reasoning.
  • Deita Quality (QA): Automated reward model score for response helpfulness and correctness.
  • Instruction Following Difficulty (QA): Model-based measure of challenge in fulfilling all prompt constraints.
  • Fail Rate (QA): Fraction of responses flagged as incorrect.
• LLM-as-Judge Scoring: LLMs like GPT-4 rate both prompt and prompt-response pairs for Difficulty, Relevance, Clarity, Coherence, Completeness, Complexity, Correctness, and Meaningfulness.
• Heuristic Evaluation: Includes length-based metrics such as Response Length (QA), computed as token count.

A formal example is the data-efficiency metric: $$DE_{i,M} = \frac{S_{i,M}^{\mathrm{SFT}} - S_{M}^{\mathrm{Base}}}{|D_i|}$$ where $S_{i,M}^{\mathrm{SFT}}$ is the benchmark score of model $M$ post-SFT on $D_i$, $S_{M}^{\mathrm{Base}}$ is the base model's score, and $|D_i|$ is dataset size. This metric, along with others, enables precise dataset comparisons across scale and domain (Cai et al., 16 Dec 2025).

3. Interactive Data Lineage Explorer and Contamination Detection

ODA's lineage explorer models dataset genealogy as a directed graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where each node $v_i$ represents a dataset, and $(v_i,v_j)\in\mathcal{E}$ denotes derivation (by fusion, distillation, or reformulation) of $D_j$ from $D_i$. The pipeline uses multi-agent scripts to extract sources from metadata (READMEs, papers, blogs), canonicalize names, and build bottom-up lineages with confidence scoring.

Users can trace dataset ancestry up to 5–10 levels deep, identify "super-aggregators" that fuse hundreds of upstream datasets, and highlight contamination scenarios where validation/evaluation benchmarks inadvertently appear in training data (e.g., Omni-MATH in Big-Math-RL-Verified) (Cai et al., 16 Dec 2025).

Function	Mechanism	Example Outcome
Ancestry tracing	Multi-level graph traversal	Up to 11 hops for Math
Super-aggregator ID	Node-degree, cross-domain fusion analysis	AM-Thinking cites 435 nodes
Contamination alerts	Detection of evaluation sets in training sources	Omni-MATH leak in Big-Math-RL-Verified

4. Experimental Setup and Benchmark Suite

The ODA study spans over 120 post-training SFT corpora, exceeding 40 million examples, across General, Math, Code, Science, and cross-domain mixes. Experiments are conducted with at least three fixed base models (Llama3.1-8B, Qwen2.5-7B, Qwen3-8B). Evaluation is standardized across 22 benchmarks:

• General: DROP, IFEval, AGIEval, MMLU-PRO (0–5-shot).
• Math: Omni-MATH, OlympiadBenchMath, GSM8K, MATH-500, AIME_2024/25, HMMT_Feb_2025, BRUMO_2025, CMIMC_2025.
• Code: HumanEval, HumanEval+, MBPP, LiveCodeBench (v5).
• Reasoning: ARC_c, BBH, KOR-Bench, CaLM, GPQA.

Metrics used include accuracy, average sub-task accuracy, pass@1, and global composite scores, over a total scale of more than 600 SFT runs and 10,000 evaluations, processing in excess of 40 million samples (Cai et al., 16 Dec 2025).

5. Empirical Insights and Analysis

Key findings from extensive ODA analysis include:

• Data Complexity vs Performance: Response length (QA_A_Length) shows the highest correlation with performance gains (Spearman $\rho\approx0.81$ in Math), whereas prompt complexity alone (Q_Complexity) is non-indicative ($\rho\approx0$).
• Data Efficiency Ceiling: Highly curated small sets (LIMA, LIMO) are efficient but reach a performance plateau or regress on weaker base models. Large, diverse aggregators (AM-Thinking) yield both robust gains and high final accuracy despite moderate data efficiency.
• Lineage Redundancy: The global dataset genealogy graph (seeded from 70 datasets) expands to 411 nodes and 941 edges (~2.29 edges/node), highlighting systemic reuse and redundancy. Benchmark contamination is common—benchmarks such as Omni-MATH and LiveCodeBench are frequently embedded in training corpora.
• Genealogical Structures: Math datasets display the deepest lineages (average depth 5.18, max 11), with sources like EleutherAI/hendrycks_math and openai/gsm8k reused 16 and 13 times, respectively. Code datasets center on programming contest sources, and science datasets often repurpose mathematical corpora (average lineage depth 3.71).
• Data Mixing Laws: Emergent principles indicate that "moderate efficiency plus sufficient volume" consistently outperforms extreme efficiency alone—diversified, large-volume sets stabilize learning for weaker base models. Cross-domain "super-aggregators" (e.g., AM-Thinking-v1-Distilled, which cites 435 nodes) consistently lead leaderboard performance. Domain-specific heuristics also emerge— in code, shorter responses are positively correlated with outcomes (the QA_A_Length correlation is negative).

The platform applies Spearman rank correlation

$\rho = 1 - \frac{6\sum d_k^2}{n(n^2-1)}$

for cross-model comparisons, supporting the stability and generality of these empirical insights (Cai et al., 16 Dec 2025).

6. Implications for Data-Centric AI and Future Research

ODA fundamentally reconfigures the focus of LLM research from model-centric scaling to principled, transparent data selection and evaluation. By open-sourcing its entire toolkit—including training, evaluation, scoring modules, and all lineage graphs—ODA provides the infrastructure for reproducible, fair, and quantitative dataset comparisons.

This open architecture enables diagnostics to guide synthetic corpus construction—defining optimal lengths, complexity, and domain mixes for new datasets. The lineage tracing mechanisms expose contamination risks, protecting the integrity of evaluation benchmarks. The comprehensive scoring and mixing analyses lay the groundwork for formal "data mixing laws" analogous to model scaling laws, thereby informing strategic construction of foundation model training corpora.

The ODA paradigm elevates data from a black-box asset to a first-class scientific object, democratizing access to high-quality data evaluation and catalyzing rigorous research into the empirical laws governing LLM efficacy (Cai et al., 16 Dec 2025).