Synthetic Diagnostic Splits

• Synthetic diagnostic splits are systematically designed data partitions based on criteria like model confidence, task complexity, and ambiguity.
• They enable fine-grained evaluation by exposing targeted model behaviors and failure modes that remain hidden in random data splits.
• These splits support empirical validation of reasoning, calibration, and interpretability in domains such as QA, temporal graph analysis, and medical diagnostics.

A synthetic diagnostic split is a systematically designed division of synthetically generated data into subsets that enable fine-grained evaluation of models along specific axes of task difficulty, ambiguity, information completeness, or other diagnostically relevant properties. These splits are not arbitrary; they are constructed to expose targeted behaviors or limitations of models that would remain hidden in conventional, randomly partitioned datasets. Synthetic diagnostic splits are integral to the creation of interpretable, low-noise benchmarks for cognitive or reasoning abilities in contemporary machine learning, particularly where real-world data are insufficiently controlled or lack ground-truth interpretability.

1. Formal Principles of Synthetic Diagnostic Splitting

A synthetic diagnostic split partitions a controlled synthetic dataset according to axes tailored to the diagnostic requirements of the evaluation. The split criteria arise from intrinsic attributes of the synthetic instances, such as:

• Training dynamics metrics (confidence, variability) tracked across checkpoints of model learning (Shi et al., 2023).
• Graph-theoretic parameters (periodicity, causal lag, path complexity) governing task difficulty and memory requirements (Dizaji et al., 14 Jul 2025).
• Information content and diagnostic ambiguity, measured against an expert-encoded domain ontology or knowledge graph (Song et al., 13 Feb 2026).

Unlike natural splits (e.g., time-based, i.i.d. random), diagnostic splits are mathematically and procedurally derived from features of the generative process, the model's evolving uncertainty, or logical structure.

2. Methodologies for Constructing Synthetic Diagnostic Splits

Procedures for synthetic diagnostic splitting are domain- and benchmark-specific, but share common steps:

A. Training Dynamics-Based Splits

In the context of synthetic commonsense question answering, splits are derived from model-specific metrics: for each synthetic QA item, masked LLM (MLM) scoring yields a pair of training dynamics signatures, mean confidence ($\mu$) and variability ($\sigma$), computed over multiple checkpoints. The set is partitioned into:

• Easy-to-learn: Top $\alpha\%$ by $\mu$
• Ambiguous: Top $\beta\%$ by $\sigma$
• Hard-to-learn: Bottom $\alpha\%$ by $\mu$

Post-selection filtering (removal of mislabeled items, false-negatives, easy distractors) can be applied to enhance diagnostic purity. Typical settings adopt $\alpha = \beta = 33\%$, ensuring each slice is balanced (Shi et al., 2023).

B. Controlled Attribute Splits in Temporal/Relational Data

Synthetic diagnostic splits in temporal graph learning are driven by task-generative parameters (period, memory lag, path length):

• Periodicity splits: Each period, defined by $k$ unique graphs and block length $\sigma$0, is kept intact. Splits maintain temporal coherence: Train = $\sigma$1 steps; Val = $\sigma$2; Test = $\sigma$3.
• Causal/long-range splits: Proportional slicing of the full timeline by fixed fractions ($\sigma$4 train, $\sigma$5 val, rest test), maintaining chronological order (Dizaji et al., 14 Jul 2025).

C. Ontology-Based Information Completeness and Complexity Splits

In domains such as psychiatric diagnosis, splits are aligned with axes reflecting clinical reasoning:

• Information completeness $\sigma$6: Fraction of required diagnostic criteria expressed in a synthetic case, regulated at sampling time ($\sigma$7 for complete, $\sigma$8 for partial).
• Diagnostic complexity $\sigma$9: Number of disorders consistent with available evidence ($\alpha\%$0 for unambiguous, $\alpha\%$1 for ambiguous differential, $\alpha\%$2 for resolved cases).

Each split type yields distinct case pools (e.g., "Medical Chart" vs. "Patient Self-Report" vs. ambiguous vs. resolved differential diagnosis), enabling multi-dimensional probing (Song et al., 13 Feb 2026).

3. Canonical Examples Across Domains

Benchmark	Split Axis/Criteria	Diagnostic Dimensions
QaDynamics (Shi et al., 2023)	Model confidence (μ), variability (σ)	Hardness, ambiguity
T-GRAB (Dizaji et al., 14 Jul 2025)	Period (k, n), memory lag (ℓ), path (d)	Temporal & spatial difficulty
MentalBench (Song et al., 13 Feb 2026)	Completeness (C), ambiguity (D)	Diagnostic uncertainty, info

• QaDynamics: "Hard-to-learn" slices, filtered by diagnostic cleaning, outperform full-set fine-tuning and provide interpretable model diagnostics.
• T-GRAB: Difficulty parameters controllably increase required memory or reasoning, and splits guarantee task units are never fractured across subsets.
• MentalBench: Four main synthetic splits reflect both completeness and complexity, mapping to accuracy and calibration struggles of LLMs.

4. Implications for Model Assessment and Analysis

Synthetic diagnostic splits are explicitly engineered to provide:

• Low-noise and interpretable evaluations: Each split contains instances with known, controllable properties, enabling fine-grained attribution of model strengths and weaknesses.
• Diagnosis of reasoning or calibration failures: Hard and ambiguous slices localize error modes (e.g., over-diagnosis, under-diagnosis, incorrect multi-answering), as evidenced by error-type analyses (Song et al., 13 Feb 2026).
• Empirically validated performance separation: In QA, models trained on only the “diagnostic” 33% hard slice can outperform those trained on the full synthetic set and even LLM-generated data (Shi et al., 2023).

A plausible implication is that routine random splits may significantly underestimate the brittleness or blind spots of models, since challenging edge cases are diluted or omitted.

5. Mathematical and Procedural Specification

Formulas for diagnostic splitting are explicit:

• QaDynamics: $\alpha\%$3, $\alpha\%$4; split sets by quantiles of $\alpha\%$5, $\alpha\%$6 (Shi et al., 2023).
• T-GRAB: For periodic tasks, $\alpha\%$7, $\alpha\%$8, $\alpha\%$9 (no period split across boundaries). For causal/long-range, $\mu$0, $\mu$1, remainder test (Dizaji et al., 14 Jul 2025).
• MentalBench: Information completeness $\mu$2, diagnostic complexity $\mu$3 as count of consistent diagnoses (Song et al., 13 Feb 2026).

These procedures ensure reproducibility and transparency of diagnostic assessment.

6. Empirical Findings and Interpretative Value

Empirical results consistently show that diagnostic splits:

• Isolate model failure modes unobservable in aggregate statistics.
• Produce substantial accuracy gains by concentrating training on high-informative, high-difficulty synthetic subsets.
• Reveal calibration deficiencies (e.g., LLMs failing to throttle the number of diagnoses in ambiguous cases; over-commitment vs. under-commitment) that are invisible to average accuracy.
• Identify the impact of data cleaning and distractor pruning, each with quantifiable contributions to final model performance (Shi et al., 2023).

This suggests that diagnostic splits are crucial not only for evaluation but also for advancing model robustness and interpretability by focusing both probing and development effort on systematically challenging regions of the problem space.

7. Limitations and Scope

Synthetic diagnostic splits, while highly informative, depend on the fidelity and scope of the synthetic generative process. Any biases, incompleteness, or misalignment in the underlying ontologies or difficulty axes will propagate into the evaluation. Additionally, their relevance to real-world distributional generalization is bounded by the representativeness of the crafted synthetic challenges.

Nevertheless, in controlled settings—especially zero-shot QA, structured diagnostic reasoning, and learning on temporal graphs—synthetic diagnostic splits provide unique, indispensable tools for rigorous model analysis and comparison. Their adoption in contemporary benchmarks marks an evolution in empirical machine learning methodology towards interpretable and adversarially informative evaluation strategies (Shi et al., 2023, Dizaji et al., 14 Jul 2025, Song et al., 13 Feb 2026).