Demystifying Data Organization for Enhanced LLM Training

Abstract: LLMs have revolutionized various fields, yet their training efficiency is heavily reliant on effective data curation. While data selection has been widely studied, the strategic data organization for enhanced training remains an underexplored area, particularly since current LLMs are often trained for only one or a few epochs. This paper systematically explores the influence of data organization on LLM training by reusing pre-computed sample-level scores originally generated for data efficiency, thereby incurring minimal additional computational overhead. We identify and formalize four key guidelines for optimizing data organization: Boundary Sharpening, Cyclic Scheduling, Curriculum Continuity, and Local Diversity. Guided by them, we introduce two novel data ordering methods termed STR and SAW. Extensive experiments across different model scales and data sizes, encompassing both pre-training and SFT stages, validate the effectiveness of our summarized guidelines. They also demonstrate the robustness of our proposed data ordering methods in enhancing the stability and performance of LLM training. Github Link: https://github.com/microsoft/data-efficacy/

• The paper demonstrates that innovative data organization significantly improves LLM training performance by leveraging pre-computed sample scores.
• It introduces algorithmic strategies such as Boundary Sharpening, Cyclic Scheduling, and Curriculum Continuity to optimize training sequences.
• Empirical results show enhanced accuracy and gradient stability across domains and model scales with minimal additional computational cost.

Demystifying the Role of Data Organization in LLM Training

Motivation and Context

The paper "Demystifying Data Organization for Enhanced LLM Training" (2605.30334) addresses the largely unexamined dimension of data ordering in the training of LLMs. While previous research has rigorously studied data selection and curation—spanning acquisition, filtering, mixing, and deduplication—this work underscores that training efficacy is influenced not only by what data is selected but also by the sequence in which data is presented, especially under the constraint of few training epochs typical in current LLM pipelines. Leveraging pre-computed sample-level scores, primarily obtained for selection purposes, the authors demonstrate that strategic reordering incurs negligible additional computational overhead and can substantially improve training stability and downstream performance.

Formalization and Methodological Contributions

The authors propose a unified formalism, distinguishing three stages: data scoring ($\bm{\gamma}$), data selection ($f_s$), and data organization ($f_o$). Data organization alters the training sequence (ordering) rather than the composition (set membership) of training data. Critically, the same score vector used for selection can be repurposed to induce alternative orderings. Through this lens, curriculum learning becomes a special case where samples are sorted in ascending score order.

The paper systematically identifies four organizational guidances, each operationalized as a concrete algorithmic strategy:

1. Boundary Sharpening: Control the score distribution at initialization and termination—start with low-score (easy/low-noise) data for stable optimization, end with high-score (quality/complex) data to maximize final model competencies.
2. Cyclic Scheduling: Periodically revisit the full score spectrum to prevent catastrophic forgetting, implemented as "Folding Ordering" (FO), which partitions sorted data cyclically rather than strictly monotonic.
3. Curriculum Continuity: Ensure smooth transitions and attribute continuity across training phases ("Zig-zag Ordering" [ZIG]), mitigating variance spikes in gradients.
4. Local Diversity: Inject stochastic shuffling into local windows ("Jittering Ordering" [JIT]) to maintain batch-level attribute and gradient diversity, reducing overfitting tendencies.

Two cross-guidance methods synthesizing these principles are introduced: Stair Ordering (STR) and Saw Ordering (SAW). STR combines global progression and periodic review with local diversity; SAW generalizes STR by enforcing continuity constraints at region boundaries via the zig-zag mechanism.

Empirical Analysis and Numerical Results

Extensive experiments were conducted using multiple corpora—FineWeb-Edu and QuRatedPajama for general domain, DeepMath-103K for mathematical reasoning, and OpenCodeInstruct for code generation. Models ranged from 160M to 1.7B parameters, using Mistral and Qwen3 architectures. Method comparison encompassed random ordering, basic curriculum learning (CL), DELT folding, and implementations of each organizational guidance.

Numerical highlights:

• Boundary Sharpening: Ending with high-score data sharply increases accuracy; configurations concluding with low-score data underperform (see SEG analysis). Start-phase score has negligible isolated effect.
• Cyclic Scheduling: FO (periodic review) consistently surpasses CL, but performance is sensitive to the number of folding layers.
• Curriculum Continuity: ZIG (triangle-wave continuity) outperforms FO, yielding stable gradients and preventing optimization shocks (Figure 1).
• Local Diversity: JIT application yields consistent gains across sorted strategies. JIT models exhibit flatter loss landscapes under weight perturbation, indicating superior robustness (Figure 2).

STR and SAW—the two cross-guidance strategies—consistently outperformed random, CL, and DELT baselines across all domains and scales. For example, on FineWeb-Edu at 1.7B scale, average accuracy improved from 47.72% (random) to 49.85% (STR) and 50.11% (SAW), with similar patterns for downstream mathematics and code benchmarks.

Figure 3: SAW consistently outperforms Random and CL; score-index distributions exhibit more structured curricula and greater gains at higher model capacities.

Figure 4: Score-index distributions induced by different orderings (L=3), revealing structural and progressive patterns under FO, ZIG, STR, and SAW.

Figure 5: Accuracy trajectories for SEG variants (Boundary Sharpening), illustrating sharp performance boosts when ending with high-score data.

Figure 6: Perplexity trajectories (PPL) for $D_e$; FO mitigates forgetting of simple data, maintaining low PPL across training cycles.

Figure 1: Gradient norm trajectories; ZIG maintains stable gradients at cycle boundaries, preventing optimization shocks.

Figure 2: Sensitivity analysis under weight perturbation; JIT-enhanced models demonstrate slower degradation, indicative of flatter minima.

Theoretical and Practical Implications

The primary theoretical implication is that data organization, rather than just composition or filtration, fundamentally shapes optimization dynamics in LLMs. The paper establishes that sequence order—particularly at initialization, termination, and at local/transition boundaries—modulates gradient variance, batch-level diversity, and long-term retention of foundational knowledge. These effects persist across model scales and data volumes, evidenced by robust scaling law extrapolations using Chinchilla-style fits.

Practically, this approach is highly scalable: it leverages existing (possibly multi-dimensional) sample-level scores from curation pipelines without additional computational burden. As data curation sophistication increases, these scores may include quality, difficulty, factuality, or domain relevance, further enriching the utility of organizational ordering. STR and SAW thus offer flexible, plug-and-play recipes for manufacturers of large-scale corpora, improving convergence and generalization for new LLM architectures, and potentially enabling more efficient compute utilization.

Future Directions

Future research may extend these organizational principles to multi-modal, graph, or cross-lingual settings, and explore adaptive/online ordering mechanisms at scale. Additionally, deeper integration with active learning and curriculum construction for instruction tuning—especially in non-language modalities—remains an open avenue. The findings motivate further inquiry into loss landscape behavior and optimization stochasticity under various data orderings, with implications for robustness and transfer learning.

Conclusion

This paper provides a systematic account of data organization as a critical factor in LLM training, operationalizing four guidances and proposing STR and SAW strategies that demonstrably enhance model performance and stability. The approach is efficient, robust across scales, and generalizes to diverse data modalities and benchmarks. Data organization is thus established as an indispensable dimension in optimizing modern LLMs, complementing selection and filtering to yield improved accuracy, stability, and robustness.