EleutherAI Pythia Model Suite

• EleutherAI’s Pythia suite is a set of 16 autoregressive LLMs designed for controlled, reproducible studies of training dynamics and scaling laws.
• It employs a uniform decoder-only Transformer architecture with rotary embeddings, FlashAttention, and parallel attention–MLP blocks, enabling direct scale comparisons across models from 70M to 12B parameters.
• The suite facilitates empirical analyses on memorization, term-frequency effects, and bias reduction, with complete reproducibility ensured by fixed data shuffling and 154 saved checkpoints per model.

EleutherAI’s Pythia suite is a rigorously designed, fully open set of 16 autoregressive LLMs, each trained with precise experimental control to enable the study of training-time dynamics and scaling laws. Pythia models, ranging from 70M to 12B parameters, cover two parallel tracks: one suite trained on the raw Pile dataset (300B tokens) and a second on a near-deduplicated Pile (∼207B tokens). Every model in the suite is trained on the identical sequence of examples, with checkpoints and exact dataloader reproducibility provided for all 16 models and 154 saved checkpoints each, establishing Pythia as a canonical controlled laboratory for empirical LLM research (Biderman et al., 2023).

1. Model Suite Architecture

Pythia employs a decoder-only Transformer backbone with rotary embeddings, FlashAttention, parallel attention–MLP blocks, and untied input/output embeddings. Architecture is fixed across suites aside from scale, supporting methodologically sound model comparisons. Each model’s hyperparameters—including non-embedding parameter count, number of layers, hidden dimensions, and attention heads—are provided in the following table:

Model Size	Non-Embedding Params	Layers	Hidden Dim ($d$)	Attention Heads
70M	18.9M	6	512	8
160M	85.1M	12	768	12
410M	302M	24	1024	16
1.0B	806M	16	2048	8
1.4B	1.21B	24	2048	16
2.8B	2.52B	32	2560	32
6.9B	6.44B	32	4096	32
12B	11.3B	36	5120	40

This uniformity allows for direct scaling analyses; loss curves for Pythia models follow the same two-term power law in model size ($N$) and data tokens ($D$) as previously documented in Kaplan et al. (2020):

$L(N, D) \approx a N^{-\alpha} + b D^{-\beta}$

with exponents $\alpha, \beta \approx 0.05$–$0.25$ for the class of LLM autoregressors.

2. Training Protocol and Dataset Control

All models are trained on “The Pile” (300B tokens) or its deduplicated variant (reduced to ∼207B tokens via MinHashLSH, Jaccard threshold 0.87). The entire Pythia suite leverages a single, fixed shuffle of the data, ensuring that, for any training step $t$, every model has been exposed to an identical prefix of examples. Each model is trained for 300B tokens (approximately 143,000 iterations at 2,097,152 tokens/step), with checkpoints recorded at step 0, exponentially (steps $2^k$ for $k = 0...9$), and then every 1,000 steps (totaling 154 per model).

Reproducibility is enforced by providing pre-tokenized data shards (“mmap” layout compatible with GPT-NeoX) and a Python dataloader reconstruction script that restores the exact sequence of training data seen per checkpoint.

3. Analytical Methodologies and Metrics

Pythia enables in-depth interrogation of LLM properties through meticulously defined methodologies.

A. Memorization:

A sequence $s$ of length $N$0 is $N$1–memorized if, when conditioned on its first $N$2 tokens, the model outputs the next $N$3 tokens exactly. For Pythia, $N$4 (per Carlini et al. 2021).

B. Term-Frequency Effects:

For each target fact or item $N$5, frequency $N$6 is computed over the training prefix. Items are binned by $N$7, measuring downstream accuracy $N$8 and characterizing $N$9 as a function of both model size and training progression.

C. Few-Shot and Zero-Shot Metrics:

Zero-shot: Models respond to a single prompt, evaluated by log-probability or exact-match. $D$0–shot: Prefixing $D$1 labeled in-context examples, then querying as zero-shot.

D. Bias Quantification:

• WinoBias (Zhao et al. 2018): Stereotype accuracy $D$2
• CrowS-Pairs (Nangia et al. 2020, Nevéol et al. 2022): Each pair $D$3 compared by perplexity, with

$D$4

E. Case Study Protocols:

• Gender-Bias Interventions: At 93,000 and 122,000 steps (approx. 7% and 21% before end), training is resumed swapping all masculine pronouns for feminine, followed by evaluation on WinoBias and CrowS-Pairs.
• Memorization analysis: Memorization events are tracked in a sliding mini-batch window and compared to a Poisson model via Q–Q plot.
• Term-frequency: Correlations for arithmetic and TriviaQA are tracked throughout training.

4. Empirical Insights and Case Study Findings

• Scaling Laws: Loss curves for all models exhibit the expected power-law dependence; deduplication produces negligible change in test perplexity.
• Memorization: The frequency of (32,32)-memorization is statistically uniform across training steps, best modeled by a Poisson process, indicating that training data order does not bias memorization events.
• Term-Frequency Effects: Emergence of strong $D$5 correlation in models $D$62.8B parameters, but only after $D$745% of training tokens; smaller models do not display such emergent dependencies.
• Bias Reduction: Pronoun-swapping interventions conducted at 7% and 21% remaining training reduce stereotype bias on WinoBias and CrowS-Pairs; larger models show greater bias reduction. These modifications minimally affect perplexity and LAMBADA performance.
• Architectural Note: Contrary to prevailing assumptions, fully dense attention+MLP structures and large, uniform batch sizes yield robust training even at the 70M parameter scale.

5. Tools, Resources, and Reproducibility

Pythia prioritizes comprehensive reproducibility. Model code, weights, and all 154 training checkpoints per model are publicly accessible. Core resources include:

• Repository: https://github.com/EleutherAI/pythia
• Model weights: Published on HuggingFace (from 70M up to 12B-deduped variants)
• Data: Raw Pile shards (https://pile.eleuther.ai/) and deduplicated Pile (HuggingFace: EleutherAI/raw_deduplicated_pile)

• Reproduction protocol:

  1. Clone repository; install GPT-NeoX and all dependencies
  2. Fetch pre-tokenized data with scripts/download_data.sh
  3. Restore dataloader state per checkpoint using scripts/reconstruct_dataloader.py
  4. Re-execute analysis pipelines for memorization, bias, and term-frequency studies using provided Jupyter notebooks

Pythia’s methodologically controlled, open, and reproducible design establishes it as the first fully public suite supporting rigorous, time-resolved scaling-law studies in autoregressive LLMs (Biderman et al., 2023).

6. Contributions and Research Significance

Pythia enables controlled, large-scale comparative studies previously infeasible with closed or partially open LLM suites. Its synchrony of data order, scaling, and complete checkpoint release uniquely positions it for longitudinal analyses of phenomena including memorization, scaling behavior, bias emergence, and dataset effects. Pythia’s reproducibility, fidelity to open science practices, and breadth of analytic coverage provide a foundational resource for empirical investigation into LLM development, facilitating advances in interpretability, reliability, and fairness assessment across the research community (Biderman et al., 2023).