Sparse Chinchilla: Scaling Laws and Sparse Models

• Sparse Chinchilla is a research framework that integrates Chinchilla scaling principles with sparsity in model circuits, dynamic masking, and pre-training for efficient compute allocation.
• It leverages systems-level innovations like CSR-based sparse execution and dynamic mask attention to reduce memory and FLOPs while maintaining model accuracy.
• It introduces a modified scaling law based on average active parameters, unifying dense and sparse paradigms for optimal training and cost-effective deployment.

“Sparse Chinchilla” is an Editor’s term for a cluster of research directions that connect the Chinchilla family of models and scaling laws to sparsity at several levels: sparse causal circuits inside large transformers, true sparse kernels in training systems, sparse attention for long-context training, and sparse pre-training laws that replace a fixed parameter count with the average active parameter count over training. In the mechanistic sense, Chinchilla-70B exhibits a multiple-choice label-selection capability whose final stage is concentrated in a relatively small set of components and low-dimensional subspaces (Lieberum et al., 2023). In the systems and optimization senses, sparsity appears as CSR-based sparse execution without binary masks (Wesselink et al., 2024), as trainable dynamic mask sparse attention trained under Chinchilla Scaling Law settings (Shi et al., 4 Aug 2025), and as a unified dense–sparse scaling law based on average parameter count over pre-training (Jin et al., 21 Jan 2025).

1. Conceptual scope

The phrase “Sparse Chinchilla” does not denote a single canonical algorithm. It instead names a family of attempts to retain the compute-optimal logic associated with Chinchilla while replacing some dense assumptions with sparse ones. The relevant literature separates into at least four strands: internal circuit sparsity, implementation-level sparsity, sparse attention, and sparse pre-training. A further strand extends Chinchilla-style reasoning to deployment settings in which inference cost or test-time sampling materially changes the optimal allocation of compute.

Dimension	Representative mechanism	Source
Circuit sparsity	45 output nodes and rank-3 Q/K/V subspaces in Chinchilla-70B	(Lieberum et al., 2023)
True sparse execution	MKL CSR kernels without binary masks	(Wesselink et al., 2024)
Sparse attention	Dynamic Mask Attention with $O(n w d_h)$ compute	(Shi et al., 4 Aug 2025)
Sparse scaling laws	Average parameter count over pre-training	(Jin et al., 21 Jan 2025)

The scaling-law backdrop is the Chinchilla joint loss form

$L(N,T)=E+\frac{A}{N^{\alpha}}+\frac{B}{T^{\beta}},$

together with the dense-transformer approximation $C_{\text{train}} \approx 6NT$ for training compute (Sardana et al., 2023). Subsequent work argues that future scaling studies should use total parameters and compute, because much of the Kaplan–Chinchilla discrepancy can be attributed to non-embedding parameter counting and small-scale analysis (Pearce et al., 2024). This places “Sparse Chinchilla” within a precise methodological lineage: it is not merely about pruning or masks, but about redefining what counts as the effective parameter axis and the effective compute axis without discarding the empirical logic of Chinchilla-style optimality.

2. Mechanistic sparsity inside Chinchilla-70B

The most literal sense of “Sparse Chinchilla” comes from mechanistic interpretability in the 70B-parameter Chinchilla model. In a case study on multiple-choice question answering, the capability under analysis is not domain knowledge itself, but the mapping from the correct answer text to the correct answer label $A/B/C/D$. The model is a decoder-only transformer with 80 layers, 64 attention heads per layer, RMSNorm before each attention and MLP block and before the final unembedding, and linear relative positional embeddings in the Transformer-XL style. Under fixed final-residual RMS, logits can be written as a linear sum over MLP outputs and attention-head outputs, which makes direct logit attribution straightforward (Lieberum et al., 2023).

The central empirical result is a strong component-sparsity claim. Among all heads and MLPs, 45 nodes—32 heads and 13 MLPs—explain 80% of the summed positive direct effect on the correct-label logit. These components are termed “output nodes.” Activation patching then shows that patching all 45 output nodes from an intervention prompt into an original prompt recovers most of the model’s MMLU performance in loss and top-1 label accuracy. This is a narrow but clear example of a capability whose final-stage implementation is localized to a small subset of the network rather than diffusely distributed across all layers.

Attention visualization refines this picture into four functional classes: Correct Letter heads, Uniform heads, Single-letter heads, and Amplification heads. Correct Letter heads are the canonical sparse subcircuit. At the final token they attend sharply to the correct label token and write a representation that increases the logit for that label. The proposed decomposition is semantically specific: queries combine an “are-you-a-label?” feature with an “Nth label” feature, keys encode “item number in the enumeration,” and values encode token identity. This yields a circuit in which the model can determine which answer is correct upstream, then use a small set of late heads to map that fact onto the appropriate letter.

A second sparsity claim concerns low-rank structure. Using 1024 MMLU prompts, the authors collect queries at the final token and keys and values at label positions for Correct Letter heads, decompose queries and keys into means plus centered deltas, and find that $q_\delta \cdot k_\delta$ carries almost all information relevant to label discrimination. SVD/PCA on the centered query/key deltas and on values reveals a knee at three components: for keys and queries, 3 principal components capture roughly 65–80% of variance; for values, 3 components capture roughly 80–90% of variance. Rank-3 approximations preserve both direct effects and total effects on ordinary MMLU prompts, so the relevant Q/K/V behavior is functionally 3-dimensional on that distribution.

The geometric interpretation is an approximate “Nth item in an enumeration” feature. For a canonical Correct Letter head, projecting query and key deltas onto the first three singular vectors produces four distinct key clusters, one per label $A/B/C/D$, arranged approximately as vertices of a tetrahedron. Query deltas align with those directions more diffusely. This supports a sparse-subspace interpretation: the head’s behavior relevant to multiple-choice label selection is concentrated in a tiny subspace of its original representation space. At the same time, the paper identifies an important limitation. When answer labels are randomized or replaced by numbers $1$–$4$, the explanation becomes partial or fails; randomized capital letters retain only partial recovery, and numeric labels receive near-zero attention from Correct Letter heads. The result is therefore not full symbolic abstraction, but sparse, low-rank structure that is distribution-sensitive.

3. Systems-level sparsity and sparse attention

A different sense of “Sparse Chinchilla” concerns whether sparse models are actually executed sparsely. Nerva addresses precisely this issue by implementing static sparse neural networks in C++ using Intel MKL sparse matrix operations and CSR storage, thereby eliminating binary masks. The distinction is operational rather than merely notional: in mask-based dense code, zeroed weights are still stored and multiplied, whereas in Nerva only non-zero parameters are stored and traversed. On CIFAR-10 MLP experiments, this yields substantially reduced memory and training time at high sparsity while maintaining accuracy comparable to PyTorch; at 99% sparsity, runtime is reduced by a factor of $4\times$ relative to a PyTorch model using masks (Wesselink et al., 2024).

The library’s compute model is explicit. Forward propagation uses $Y = SX$ with sparse–dense multiplication, backpropagation uses $L(N,T)=E+\frac{A}{N^{\alpha}}+\frac{B}{T^{\beta}},$0 for input gradients, and weight gradients are computed only for non-zero positions. Memory likewise scales with non-zeros plus CSR index overhead rather than dense matrix shape. For the MLP $L(N,T)=E+\frac{A}{N^{\alpha}}+\frac{B}{T^{\beta}},$1, saved weights occupy 15 MB when dense, 295 KB at density $L(N,T)=E+\frac{A}{N^{\alpha}}+\frac{B}{T^{\beta}},$2, and 37 KB at density $L(N,T)=E+\frac{A}{N^{\alpha}}+\frac{B}{T^{\beta}},$3. The paper also emphasizes a threshold phenomenon: below roughly 80% sparsity, dense MKL GEMM may remain faster, whereas true sparse kernels become advantageous at higher sparsity and on larger models. In this usage, “Sparse Chinchilla” implies that any sparse scaling law only becomes physically meaningful if the implementation uses genuinely sparse kernels rather than mask-based dense surrogates.

Sparse attention provides a second systems-level route. Dynamic Mask Attention introduces a trainable dynamic mask sparse attention mechanism for long-context LLMs. The method retains a full KV cache but makes attention computation sparse through two mechanisms: a content-aware sparse mask generated from value representations, and position-aware sparse attention computation that skips unnecessary calculation regions. DMA has $L(N,T)=E+\frac{A}{N^{\alpha}}+\frac{B}{T^{\beta}},$4 compute and $L(N,T)=E+\frac{A}{N^{\alpha}}+\frac{B}{T^{\beta}},$5 memory, matching the complexity class of sliding-window attention while allowing the set of attended positions to be selected dynamically from content rather than from a static geometric rule (Shi et al., 4 Aug 2025).

This attention mechanism is explicitly positioned under Chinchilla Scaling Law settings. Comparative studies report that DMA outperforms multi-head attention, sliding window attention, multi-head latent attention, and native sparse attention in terms of perplexity under those settings. In a 1.7B parameter model, DMA also outperforms multi-head attention on standard benchmarks and on needle-in-a-haystack evaluation. The reported kernel benchmarks show substantial speedups over PyTorch SDPA, increasing with sequence length. In this strand of the literature, “Sparse Chinchilla” denotes sparse compute rather than sparse capacity: complete information is retained in the KV cache, but the model learns to spend FLOPs only on selected regions.

4. Sparse pre-training and the average-parameter scaling law

The most direct scaling-law formulation of “Sparse Chinchilla” appears in work on sparse pre-training. The key observation is that pruning during training makes the parameter count dynamic, so the standard Chinchilla law with a fixed $L(N,T)=E+\frac{A}{N^{\alpha}}+\frac{B}{T^{\beta}},$6 is not directly applicable. The proposed remedy is to replace the fixed parameter count with the average active parameter count over training,

$L(N,T)=E+\frac{A}{N^{\alpha}}+\frac{B}{T^{\beta}},$7

and then to use the modified law

$L(N,T)=E+\frac{A}{N^{\alpha}}+\frac{B}{T^{\beta}},$8

This law reduces exactly to the original Chinchilla form when $L(N,T)=E+\frac{A}{N^{\alpha}}+\frac{B}{T^{\beta}},$9 is constant, and it is presented as a single law that fits both dense and sparsely pre-trained LLMs (Jin et al., 21 Jan 2025).

Empirically, the paper reports the first systematic exploration of optimal sparse pre-training configurations for LLMs through 80 unique pruning schedules across different sparsity levels and training durations. The main schedule result is that initiating pruning at 25% of total training compute and concluding at 75% achieves near-optimal final evaluation loss. The recommended training structure has a dense phase, an iterative pruning phase, and a sparse recovery phase. Within the explored regime, sparse pre-training achieves the same final model quality as dense pre-training for equivalent compute budgets, but yields a smaller final model and thus potential inference savings.

The paper also gives a concrete lifetime-compute example. For a target loss of 1.89 and 100T inference tokens, an 80% sparse “Sparse Chinchilla” prescription yields an average parameter count of 28.0B, a final parameter count of 14.0B, training tokens of 16.6T, and lifetime FLOPs of $C_{\text{train}} \approx 6NT$0, corresponding to a 31.4% additional lifetime compute saving beyond a dense Beyond-Chinchilla prescription. A plausible implication is that sparse pre-training decouples the parameter count that governs training quality from the parameter count that governs deployment cost: the training trajectory behaves like a dense model of size $C_{\text{train}} \approx 6NT$1, while inference is performed with the smaller final sparse model.

5. Inference-aware and test-time-aware extensions

Chinchilla’s original optimization target is training loss under a fixed training compute budget. Several later works argue that this objective is incomplete once inference demand is large or test-time scaling is used. An inference-aware extension modifies the optimization problem by minimizing total training-plus-inference cost subject to a target loss. In the dense case, training FLOPs are approximated as $C_{\text{train}} \approx 6NT$2 and inference FLOPs as $C_{\text{train}} \approx 6NT$3, which shifts the optimal allocation toward smaller models trained on more tokens when inference demand is substantial. For a 30B-Chinchilla-quality target and $C_{\text{train}} \approx 6NT$4 inference tokens, the compute-optimal solution becomes a 13.6B-parameter model trained on $C_{\text{train}} \approx 6NT$5 as many tokens, reducing total FLOPs by 28% relative to a Chinchilla-style allocation (Sardana et al., 2023).

This dense inference-aware result matters to “Sparse Chinchilla” for two reasons. First, it makes clear that Chinchilla-optimality is not unique; the optimum depends on whether one minimizes training loss, lifetime FLOPs, or monetary cost. Second, it suggests that sparse or conditionally active architectures could exploit a decoupling between total capacity and active inference cost even more aggressively than dense models. The sparse-pre-training law based on $C_{\text{train}} \approx 6NT$6 can be read as one such decoupling, but the same logic also points toward MoE- or sparse-attention variants in which the training and inference cost models differ structurally.

Test-time scaling pushes the argument further. Train-to-Test ($C_{\text{train}} \approx 6NT$7) scaling laws jointly optimize model size $C_{\text{train}} \approx 6NT$8, training tokens $C_{\text{train}} \approx 6NT$9, and the number of inference samples $A/B/C/D$0 under separate training and inference compute budgets. With $A/B/C/D$1 and $A/B/C/D$2, smaller models receive more samples under a fixed test-time budget. Across eight downstream tasks, $A/B/C/D$3 forecasts move the optimum “radically into the overtraining regime,” far outside standard pretraining scaling suites, and empirical validation shows that heavily overtrained small models outperform Chinchilla-optimal checkpoints under equal training and inference budgets (Roberts et al., 1 Apr 2026).

In this regime, “Sparse Chinchilla” becomes a deployment-aware program rather than only a pre-training law. This suggests that sparse pre-training, sparse attention, and test-time scaling are not separate modifications but potentially complementary ones: a smaller or sparsely active model that is trained longer and sampled more often can be compute-optimal under end-to-end constraints even when a dense Chinchilla prescription would choose a larger model.

6. Methodological disputes, biases, and open problems

The “Sparse Chinchilla” literature inherits unresolved issues from the Chinchilla scaling-law literature itself. One line of work argues that the Kaplan–Chinchilla discrepancy is largely explained by counting non-embedding rather than total parameters and by performing the analysis at small scale; simulating Chinchilla under those conditions yields local exponents close to Kaplan’s, leading to the recommendation that future scaling studies use total parameters and compute (Pearce et al., 2024). This is directly relevant to sparse settings, because sparse models create further ambiguity about whether one should count total parameters, non-zero parameters, active parameters, or average active parameters.

A second line of critique targets Chinchilla’s parametric fitting procedure. A replication attempt of Hoffmann et al.’s third approach reports that the published estimates are inconsistent with the first two estimation methods, fail at fitting data reconstructed from the original plots, and report implausibly narrow confidence intervals; the rederived law is instead compatible with the earlier estimation procedures (Besiroglu et al., 2024). For sparse extensions this has a clear implication: if small errors in $A/B/C/D$4 and $A/B/C/D$5 already induce large shifts in optimal tokens-per-parameter in the dense case, the room for bias is larger once sparsity, dynamic masking, or time-varying parameter counts are introduced.

A third critique targets Chinchilla Approach 2 specifically. IsoFLOP parabola fits are shown to introduce systematic biases even on noise-free synthetic data, arising from sampling-grid width, uncentered IsoFLOP sampling, and loss-surface asymmetry $A/B/C/D$6. Applied to published Llama 3 IsoFLOP data, these biases imply a parameter underallocation corresponding to 6.5% of a $A/B/C/D$7 FLOP training budget and \$A/B/C/D$8412K–\$2.9M (Czech et al., 21 Mar 2026). The same work argues that Chinchilla Approach 3, especially when implemented via Variable Projection, largely eliminates these biases and scales naturally to richer formulations involving MoE sparsity, pruning, precision, data quality, or multimodal terms.

The open problems therefore cut across all meanings of the phrase. Mechanistic sparsity in Chinchilla-70B is real but distribution-limited; low-rank “Nth item in an enumeration” structure is only a partial explanation off distribution. True sparse execution yields real savings, but only above sparsity thresholds and, in Nerva’s case, only on CPU. Sparse attention can preserve a full KV cache while lowering compute, but current evidence is centered on long-context attention rather than full-model sparse pre-training. Average-parameter scaling laws unify dense and sparse pre-training empirically, yet they assume an idealized link between FLOPs and active parameters that many current frameworks do not realize in hardware. The cumulative picture is not a single theorem but a research program: Chinchilla-style optimality can be reformulated around sparse components, sparse subspaces, sparse kernels, sparse attention, and sparse training trajectories, provided that the counting conventions and fitting procedures remain explicit and technically sound.