Code Sparsification: Concepts & Methods

• Code sparsification is the reduction of coordinates or constraints while preserving invariants, such as approximating the Hamming weight in linear codes.
• The methodology includes weighted (1 ± ε) and one-sided sparsifiers with tight existential bounds and NP-hardness in efficient computation.
• Applications extend to graph cut sparsification, optimized numerical kernels, and source-code analysis in learned models.

Code sparsification denotes the reduction of coordinates, constraints, operations, tokens, or parameters while preserving a specified invariant. In the coding-theoretic formulation, the notion was introduced as an analogue of cut sparsification in graphs: for a linear code $\mathcal{C} \subseteq \mathbb{F}_q^n$ of dimension $k$, one seeks a small weighted coordinate subset whose projection approximately preserves the Hamming weight of every codeword (Khanna et al., 2023). In other research areas, the same phrase is also used for sparsity-specific specialization of sparse numerical kernels, long source-code models, and learned architectures that process code or sparse data (Herholz et al., 2021, Yang et al., 2023).

1. Formal problem statements

A $(1 \pm \epsilon)$-sparsification of size $s$ for a code $\mathcal{C} \subseteq \mathbb{F}_q^n$ is given by a weighted set $S \subseteq [n]$ with $|S| \leq s$ such that for every codeword $c \in \mathcal{C}$ the projection $c|_S$ has weighted Hamming weight that is a $(1 \pm \epsilon)$ approximation of the Hamming weight of $k$0 (Khanna et al., 2023). The defining inequality is

$k$1

where

$k$2

A distinct but closely related notion is the one-sided $k$3-sparsifier. For a linear code $k$4, a set $k$5 is a one-sided $k$6-sparsifier if for every codeword $k$7,

$k$8

In this setting, the sparsifier is unweighted, and the objective is to find $k$9 of minimum size while retaining at least an $(1 \pm \epsilon)$0 fraction of the weight of every codeword (Grigorescu et al., 3 Oct 2025). The projected code is written $(1 \pm \epsilon)$1, and the distance guarantee is often stated as $(1 \pm \epsilon)$2 (Gharan et al., 5 Feb 2025).

These formulations generalize graph and hypergraph cut sparsification. The connection is explicit: code sparsification immediately implies known results on graph and hypergraph cut sparsification up to polylogarithmic factors, and also yields spectral Cayley graph sparsifiers over $(1 \pm \epsilon)$3 by Cayley graphs (Khanna et al., 2023).

2. Existence theorems and quantitative bounds

The foundational existential theorem states that every linear code over a finite field admits a small weighted sparsifier. Specifically, every linear code $(1 \pm \epsilon)$4 of dimension $(1 \pm \epsilon)$5 has a $(1 \pm \epsilon)$6-sparsifier of size

$(1 \pm \epsilon)$7

with the bound tight up to polylogarithmic terms (Khanna et al., 2023). A key structural ingredient is a generalized Karger-style counting theorem: for any code and any integer $(1 \pm \epsilon)$8, there is a subset of at most $(1 \pm \epsilon)$9 coordinates $s$0 such that on the remaining coordinates $s$1, the restricted code has at most $s$2 codewords of weight at most $s$3 for every integer $s$4 (Khanna et al., 2023).

For unweighted one-sided sparsification, a sharper coordinate-subset theorem is known. For every $s$5-dimensional linear code $s$6 there exists a set $s$7 of size at most

$s$8

such that for any codeword $s$9,

$\mathcal{C} \subseteq \mathbb{F}_q^n$0

and in particular $\mathcal{C} \subseteq \mathbb{F}_q^n$1 (Gharan et al., 5 Feb 2025). The proof is existential rather than constructive: it uses equivalence classes on $\mathcal{C} \subseteq \mathbb{F}_q^n$2 defined by flipping sets by codewords, chooses a maximal set in each class, and combines a counting argument with Hoeffding’s inequality.

These bounds have immediate applications. The weighted theory gives near-linear size sparsifiers for CSPs over $\mathcal{C} \subseteq \mathbb{F}_q^n$3-valued variables whose unsatisfying assignments can be expressed as the zeros of a linear equation modulo a prime $\mathcal{C} \subseteq \mathbb{F}_q^n$4, yields a complete characterization of ternary Boolean CSPs that admit near-linear size sparsification, and gives the first proof of the existence of spectral Cayley graph sparsifiers over $\mathcal{C} \subseteq \mathbb{F}_q^n$5 by Cayley graphs (Khanna et al., 2023). The one-sided theory implies that any connected graph $\mathcal{C} \subseteq \mathbb{F}_q^n$6 with $\mathcal{C} \subseteq \mathbb{F}_q^n$7 edges and $\mathcal{C} \subseteq \mathbb{F}_q^n$8 vertices has at least $\mathcal{C} \subseteq \mathbb{F}_q^n$9 many $S \subseteq [n]$0-thin subgraphs (Gharan et al., 5 Feb 2025).

3. Redundancy, chain length, and moonflowers

Later work recast sparsifiability in terms of intrinsic structural parameters. For a code $S \subseteq [n]$1, the non-redundancy $S \subseteq [n]$2 is

$S \subseteq [n]$3

This parameter is a universal lower bound on sparsifier size, and in the unweighted case it is also sufficient up to polylogarithmic factors: $S \subseteq [n]$4 For linear codes over a field, $S \subseteq [n]$5 equals the dimension of the code (Brakensiek et al., 2024).

For weighted sparsification, the controlling parameter is chain length rather than non-redundancy. If $S \subseteq [n]$6 denotes the largest sequence of codewords and coordinates $S \subseteq [n]$7 such that $S \subseteq [n]$8 and for all $S \subseteq [n]$9, $|S| \leq s$0, then

$|S| \leq s$1

Thus, in the weighted setting, chain length precisely determines sparsifiability up to polylogarithmic factors (Brakensiek et al., 2024).

A complementary extremal-combinatorial line introduces moonflowers. A family of sets $|S| \leq s$2 is a $|S| \leq s$3-moonflower if each set $|S| \leq s$4 contains at least one element that is absent from all the others. Equivalently, there is a set $|S| \leq s$5 such that $|S| \leq s$6 are non-empty and pairwise disjoint. For a code $|S| \leq s$7 and the support family $|S| \leq s$8, the maximum size $|S| \leq s$9 of a moonflower in $c \in \mathcal{C}$0 equals $c \in \mathcal{C}$1 (Lovett et al., 9 May 2026).

This correspondence leads to improved sparsification bounds. Earlier upper bounds had a polylogarithmic dependence on the block length,

$c \in \mathcal{C}$2

where $c \in \mathcal{C}$3. The moonflower method improves this to

$c \in \mathcal{C}$4

and proves a matching lower bound of

$c \in \mathcal{C}$5

showing that logarithmic dependence on the block length is necessary (Lovett et al., 9 May 2026). In this sense, redundancy and moonflower number play the role that cut-counting and extremal set-system structure play in classical graph sparsification.

4. Computational status and adjacent coding-theoretic variants

Existence and optimization are sharply separated. The minimal one-sided $c \in \mathcal{C}$6-sparsifier problem is NP-hard via a reduction from the Nearest Codeword Problem, and the paper also proves hardness of approximation results: finding a one-sided $c \in \mathcal{C}$7-sparsifier $c \in \mathcal{C}$8 so that $c \in \mathcal{C}$9 is NP-hard for any constant $c|_S$0, and stronger inapproximability follows under standard complexity assumptions (Grigorescu et al., 3 Oct 2025). A common misconception is that existential sparsification theorems automatically imply efficient algorithms; in fact, the general weighted theory is explicitly existential, and no efficient polynomial time algorithm is currently known for general codes because detecting or encoding all low-weight codewords is as hard as the minimum distance problem in coding theory (Khanna et al., 2023).

A distinct representation-level problem is sparsifying parity-check matrices rather than sparsifying the code as a set of codewords. If $c|_S$1 is a parity-check matrix, invertible row operations preserve the code, so one may seek an equivalent $c|_S$2 with fewer one-entries. The proposed heuristics are greedy local search and simulated annealing, with objective $c|_S$3 equal to the number of ones in $c|_S$4, and the resulting matrices provide faster ML decoding procedures, especially for large codes (Russo et al., 2020).

Distributed gradient coding introduces yet another variant of structure-preserving sparsification. The Sparse Gaussian gradient code and the Expansion-Preserving gradient code both follow a two-step framework: first generating a random matrix and then applying distinct sparsification procedures. The Sparse Gaussian code uses

$c|_S$5

with a correlated multivariate Gaussian masked by Bernoulli random variables so that first- and second-order moments match BIBD incidence structure in expectation. The Expansion-Preserving code starts from a weighted expander-like matrix and applies DegreePreservingSparsify so that weighted degrees are preserved exactly and the Laplacian perturbation satisfies

$c|_S$6

Both codes achieve worst-case error performance comparable to BIBD gradient code when such a code with the same parameters exists, while substantially extending the feasible range of system parameters (Jiang et al., 11 Apr 2026).

5. Sparsity-specific specialization of executable code

In numerical computing, code sparsification often means compiling or generating code that is specialized to a fixed sparsity pattern. The central premise is that symbolic structure is known before repeated numerical evaluation, so loops, expressions, and memory layouts can be specialized away from generic sparse-library execution.

System	Core mechanism	Reported result
Sympiler (Cheshmi et al., 2017)	decoupling symbolic analysis from numerical manipulation; inspector-guided transformations	average speedups over Eigen and CHOLMOD of 3.8X and 1.5X
Expression-tree generator (Herholz et al., 2021)	whole-expression pre-analysis and unrolling; redundant expression elimination; grouping; leaf harvesting	speedups of two orders of magnitude on CPUs and three orders of magnitude on GPUs
SpComp (Basak et al., 2023)	Essential Indices Analysis; piecewise-regular loops; no run-time overhead	79% performance gain against TACO, 83% against the piecewise-regular code generator, and 65% against CHOLMOD

Sympiler transforms sparse matrix codes by decoupling the symbolic analysis phase from the numerical manipulation stage. It applies inspector-guided transformations such as Variable Iteration Space Pruning and 2D Variable-Sized Blocking so that the generated code can remove indirect accesses and unnecessary loop iterations, especially for sparse triangular solve and sparse Cholesky factorization (Cheshmi et al., 2017).

The expression-tree approach symbolically executes unoptimized C++ code with a custom Symbolic class, constructs a complete expression DAG for every output value, and then performs structural and algebraic hashing, local and global decomposition, grouping of identical expression structures, leaf harvesting, algebraic simplification, and hardware-aware kernel generation. The method assumes a fixed sparsity pattern, supports reverse-mode automatic differentiation, and specializes kernels for SIMD, TBB, OpenMP, Cuda, or HIP backends (Herholz et al., 2021).

SpComp provides a fully automatic compile-time technique for matrix operations with fixed sparsity structure. Its Essential Indices Analysis statically identifies essential data indices and essential iteration indices, detects fill-in, and emits indirect-reference-free, piecewise-regular loops. The generated code is applicable to complex matrix operations generating sparse output and handling mutable sparse matrices, including sparse Cholesky decomposition (Basak et al., 2023).

6. Source-code analysis and sparsification in learned models

A related usage of code sparsification appears in models that analyze source code. SparseCoder integrates sparse attention and learned token pruning so that Transformer-based source-code models can handle significantly longer input sequences—at least twice as long within the reported hardware limits—and are four times faster in runtime, with a 50% reduction in FLOPs and a negligible performance drop of less than 1% compared to Transformers using sparse attention. The inference FLOPs scale linearly with token length rather than quadratically, and the model visualizes non-trivial tokens layer-wise for interpretability (Yang et al., 2023).

Neural sparsification methods for the models surrounding code and sequence processing follow a different formalism but share the same objective of removing redundant computation. Dep-$c|_S$7 replaces the mean-field approximation for binary gates with dependency modeling parameterized by an MLP, and on ImageNet with ResNet50 it prunes about 37% of FLOPs at only 1.38% accuracy drop; it also reduces Total Time to Solution by 20–40% compared to $c|_S$8-HC (Li et al., 2021). Feather uses a Straight-Through Estimator together with a new thresholding operator

$c|_S$9

and gradient scaling, attaining ImageNet/ResNet-50 Top-1 accuracies of 76.93 at 90% sparsity, 75.27 at 95%, 72.92 at 98%, and 68.85 at 99% sparsity (Georgoulakis et al., 2023). In gated recurrent networks, Bayesian sparsification extends Sparse Variational Dropout from weights and neurons to gate preactivations and information flow in LSTM, making some gates and information flow components constant and achieving, for example, about 19,747x compression on IMDb in the W+G+N setting (Lobacheva et al., 2018).

These learned-model results are not code sparsification in the strict sense of projecting a code $(1 \pm \epsilon)$0 onto a small coordinate set. They nonetheless instantiate the same structural principle: sparsity is exploited either by selecting a small support that preserves a global objective, or by specializing a computation to the subset of coordinates, tokens, or parameters that materially affect the outcome.