Sparse Compositional Functions

• Sparse compositional functions are defined as hierarchical representations that decompose complex input–output maps into a polynomial number of low-arity functions with sparse dependencies.
• They leverage local interactions to replace exponential dependence on ambient dimensions with a polynomial scaling based on low-dimensional, reusable constituent functions.
• These concepts underpin architectures like Vector Networks, AC-SINDy, and Sparse Coding Transformers, enabling interpretable, efficient deep models that generalize well out-of-distribution.

Sparse compositional functions are functions whose global input–output map is assembled from a polynomial number of low-arity constituent functions, sparse dependency subsets, or sparse computational subgraphs, rather than from a single dense high-dimensional transformation. In recent work, the term has been used in several closely related senses: a bounded-fan-in directed acyclic graph (DAG) of local functions, a hierarchical composition over low-dimensional variable subsets, a sparse arithmetic circuit, or a sample-specific sparse combination of reusable operator atoms. Across these formalisms, the common thesis is that deep models can avoid the full curse of dimensionality when the target function is structured by hierarchical locality and sparse interaction, and that compositional generalization improves when this structure is encoded explicitly rather than left implicit in dense parameters (Danhofer et al., 3 Jul 2025, Huang et al., 25 May 2026).

1. Formal characterizations

A canonical formalization treats a target as compositionally sparse if it can be represented as the composition of at most $\mathcal{O}(\mathrm{poly}\, d)$ constituent functions, each depending on at most a small constant number of variables. This viewpoint is naturally expressed by a DAG whose leaves are inputs, whose internal nodes are low-dimensional constituent functions, and whose root is the final output (Danhofer et al., 3 Jul 2025).

A closely related statistical formulation writes the target as

$$f(x_1,\dots,x_p) = h\!\left(g_1(\mathbf{x}_{U_1}),\ldots,g_m(\mathbf{x}_{U_m})\right),$$

where each $U_i \subset \{1,\dots,p\}$ is typically small, each $g_i$ captures a local interaction, and $h$ composes these local outputs into the final response. In this formulation, sparsity lies in the dependency structure of the function itself: only small groups of variables interact directly, and higher-order effects are formed by recursively combining lower-order constituents (Lin et al., 14 May 2026).

DAG-based learning theory makes this local structure explicit by assigning to each node $v^{(\ell)}$ a function

$$g_{v^{(\ell)}}:\mathbb{R}^{d_{\mathrm{in}}(v^{(\ell)})}\to\mathbb{R},$$

where $$d_{\mathrm{in}}(v^{(\ell)})=|\mathrm{pa}(v^{(\ell)})|$$ is the number of parents of that node. The target is then the layered composition of nodewise functions across the DAG. This representation accommodates multi-index models, binary tree structures, and general hierarchical architectures (Huang et al., 25 May 2026).

A more algebraic formulation appears in arithmetic-circuit models of nonlinear dynamics. There, the function class is written as

$$f(x)=\sum_{k=1}^m \psi_k(x), \qquad \psi_k(x)=\prod_{j\in S_k}(w_{k,j}^T x),$$

so that nonlinear interactions are constructed compositionally from linear forms and multiplicative interactions instead of being selected from an explicitly enumerated feature library (Racioppo, 20 Apr 2026).

These definitions are stricter than standard sparsity and more informative than generic compositionality. Standard sparsity usually means a small support in one fixed representation, such as few nonzero coefficients. Generic compositionality merely says that a function is built by composing simpler functions, but those constituent functions may still be high-dimensional. Sparse compositionality combines both ideas: the computation is hierarchical, and each local module is itself low-dimensional or sparse (Danhofer et al., 3 Jul 2025).

An earlier kernel-theoretic analogue appears in compositional kernels defined recursively over a computation skeleton. Their associated random features are sparse because each feature depends on only a small number of random paths in the composition tree, again making locality and hierarchy the operative structural principles (Daniely et al., 2017).

2. Approximation theory, computability, and the curse of dimensionality

The principal theoretical motivation for sparse compositional functions is that they replace ambient dimension by local interaction dimension in approximation rates. A position paper on deep learning theory presents the contrast in stark terms: shallow networks with one hidden layer require complexity $\mathcal{O}(\epsilon^{-d})$ to approximate a compositionally sparse target to accuracy $$f(x_1,\dots,x_p) = h\!\left(g_1(\mathbf{x}_{U_1}),\ldots,g_m(\mathbf{x}_{U_m})\right),$$0, whereas deep networks that mimic the target DAG require complexity $$f(x_1,\dots,x_p) = h\!\left(g_1(\mathbf{x}_{U_1}),\ldots,g_m(\mathbf{x}_{U_m})\right),$$1 (Danhofer et al., 3 Jul 2025).

This perspective is sharpened by a finite-precision computability result. If a function $$f(x_1,\dots,x_p) = h\!\left(g_1(\mathbf{x}_{U_1}),\ldots,g_m(\mathbf{x}_{U_m})\right),$$2 is computable in time polynomial in the input and output bit-depths, then for every finite precision there exists a bounded-fan-in Boolean circuit of polynomial size and depth computing the discretized map. Replacing each gate by a constant-size neural emulator yields a deep network of polynomial size and depth achieving accuracy

$$f(x_1,\dots,x_p) = h\!\left(g_1(\mathbf{x}_{U_1}),\ldots,g_m(\mathbf{x}_{U_m})\right),$$3

The paper states this as a bridge from efficient Turing computability to bounded-fan-in DAGs and then to deep neural approximation, thereby making compositional sparsity a structural consequence of efficient computation at finite precision (Poggio, 13 Oct 2025).

Norm-constrained approximation theory extends this picture to overparameterized regimes. For DAG-structured sparse compositional targets, recent results show that deep ReLU networks with a Frobenius-norm product constraint can approximate the target at rates governed by the local input dimensions $$f(x_1,\dots,x_p) = h\!\left(g_1(\mathbf{x}_{U_1}),\ldots,g_m(\mathbf{x}_{U_m})\right),$$4 of the constituent functions rather than the ambient dimension $$f(x_1,\dots,x_p) = h\!\left(g_1(\mathbf{x}_{U_1}),\ldots,g_m(\mathbf{x}_{U_m})\right),$$5. The same work makes this explicit for representative subclasses: in multi-index models the relevant dimension is the projection rank $$f(x_1,\dots,x_p) = h\!\left(g_1(\mathbf{x}_{U_1}),\ldots,g_m(\mathbf{x}_{U_m})\right),$$6, and in binary tree structures the ambient dimension enters only through $$f(x_1,\dots,x_p) = h\!\left(g_1(\mathbf{x}_{U_1}),\ldots,g_m(\mathbf{x}_{U_m})\right),$$7 (Huang et al., 25 May 2026).

Generalization theory has also been adapted to this setting. For sparse ReLU DAGs, Rademacher-complexity bounds are expressed in terms of the maximum indegree of the graph and a pathwise norm product rather than the norms of dense layer matrices. For convolutional networks, the analysis depends on convolutional filter norms rather than the norms of associated Toeplitz matrices, and the paper reports that these bounds may be orders of magnitude better than standard norm-based bounds while being almost non-vacuous on simple classification problems (Galanti et al., 2023).

Taken together, these results support a unifying claim: sparse compositionality is not merely a representational convenience but a mechanism by which deep models can exchange exponential dependence on ambient dimension for polynomial dependence on local arity, smoothness, or intrinsic interaction order (Danhofer et al., 3 Jul 2025, Huang et al., 25 May 2026).

3. Structural model classes

Recent architectures operationalize sparse compositional functions by changing the object that is inferred, stored, or pruned.

Vector Networks replace each dense layer matrix with a dictionary of reusable rank-1 weight atoms. Each atom is

$$f(x_1,\dots,x_p) = h\!\left(g_1(\mathbf{x}_{U_1}),\ldots,g_m(\mathbf{x}_{U_m})\right),$$8

and after inference the effective sample-specific operator is

$$f(x_1,\dots,x_p) = h\!\left(g_1(\mathbf{x}_{U_1}),\ldots,g_m(\mathbf{x}_{U_m})\right),$$9

Because only a sparse set of coefficients is active, the resulting operator is low-rank and sample-specific, with

$U_i \subset \{1,\dots,p\}$0

Inference minimizes a layer-local energy balancing input reconstruction, sparsity, and top-down consistency, and learning then updates only the selected atoms through local residual signals under the paper’s “Atomic-Hebb” rule. The model is therefore described as a structural approach to sparse compositional function learning rather than a dense network with a sparsity penalty (Pokel et al., 27 May 2026).

AC-SINDy recasts sparse system identification as the learning of a sparse arithmetic circuit. It replaces explicit basis libraries with masked linear maps and multiplicative interaction layers, so that nonlinear features are built compositionally rather than enumerated in advance. Sparsity is imposed directly on circuit edges via a pruning criterion based on the first-order approximation

$U_i \subset \{1,\dots,p\}$1

and the resulting pruned graph remains interpretable as a symbolic expression. A second component separates latent state inference from dynamics identification through a learned encoder, shared dynamics, and multi-step supervision (Racioppo, 20 Apr 2026).

Homological Neural Networks use a two-stage pipeline in which Information Filtering Networks first infer a sparse dependency graph and then map that graph into a fixed-wiring sparse feedforward network. In the specific instantiation described in the paper, the dependency extractor is the Maximally Filtered Clique Forest, which accepts a candidate clique expansion $U_i \subset \{1,\dots,p\}$2 only when

$U_i \subset \{1,\dots,p\}$3

is positive. The resulting HNN contains trainable edges only when a lower-order interaction is a subset of a higher-order interaction, and its all-layer readout preserves additive contributions from multiple interaction orders (Lin et al., 14 May 2026).

Sparse Coding Transformer reformulates attention as sparse coding over learned encoding and decoding dictionaries. The sparse coefficients are computed with the soft-thresholding proximal operator

$U_i \subset \{1,\dots,p\}$4

and target-task coefficients are estimated from context coefficients through

$U_i \subset \{1,\dots,p\}$5

In this formulation, attention becomes a rule encoder, rule transfer mechanism, and reconstruction module, with sparsity intended to localize the active compositional relations (Chen et al., 25 Nov 2025).

Sparse Mixture-of-Experts models provide a complementary viewpoint in which the sparse compositional function is distributed over experts. The relevant computation is

$U_i \subset \{1,\dots,p\}$6

where the selected expert set $U_i \subset \{1,\dots,p\}$7 has cardinality $U_i \subset \{1,\dots,p\}$8. The central claim is not that extreme sparsity is universally optimal, but that the appropriate degree of expert participation depends on the number of compositional factors that must be combined (Zhao et al., 2024).

4. Optimization, emergence, and architectural bias

Sparse compositional structure is not assumed to emerge automatically from gradient descent. A recent study argues that compositionality appears only in a narrow depth–connectivity regime. Along the connectivity axis, compositionality arises only in some specifically sparse networks and depends heavily on which connections remain rather than on weight sparsity alone. Along the depth axis, compositionality peaks at a target-dependent intermediate depth; both shallower and deeper networks fail, and training converges to fractured entangled representations rather than compositional circuits when the regime is violated (Do et al., 18 Jun 2026).

That work formalizes the rarity of such solutions by defining the compositional volume ratio

$U_i \subset \{1,\dots,p\}$9

the proportion of near-optimal parameter space occupied by compositional solutions. The paper states width-dilution and depth non-monotonicity results: for fixed depth, the compositional volume ratio vanishes as width grows, and for a target requiring $g_i$0 primitives, compositional solutions are maximized near the minimum depth needed for a balanced binary composition tree (Do et al., 18 Jun 2026).

To recover favorable connectivity, the same study introduces Similarity-based Pruning, which prunes redundant neurons using cosine similarity of feature maps,

$g_i$1

and a heuristic depth predictor based on PNG compression ratio. The practical claim is that the correct sparse wiring must be discovered or enforced; generic pruning with the same parameter count does not recover compositional structure (Do et al., 18 Jun 2026).

Other models embed this bias directly into the learning procedure. Vector Networks separate fast recombination from slow storage by inferring a sparse code for the current input and then updating only the selected atoms. Homological Neural Networks decouple topology discovery from weight learning by fixing the sparse graph before end-to-end optimization. AC-SINDy applies sparsity to edges of the computational graph rather than to coefficients in an oversized basis library (Pokel et al., 27 May 2026, Lin et al., 14 May 2026, Racioppo, 20 Apr 2026).

The optimization consequences are not uniform across architectures. In sparse MoE models, the optimal sparsity level is explicitly intermediate rather than minimal or maximal, because approximation error decreases as more experts are activated while estimation error increases with effective model complexity. The theory in that paper frames the design problem as balancing these two terms, so that expert activation should scale with task complexity, training-set size, and model complexity (Zhao et al., 2024).

5. Empirical evidence across domains

Empirical work consistently evaluates sparse compositional function models on tasks where training and test distributions differ by novel recombinations of familiar primitives.

For Vector Networks, the reported benchmarks are spatial decoding, 1D function composition, n-body force composition, and compositional MNIST. The central pattern is that VN matches strong baselines in-distribution while often achieving out-of-distribution error about an order of magnitude lower when familiar factors must be recombined in novel ways. In n-body dynamics, the paper states that the 3-layer VN remains consistently low across OOD settings and outperforms MLP, GNN, Transformer, Mamba, FiLM, and a factorized control; in the body-dropout analysis, VN is the only model that stays in the low-OOD region as the number of bodies decreases (Pokel et al., 27 May 2026).

For AC-SINDy, experiments emphasize recovery of interpretable governing equations and favorable scaling. On a 2D nonlinear system with true dynamics

$g_i$2

the recovered model is

$g_i$3

On noisy data, the latent filtering plus multi-step formulation recovers smooth dynamics and dominant terms, and the paper argues that explicit polynomial libraries scale like $g_i$4 while AC-SINDy uses approximately $g_i$5 parameters for $g_i$6 active compositional terms (Racioppo, 20 Apr 2026).

For Homological Neural Networks, synthetic tasks with known sparse hierarchies provide the clearest quantitative evidence. In the data-rich regime $g_i$7, the mean ranks reported are: HNN (oracle) $g_i$8, HNN (m-s) $g_i$9, HNN (marginal) $h$0, HNN (rand oracle) $h$1, and MLP-HNN $h$2. On the OpenML-CTR23 real-world tabular regression benchmark, the median-split HNN achieves mean rank $h$3, the marginal HNN $h$4, XGBoost $h$5, and Random Forest $h$6. The paper also states that HNNs are often one to two orders of magnitude sparser than dense MLPs while matching or outperforming them on average (Lin et al., 14 May 2026).

For the Sparse Coding Transformer, the principal benchmarks are S-RAVEN and RAVEN. On S-RAVEN, with 8 layers and 40M training tasks, the reported accuracies are $h$7 for the Transformer, $h$8 for HYLA, and $h$9 for the sparse-coding model. On RAVEN, the paper states that at high PSNR thresholds the baseline collapses to nearly $v^{(\ell)}$0 of samples above PSNR $v^{(\ell)}$1, while the proposed method retains around $v^{(\ell)}$2–$v^{(\ell)}$3 high-quality samples; the coefficient-transfer mechanism further improves structure qualitatively (Chen et al., 25 Nov 2025).

For Sparse Mixture-of-Experts, evidence comes from SRAVEN and SKILL-MIX. On SRAVEN, Top-1 routing performs worst overall on OOD accuracy, Top-2 is competitive only on easier tasks, and the best routing choice shifts upward in $v^{(\ell)}$4 from about $v^{(\ell)}$5 onward as task difficulty increases. On SKILL-MIX, Top-2 works very well for $v^{(\ell)}$6 and $v^{(\ell)}$7, but for harder tasks such as $v^{(\ell)}$8, using $v^{(\ell)}$9 or $$g_{v^{(\ell)}}:\mathbb{R}^{d_{\mathrm{in}}(v^{(\ell)})}\to\mathbb{R},$$0 experts per token gives the best results. The empirical message is that the optimal expert count grows with compositional difficulty but that overly dense activation can hurt easy tasks (Zhao et al., 2024).

These studies do not establish a single universal mechanism, but they do converge on a common empirical motif: when the held-out distribution is defined by novel recombination rather than by covariate drift alone, models that explicitly select, compose, or wire sparse reusable components tend to degrade more slowly than dense baselines (Pokel et al., 27 May 2026, Lin et al., 14 May 2026, Chen et al., 25 Nov 2025).

6. Conceptual boundaries, misconceptions, and open questions

A recurring misconception is that sparse compositionality is equivalent to ordinary sparsity. The recent literature explicitly rejects this. Sparse compositionality concerns sparse dependency structure, bounded local arity, or sparse operator reuse, whereas standard sparsity concerns a flat support pattern in one representation. This distinction matters because several papers report that sparsity alone is insufficient: randomized sparse wiring underperforms oracle wiring in HNNs, sparsity-matched autoencoders underperform Vector Networks, and generic pruning does not recover compositional circuits (Lin et al., 14 May 2026, Pokel et al., 27 May 2026, Do et al., 18 Jun 2026).

A second misconception is that if a sparse compositional representation exists, gradient descent will reliably discover it. The theoretical and empirical record is more cautious. The position paper argues that approximation is comparatively well understood, whereas optimization and learnability remain open. It further notes that, assuming one-way functions exist, there are polynomial-size Boolean circuit classes that are not efficiently learnable in the distribution-free setting by any polynomial-time algorithm. At the same time, many important subclasses remain learnable, including sparse Boolean polynomials and Fourier-sparse set functions (Danhofer et al., 3 Jul 2025).

A third issue is that “compositional” has multiple domain-specific meanings. In compositional data analysis, for example, the predictors lie on the simplex and require log-ratio geometry. A sparse network lasso for compositional covariates uses the symmetric log-contrast model with the zero-sum constraint

$$g_{v^{(\ell)}}:\mathbb{R}^{d_{\mathrm{in}}(v^{(\ell)})}\to\mathbb{R},$$1

together with graph-based clustering of per-sample coefficients. This is a distinct use of “compositional” from DAG-based sparse compositional functions, although both settings employ sparsity and structured dependence (Okazaki et al., 2021).

Current limitations are also explicit. The Sparse Coding Transformer has been tested primarily on relatively small-scale Transformer settings, and integration into large pre-trained models has not yet been explored. The same paper states that broader generative benchmarks remain unexamined (Chen et al., 25 Nov 2025). For architecture-design methods based on inferred sparse structure, the quality of the inferred graph is a practical bottleneck: in the compositional-data network-lasso setting, performance deteriorates as graph quality decreases, and when the graph becomes noisy, sparse network lasso without the compositional zero-sum constraint can outperform the proposed method (Okazaki et al., 2021).

The broader theoretical frontier concerns discovery rather than representation. Efficient Turing computability, approximation theory, and norm-based excess-risk bounds all support the idea that sparse compositional structure is a favorable hypothesis class (Poggio, 13 Oct 2025, Huang et al., 25 May 2026). What remains unresolved is when standard training procedures can identify the correct sparse computation graph from finite data, and when explicit architectural constraints, pruning rules, or local inference procedures are required. This suggests that future progress will likely depend less on increasing dense capacity alone than on improving mechanisms for discovering, enforcing, and reusing the right sparse hierarchy of constituent computations.