Fixed Parameter Expansion (FPE) in Neural Networks

• Fixed Parameter Expansion (FPE) is a neural network method that replaces a neuron with multiple children by partitioning its weights disjointly, keeping the non-zero parameter count fixed.
• FPE reduces feature interference and polysemanticity by separating overlapping connections, thereby enhancing accuracy in high-superposition scenarios.
• Empirical observations suggest that FPE improves performance on both symbolic tasks and real-world models by reallocating fixed parameters for better representational organization.

Searching arXiv for the target paper and directly relevant context papers on superposition/polysemanticity.

arXiv search: (Kong et al., 6 Oct 2025)

Fixed Parameter Expansion (FPE) is a neural-network transformation introduced in "Expand Neurons, Not Parameters" (Kong et al., 6 Oct 2025) for increasing network width without increasing the number of non-zero parameters. The method replaces a neuron with multiple children and partitions the parent’s weights disjointly across them, so that each child inherits a non-overlapping subset of connections. In the formulation available from the abstract, FPE is motivated by interference reduction: performance gains are linked to a decrease in collisions among multiple features that would otherwise share the same neurons, and the method is presented as an interpretability-grounded way to use width against superposition under a fixed non-zero parameter budget.

1. Definition and construction

FPE is defined by a local replacement rule. A single neuron is replaced by multiple child neurons, and the original neuron’s weights are partitioned disjointly across those children, with each child receiving a non-overlapping subset of the parent’s connections (Kong et al., 6 Oct 2025). The defining constraint is that the network’s number of non-zero parameters does not increase, even though the number of neurons does.

This makes FPE distinct from ordinary widening. Standard widening typically increases width by adding additional parameters. FPE instead widens the network while holding fixed the count of non-zero parameters. The construction therefore targets representational organization rather than raw parameter growth. A plausible implication is that FPE belongs to a broader class of width reallocations in which capacity is altered structurally, not by increasing sparsity-adjusted model size.

2. Mechanistic rationale: interference, entanglement, and superposition

The abstract links FPE’s gains to a decrease in interference between multiple features that would otherwise share the same neurons (Kong et al., 6 Oct 2025). In that account, shared-neuron usage produces entanglement, and reducing this entanglement at fixed non-zero parameter count is the central purpose of the method. The reported interpretation is explicitly aligned with polysemanticity and the superposition hypothesis.

Within this framing, FPE is not merely a parameter-preserving network surgery. It is an intervention on feature allocation. By partitioning a parent neuron’s incident weights into disjoint subsets assigned to separate children, FPE reduces collisions among features that would otherwise be co-localized in the same unit. This suggests that the operative variable is not width in isolation, but width as a mechanism for separating features under sparse connectivity constraints.

The abstract further states that the benefits of FPE grow with increasing interference and that, when polysemantic load is high, accuracy improvements are the largest. That claim places interference at the center of the proposed explanatory picture. It also indicates that FPE is expected to be most effective in representational regimes where superposition pressure is already substantial.

3. Reported empirical behavior

The reported empirical evidence covers both symbolic tasks and non-symbolic models (Kong et al., 6 Oct 2025). On symbolic tasks, specifically Boolean code problems, clause-aligned FPE is said to systematically reduce polysemanticity metrics and yield higher task accuracy. The abstract also notes that random splits of neuron weights approximate these gains.

That latter observation is important because it constrains the claimed mechanism. The abstract interprets the result as indicating that reduced collisions, not precise assignment, are a primary driver. In other words, the improvement is not presented as depending exclusively on a highly specialized decomposition of the parent neuron. Random partitioning already approximates the effect, which is consistent with a collision-reduction explanation rather than one requiring exact semantic alignment of every child.

The same abstract transfers the argument to real models. It reports experiments on classifiers over CLIP embeddings and on deeper multilayer networks, and states that widening networks while maintaining a constant non-zero parameter count consistently increases accuracy. The article-length details needed to specify architectures, optimization procedures, or metric definitions are not available in the supplied record, but the abstract-level claim is unambiguous: the purported benefit is not restricted to synthetic symbolic settings.

4. Architectural and systems significance

In the available formulation, FPE is presented as a way to leverage width against superposition without increasing the number of non-zero parameters (Kong et al., 6 Oct 2025). This gives the method a dual interpretation. Architecturally, it is a width reallocation strategy. Computationally, it is a sparsity-aware design principle: the relevant resource is not total dense parameter count, but the movement and use of non-zero weights.

The abstract explicitly connects this to hardware considerations. It states that the direction is well matched to modern accelerators, where memory movement of non-zero parameters, rather than raw compute, is the dominant bottleneck. This situates FPE within a systems regime in which parameter traffic can be more important than nominal arithmetic cost. A plausible implication is that FPE is especially attractive when widening can be accommodated by accelerator-friendly execution while avoiding growth in the memory footprint associated with non-zero weights.

The same framing also clarifies what the method does not claim. It is not described as a free increase in dense expressive capacity. Rather, it is presented as a way to reorganize a fixed non-zero parameter budget so that features interfere less and accuracy improves under that same budget.

5. Relation to adjacent terminology and common confusions

The acronym FPE is heavily overloaded in the arXiv literature. In stochastic-process and statistical-physics work it commonly denotes the Fokker–Planck equation (Bianucci et al., 2020). In time-series model selection it denotes Akaike’s Final Prediction Error (Jirak et al., 2024). In robust covariance estimation it denotes Tyler’s Fixed-Point Estimator (Pascal et al., 2013). Fixed Parameter Expansion is therefore a distinct usage and should not be conflated with those established meanings.

A second common confusion concerns width. FPE does not mean arbitrary network expansion. Its defining feature is expansion at a fixed non-zero parameter count. A third confusion concerns assignment specificity. The abstract does not present clause alignment as the sole operative ingredient; it explicitly reports that random splits approximate the gains, which shifts emphasis from exact semantic routing to collision reduction.

6. Documentary status and evidentiary limits

The supplied record for "Expand Neurons, Not Parameters" states that the provided paper content is actually an ICLR formatting template and does not contain the scientific exposition, experiments, or results section needed for a full technical reconstruction (Kong et al., 6 Oct 2025). The same record specifies that the available content does not include parameter-count formulas, the neuron-expansion construction in full detail, the weight-partitioning rule beyond the abstract statement, the clause-aligned versus random split description, the theoretical argument relating FPE to polysemanticity or superposition, experiment sections, tables, figures, numerical results, discussion of CLIP embedding classifiers or deeper multilayer networks, or accelerator and memory-movement analysis.

Accordingly, the concept can presently be characterized with confidence only at the level stated in the abstract: FPE replaces a neuron by multiple children, partitions the parent’s weights disjointly across them, preserves the number of non-zero parameters, and is claimed to improve performance by reducing interference and polysemanticity, with larger gains under higher polysemantic load. More detailed encyclopedic treatment of its exact algorithms, proofs, ablations, and quantitative effect sizes awaits access to the complete paper text.