Iterative Neural Similarity Deflation in RNNs

• Iterative Neural Similarity Deflation (INSD) is a training method for RNNs that overcomes simplicity bias by penalizing linear predictivity from previous solutions.
• It projects recurrent activity into the readout null-space to preserve essential output directions while encouraging exploration of distinct dynamical motifs.
• INSD uncovers alternative solutions, such as rotational and oscillatory subspaces, which can improve task generalization and robustness under challenging conditions.

Searching arXiv for the specified paper and closely related deflation work. Iterative Neural Similarity Deflation (INSD) is a training procedure for recurrent neural networks (RNNs) designed to discover alternative task-solving mechanisms beyond the simplicity bias commonly observed in standard task-trained RNNs. In the formulation introduced in “Discovering alternative solutions beyond the simplicity bias in recurrent neural networks” (Qian et al., 25 Sep 2025), INSD augments the ordinary task loss with a penalty on the linear predictivity of neural activity relative to previously trained networks, while projecting activity into the readout null-space so that necessary output-potent directions are not penalized. The method was proposed in the context of neuroscience-style tasks, where standard RNNs trained on the same task often collapse onto effectively the same solution, typically involving fixed-point attractors or other low-dimensional dynamical motifs; INSD is intended to force exploration of qualitatively distinct solutions that remain functionally valid and can, in some regimes, generalize better under difficult or out-of-distribution conditions (Qian et al., 25 Sep 2025).

1. Origin and problem setting

Training RNNs on neuroscience-style tasks has become a common strategy for generating hypotheses about how neural circuits might implement computation. The motivating observation for INSD is that task-trained RNNs possess a strong simplicity bias: when trained repeatedly on the same task, they often converge to effectively the same solution, typically composed of fixed-point attractors or other low-dimensional dynamical motifs (Qian et al., 25 Sep 2025). These solutions are readily interpretable, but the resulting collapse is counterproductive when the objective is to generate genuinely distinct mechanistic hypotheses for neural computation.

INSD was introduced to break this inductive bias by penalizing linear predictivity of neural activity produced by standard task-trained RNNs (Qian et al., 25 Sep 2025). The stated goal is not merely to perturb trained models, but to obtain an alternative class of solutions that remain successful on the task while differing under multiple analytical lenses, including representational similarity metrics, dynamical systems analysis, and the linear decodability of task-relevant variables (Qian et al., 25 Sep 2025).

A broader context for the terminology of “deflation” appears in earlier work on neural-network-based solution finding for nonlinear differential equations. “Structure Probing Neural Network Deflation” (Gu et al., 2020) introduces deflation operators that make already discovered solutions no longer local minimizers of the optimization energy landscape, thereby enabling discovery of multiple solutions. That work addresses PDE and BVP settings rather than recurrent neural computations, but it provides a useful antecedent for the general idea of iteratively excluding previously found solutions from subsequent searches (Gu et al., 2020). This suggests a family resemblance between classical deflation ideas and the later RNN-specific similarity-deflation construction, even though the two papers operate on different objects and loss landscapes.

2. Loss construction and similarity penalty

The modified optimization objective in INSD begins with the usual mean-squared-error task loss over one trial of duration $T$: $$L_{\rm task} \;=\;\frac{1}{T}\!\int_{0}^{T}\Big\|y(t)\;-\;y^{\star}(t)\Big\|^{2}\,dt \quad\text{with}\quad y(t)=W^{\rm out}r(t)\,,$$ where $r(t)\in\mathbb R^N$ are the network firing rates and $y^\star(t)$ the target outputs (Qian et al., 25 Sep 2025).

To discover new solutions, the $i$th network is trained with an additional deflation term that discourages reuse of any linearly predictable components of all previously found networks. If

$$R_i\;=\;\bigl[r_i(t_1);\;r_i(t_2);\;\dots\bigr]\in\mathbb R^{P\times N}$$

denotes the batch-times-time unfolded rates of network $i$, then the activity used for deflation is

$R_i^\perp \;=\;P_{\ker(W_i^{\rm out})}\,R_i\,,$

namely the rates projected into the readout null-space, so that output-potent directions are excluded from the penalty (Qian et al., 25 Sep 2025).

The total INSD loss for network $i$ is

$$L_i \;=\; L_{\rm task} \;+\;\lambda\sum_{j=1}^{i-1}S\bigl(R_i^\perp,\;R_j^\perp\bigr),$$

with $$L_{\rm task} \;=\;\frac{1}{T}\!\int_{0}^{T}\Big\|y(t)\;-\;y^{\star}(t)\Big\|^{2}\,dt \quad\text{with}\quad y(t)=W^{\rm out}r(t)\,,$$0 controlling how aggressively earlier solutions are deflated (Qian et al., 25 Sep 2025). The similarity measure $$L_{\rm task} \;=\;\frac{1}{T}\!\int_{0}^{T}\Big\|y(t)\;-\;y^{\star}(t)\Big\|^{2}\,dt \quad\text{with}\quad y(t)=W^{\rm out}r(t)\,,$$1 is the fraction of variance of $$L_{\rm task} \;=\;\frac{1}{T}\!\int_{0}^{T}\Big\|y(t)\;-\;y^{\star}(t)\Big\|^{2}\,dt \quad\text{with}\quad y(t)=W^{\rm out}r(t)\,,$$2 explained by a linear regression from $$L_{\rm task} \;=\;\frac{1}{T}\!\int_{0}^{T}\Big\|y(t)\;-\;y^{\star}(t)\Big\|^{2}\,dt \quad\text{with}\quad y(t)=W^{\rm out}r(t)\,,$$3. The construction is given through

$$L_{\rm task} \;=\;\frac{1}{T}\!\int_{0}^{T}\Big\|y(t)\;-\;y^{\star}(t)\Big\|^{2}\,dt \quad\text{with}\quad y(t)=W^{\rm out}r(t)\,,$$4

where

$$L_{\rm task} \;=\;\frac{1}{T}\!\int_{0}^{T}\Big\|y(t)\;-\;y^{\star}(t)\Big\|^{2}\,dt \quad\text{with}\quad y(t)=W^{\rm out}r(t)\,,$$5

uses a small ridge $$L_{\rm task} \;=\;\frac{1}{T}\!\int_{0}^{T}\Big\|y(t)\;-\;y^{\star}(t)\Big\|^{2}\,dt \quad\text{with}\quad y(t)=W^{\rm out}r(t)\,,$$6 for numerical stability (Qian et al., 25 Sep 2025). Accordingly,

$$L_{\rm task} \;=\;\frac{1}{T}\!\int_{0}^{T}\Big\|y(t)\;-\;y^{\star}(t)\Big\|^{2}\,dt \quad\text{with}\quad y(t)=W^{\rm out}r(t)\,,$$7

The crucial design choice is that the penalty is based on linear predictivity rather than identity of weights or direct activity matching. This makes the deflation criterion representation-centered and subspace-sensitive. Because it is imposed only in the readout null-space, it does not directly punish the activity components needed to produce the task output (Qian et al., 25 Sep 2025).

3. Iterative training procedure

INSD is defined as an iterative procedure over a sequence of networks. The first network serves as a reference model trained only on the task loss: $i$9 At each stage, the new network is explicitly discouraged from re-employing any linear subspace already used by its predecessors (Qian et al., 25 Sep 2025).

The paper characterizes the intuition for this procedure by analogy to Gram–Schmidt: each new solution must be linearly orthogonal to previous ones (Qian et al., 25 Sep 2025). Because standard trained RNNs tend to collapse onto low-dimensional attractor motifs such as fixed-point lines and ring attractors, penalizing any component that can predict prior activity is described as forcing the network to explore qualitatively different dynamical motifs, often including rotational or oscillatory subspaces (Qian et al., 25 Sep 2025). The paper further states that this pushes optimization out of the “simplicity basin” whose top principal components are shared by standard solutions, thereby revealing alternative computational ansätze (Qian et al., 25 Sep 2025).

The earlier deflation framework in (Gu et al., 2020) shares the same iterative exclusion logic, but with a different mechanism. There, known solutions $$L_{\rm task} \;=\;\frac{1}{T}\!\int_{0}^{T}\Big\|y(t)\;-\;y^{\star}(t)\Big\|^{2}\,dt \quad\text{with}\quad y(t)=W^{\rm out}r(t)\,,$$8 are removed from future optimization by multiplying the original loss $$L_{\rm task} \;=\;\frac{1}{T}\!\int_{0}^{T}\Big\|y(t)\;-\;y^{\star}(t)\Big\|^{2}\,dt \quad\text{with}\quad y(t)=W^{\rm out}r(t)\,,$$9 by a deflation factor such as

$r(t)\in\mathbb R^N$0

or, equivalently, a product-form alternative (Gu et al., 2020). As $r(t)\in\mathbb R^N$1, the deflation factor diverges, producing a barrier around known solutions. INSD does not use that singular loss geometry; instead, it penalizes cross-network linear predictivity in neural state space (Qian et al., 25 Sep 2025). This suggests that “deflation” in INSD is functional rather than singular: it removes already-used representational structure rather than directly making previous parameter-space solutions inaccessible.

4. Experimental implementation

The experimental study in (Qian et al., 25 Sep 2025) evaluates INSD on three neuroscience-style tasks.

Context-Dependent Integration (“Mante”-style) consists of two noisy streams, one of which is cued for integration. 3-Bit Flip-Flop requires discrete memory of the last pulse sign on three channels. MemoryPro requires encoding an angle on a ring, maintaining it through a delay, and then reporting it (Qian et al., 25 Sep 2025).

All experiments use a rate-based RNN with $r(t)\in\mathbb R^N$2 units and $r(t)\in\mathbb R^N$3 (Qian et al., 25 Sep 2025). Recurrent initialization is

$r(t)\in\mathbb R^N$4

with $r(t)\in\mathbb R^N$5 swept in $r(t)\in\mathbb R^N$6 for standard networks, whereas INSD networks use $r(t)\in\mathbb R^N$7 (Qian et al., 25 Sep 2025). Optimization uses Adam with learning rate $r(t)\in\mathbb R^N$8, weight decay $r(t)\in\mathbb R^N$9, batch size $y^\star(t)$0, and $y^\star(t)$1 iterations or until convergence. The reported deflation hyperparameters are $y^\star(t)$2 and $y^\star(t)$3 (Qian et al., 25 Sep 2025).

The paper organizes the comparison between standard and INSD-trained networks around several analytical families: representational similarity, dynamical systems analysis, and linear decodability (Qian et al., 25 Sep 2025). This multi-view evaluation matters because the motivating claim is not merely that alternative solutions exist, but that they are substantively different in geometry, dynamics, and task-variable accessibility.

For historical comparison, (Gu et al., 2020) uses a different computational setting: fully-connected feed-forward networks of depth $y^\star(t)$4, uniform width $y^\star(t)$5, activation $y^\star(t)$6, exact boundary-condition enforcement via lifting factors, and Adam over collocation-based PDE residual losses. Those details belong to neural-network deflation for nonlinear differential equations rather than RNN dynamics, but they illustrate that deflation-based search procedures have been implemented in distinct neural modeling regimes (Gu et al., 2020).

5. Empirical behavior and discovered solution classes

The principal empirical claim of (Qian et al., 25 Sep 2025) is that standard RNNs exhibit strong collapse, whereas INSD produces solutions that appear distinct under several independent analyses.

Representational similarity

Standard RNNs, across random seeds and $y^\star(t)$7, exhibit pairwise $y^\star(t)$8; they collapse to the same line attractor, ring attractor, or fixed-point cube (Qian et al., 25 Sep 2025). In contrast, INSD solutions labeled alt-1 and alt-2 have low linear predictivity with respect to standard networks after projection to the readout null-space, often $y^\star(t)$9 (Qian et al., 25 Sep 2025). Dynamical Similarity Analysis embeddings place INSD solutions far from the cluster of standard networks (Qian et al., 25 Sep 2025).

Dynamical systems analysis

In context-dependent integration, the standard solution consists of two context-specific line attractors, described as marginally stable fixed-point lines (Qian et al., 25 Sep 2025). The corresponding alt-1 solution has no slow manifold; instead, it exhibits a web of unstable fixed points with oscillatory eigenmodes, and memory is carried by rotations (Qian et al., 25 Sep 2025).

In the 3-Bit Flip-Flop task, the standard solution is a cube of eight stable attractors with edge-aligned saddles (Qian et al., 25 Sep 2025). The alt-1 solution has no stable corners, but instead unstable fixed points plus rotational modes (Qian et al., 25 Sep 2025).

In MemoryPro, the standard solution is a ring attractor, whereas the alt-1 solution is characterized by an unstable center together with quasi-periodic rotation preserving angular order (Qian et al., 25 Sep 2025).

These comparisons are central because they indicate that INSD does not simply induce superficial decorrelation. The resulting models can shift from attractor-based storage to rotation-based or quasi-periodic mechanisms while continuing to solve the original task (Qian et al., 25 Sep 2025).

6. Decodability, robustness, and broader implications

The paper reports marked differences in linear decodability between standard and deflated solutions. In standard solutions, task variables such as context, stimulus coherence, and angle are linearly decodable at $i$0 (Qian et al., 25 Sep 2025). In alt-1 solutions, linear decoding of the irrelevant input stream collapses, context often requires RBF or higher-order features, and during memory the instantaneous linear readout vector itself rotates (Qian et al., 25 Sep 2025). This indicates that alternative solutions can encode task structure in a way that is less compatible with static linear readouts, even when output behavior remains correct.

INSD is also associated with improved out-of-distribution behavior in some regimes. For context-dependent integration under higher noise or longer trials, alt-2 sometimes outperforms the standard ensemble by $i$1 MSE improvement (Qian et al., 25 Sep 2025). In the flip-flop task at higher noise, alt-2 yields modest gains in classification accuracy. In MemoryPro at extreme memory loads or noise, alt-1 generalizes better, maintaining correct recall when standard networks fail (Qian et al., 25 Sep 2025).

The paper frames several practical implications. INSD provides a simple task-agnostic lever, namely $i$2, for harvesting multiple qualitatively distinct dynamical hypotheses from the same task (Qian et al., 25 Sep 2025). These alternatives can be tested directly against neural data to determine whether an attractor-based or oscillation-based motif better matches physiology (Qian et al., 25 Sep 2025). The paper also suggests that mixed models, combining standard and INSD networks, might unite the robustness benefits of diverse dynamics with the interpretability of low-dimensional attractors (Qian et al., 25 Sep 2025). Finally, it proposes broader applicability wherever solution degeneracy is a concern, including policy collapse, fairness constraints, and continual learning (Qian et al., 25 Sep 2025).

A plausible implication is that INSD shifts the role of task-trained RNNs in computational neuroscience: instead of producing a single interpretable mechanism favored by optimization bias, they can be used to sample a wider hypothesis space of mechanistically distinct yet task-consistent circuit models. That implication follows from the reported ability to uncover richly varied, functionally valid, dynamically distinct solutions by augmenting the task loss with the linear predictivity penalty

$i$3

(Qian et al., 25 Sep 2025).

7. Relation to prior deflation methods and conceptual boundaries

INSD belongs to a broader tradition of iterative deflation methods that seek multiple solutions by modifying the optimization problem after each success. In (Gu et al., 2020), deflation is applied to nonlinear PDEs and BVPs by multiplying the loss with a factor that diverges at known solutions, while a separate structure-probing initialization can bias the search toward oscillatory or radial branches. That framework is reported to recover multiple solution families, including 14 distinct solutions for a 2D Yamabe problem, 11 for a $i$4 high-dimensional Yamabe problem, 9 for $i$5, and over 100 distinct $i$6 pairs for a 3D reaction–diffusion system on an irregular domain, with residuals below $i$7 (Gu et al., 2020). The stated rationale is that singularity at known solutions repels gradient-based optimization from those minima while preserving unknown zeros of the original loss (Gu et al., 2020).

The conceptual distinction is therefore sharp. In the earlier PDE setting, deflation is defined at the level of already discovered function-space solutions and their $i$8 neighborhoods (Gu et al., 2020). In INSD, deflation is defined at the level of neural representations, specifically the readout-null projected recurrent activity and its linear predictivity relative to prior networks (Qian et al., 25 Sep 2025). One formulation changes the loss landscape by erecting barriers around known solutions; the other changes it by discouraging reuse of previously occupied representational subspaces.

A common misconception would be to treat INSD as merely a generic diversity regularizer. The formulation in (Qian et al., 25 Sep 2025) is more specific: it penalizes the fraction of variance of prior null-space activity explained by a linear regression from the current network’s null-space activity, rather than enforcing generic parameter diversity or output diversity. Another potential misunderstanding would be to assume that INSD seeks arbitrary difference at any cost. The projection into the readout null-space makes clear that the method is intended to preserve necessary output-potent directions while deflating only those components not required for the task output (Qian et al., 25 Sep 2025).

Taken together, these features place INSD at the intersection of representational analysis, dynamical systems theory, and iterative multi-solution optimization. Its defining contribution is to operationalize “alternative solution” not as a different parameterization of the same latent mechanism, but as a network whose internal activity is not linearly predictive of earlier solutions once output-potent structure has been factored out (Qian et al., 25 Sep 2025).