---
title: Iterative Neural Similarity Deflation in RNNs
url: https://www.emergentmind.com/topics/iterative-neural-similarity-deflation-insd
type: topic
---

# Iterative Neural Similarity Deflation in RNNs

Searching arXiv for the specified paper and closely related deflation work.
Iterative Neural Similarity Deflation (INSD) is a training procedure for recurrent neural networks (RNNs) designed to discover alternative task-solving mechanisms beyond the simplicity bias commonly observed in standard task-trained RNNs. In the formulation introduced in “Discovering alternative solutions beyond the simplicity bias in recurrent neural networks” [2509.21504], INSD augments the ordinary task loss with a penalty on the linear predictivity of neural activity relative to previously trained networks, while projecting activity into the readout null-space so that necessary output-potent directions are not penalized. The method was proposed in the context of neuroscience-style tasks, where standard RNNs trained on the same task often collapse onto effectively the same solution, typically involving fixed-point attractors or other low-dimensional dynamical motifs; INSD is intended to force exploration of qualitatively distinct solutions that remain functionally valid and can, in some regimes, generalize better under difficult or out-of-distribution conditions [2509.21504].

## 1. Origin and problem setting

Training RNNs on neuroscience-style tasks has become a common strategy for generating hypotheses about how neural circuits might implement computation. The motivating observation for INSD is that task-trained RNNs possess a strong simplicity bias: when trained repeatedly on the same task, they often converge to effectively the same solution, typically composed of fixed-point attractors or other low-dimensional dynamical motifs [2509.21504]. These solutions are readily interpretable, but the resulting collapse is counterproductive when the objective is to generate genuinely distinct mechanistic hypotheses for neural computation.

INSD was introduced to break this inductive bias by penalizing linear predictivity of neural activity produced by standard task-trained RNNs [2509.21504]. The stated goal is not merely to perturb trained models, but to obtain an alternative class of solutions that remain successful on the task while differing under multiple analytical lenses, including representational similarity metrics, dynamical systems analysis, and the linear decodability of task-relevant variables [2509.21504].

A broader context for the terminology of “deflation” appears in earlier work on neural-network-based solution finding for nonlinear differential equations. “Structure Probing Neural Network Deflation” [2007.03609] introduces deflation operators that make already discovered solutions no longer local minimizers of the optimization energy landscape, thereby enabling discovery of multiple solutions. That work addresses PDE and BVP settings rather than recurrent neural computations, but it provides a useful antecedent for the general idea of iteratively excluding previously found solutions from subsequent searches [2007.03609]. This suggests a family resemblance between classical deflation ideas and the later RNN-specific similarity-deflation construction, even though the two papers operate on different objects and loss landscapes.

## 2. Loss construction and similarity penalty

The modified optimization objective in INSD begins with the usual mean-squared-error task loss over one trial of duration \(T\):
\[
L_{\rm task} \;=\;\frac{1}{T}\!\int_{0}^{T}\Big\|y(t)\;-\;y^{\star}(t)\Big\|^{2}\,dt
\quad\text{with}\quad
y(t)=W^{\rm out}r(t)\,,
\]
where \(r(t)\in\mathbb R^N\) are the network firing rates and \(y^\star(t)\) the target outputs [2509.21504].

To discover new solutions, the \(i\)th network is trained with an additional deflation term that discourages reuse of any linearly predictable components of all previously found networks. If
\[
R_i\;=\;\bigl[r_i(t_1);\;r_i(t_2);\;\dots\bigr]\in\mathbb R^{P\times N}
\]
denotes the batch-times-time unfolded rates of network \(i\), then the activity used for deflation is
\[
R_i^\perp \;=\;P_{\ker(W_i^{\rm out})}\,R_i\,,
\]
namely the rates projected into the readout null-space, so that output-potent directions are excluded from the penalty [2509.21504].

The total INSD loss for network \(i\) is
\[
L_i \;=\; L_{\rm task} \;+\;\lambda\sum_{j=1}^{i-1}S\bigl(R_i^\perp,\;R_j^\perp\bigr),
\]
with \(\lambda>0\) controlling how aggressively earlier solutions are deflated [2509.21504]. The similarity measure \(S\) is the fraction of variance of \(R_j^\perp\) explained by a linear regression from \(R_i^\perp\). The construction is given through
\[
r^2\bigl(X,Y\bigr)
\;=\;
1\;-\;\min_{W\in\mathbb R^{N\times N}}
\frac{\|XW - Y\|_F^2}{\|Y\|_F^2}
\;=\;
\frac{\|Y_{X}\|_F^2}{\|Y\|_F^2}\,,
\]
where
\[
Y_X = X\,(X^T X + \rho I)^{-1}X^T\,Y
\]
uses a small ridge \(\rho>0\) for numerical stability [2509.21504]. Accordingly,
\[
S\bigl(R_i^\perp,R_j^\perp\bigr)
\;=\;
\frac{\bigl\|\,\bigl(R_j^\perp\bigr)_{R_i^\perp}\bigr\|_F^2} {\|R_j^\perp\|_F^2}\,.
\]

The crucial design choice is that the penalty is based on linear predictivity rather than identity of weights or direct activity matching. This makes the deflation criterion representation-centered and subspace-sensitive. Because it is imposed only in the readout null-space, it does not directly punish the activity components needed to produce the task output [2509.21504].

## 3. Iterative training procedure

INSD is defined as an iterative procedure over a sequence of networks. The first network serves as a reference model trained only on the task loss:
```text
Inputs:
  Task loss L_task(Θ)
  Similarity weight λ, ridge ρ
  Number of solutions K

1. Train the first (reference) RNN:
   Θ₁ ← argmin_Θ L_task(Θ)
   Store rates R₁^⊥.

2. For i = 2 to K do
   Initialize Θᵢ (e.g. same g‐scale).
   Repeat until convergence:
     Sample minibatch of inputs → compute rates Rᵢ
     Project: Rᵢ^⊥ ← P_{ker(Wᵢ^out)} Rᵢ
     Deflation penalty:
       Pᵢ ← 0
       For j = 1,…,i−1 do
         Compute r² = S(Rᵢ^⊥, Rⱼ^⊥)
         Pᵢ ← Pᵢ + r²
       end
     Total loss:  Lᵢ = L_task + λ·Pᵢ
     Update Θᵢ ← Θᵢ − η∇_ΘᵢLᵢ
   end
   Store Rᵢ^⊥.
end
```
At each stage, the new network is explicitly discouraged from re-employing any linear subspace already used by its predecessors [2509.21504].

The paper characterizes the intuition for this procedure by analogy to Gram–Schmidt: each new solution must be linearly orthogonal to previous ones [2509.21504]. Because standard trained RNNs tend to collapse onto low-dimensional attractor motifs such as fixed-point lines and ring attractors, penalizing any component that can predict prior activity is described as forcing the network to explore qualitatively different dynamical motifs, often including rotational or oscillatory subspaces [2509.21504]. The paper further states that this pushes optimization out of the “simplicity basin” whose top principal components are shared by standard solutions, thereby revealing alternative computational ansätze [2509.21504].

The earlier deflation framework in [2007.03609] shares the same iterative exclusion logic, but with a different mechanism. There, known solutions \(u_1,\dots,u_k\) are removed from future optimization by multiplying the original loss \(L(u)\) by a deflation factor such as
\[
D\bigl(u;\{u_i\}_{i=1}^k\bigr)
\;=\;
\sum_{i=1}^k
\bigl\|\,u - u_i\|_{L^2(\Omega)}^{-\,p_i}
\;+\;\alpha
\]
or, equivalently, a product-form alternative [2007.03609]. As \(u\to u_i\), the deflation factor diverges, producing a barrier around known solutions. INSD does not use that singular loss geometry; instead, it penalizes cross-network linear predictivity in neural state space [2509.21504]. This suggests that “deflation” in INSD is functional rather than singular: it removes already-used representational structure rather than directly making previous parameter-space solutions inaccessible.

## 4. Experimental implementation

The experimental study in [2509.21504] evaluates INSD on three neuroscience-style tasks.

**Context-Dependent Integration (“Mante”-style)** consists of two noisy streams, one of which is cued for integration.  
**3-Bit Flip-Flop** requires discrete memory of the last pulse sign on three channels.  
**MemoryPro** requires encoding an angle on a ring, maintaining it through a delay, and then reporting it [2509.21504].

All experiments use a rate-based RNN with \(N=128\) units and \(\phi=\tanh\) [2509.21504]. Recurrent initialization is
\[
W_{ij}\sim\mathcal N(0,g^2/N),
\]
with \(g\) swept in \(\{0.01,0.5,1.0,1.5\}\) for standard networks, whereas INSD networks use \(g=1\) [2509.21504]. Optimization uses Adam with learning rate \(10^{-3}\), weight decay \(10^{-5}\), batch size \(32\), and \(10^6\) iterations or until convergence. The reported deflation hyperparameters are \(\lambda=0.05\) and \(\rho=10^{-3}\) [2509.21504].

The paper organizes the comparison between standard and INSD-trained networks around several analytical families: representational similarity, dynamical systems analysis, and linear decodability [2509.21504]. This multi-view evaluation matters because the motivating claim is not merely that alternative solutions exist, but that they are substantively different in geometry, dynamics, and task-variable accessibility.

For historical comparison, [2007.03609] uses a different computational setting: fully-connected feed-forward networks of depth \(L=3\), uniform width \(N=100\), activation \(\sigma(t)=\max(0,t^3)\), exact boundary-condition enforcement via lifting factors, and Adam over collocation-based PDE residual losses. Those details belong to neural-network deflation for nonlinear differential equations rather than RNN dynamics, but they illustrate that deflation-based search procedures have been implemented in distinct neural modeling regimes [2007.03609].

## 5. Empirical behavior and discovered solution classes

The principal empirical claim of [2509.21504] is that standard RNNs exhibit strong collapse, whereas INSD produces solutions that appear distinct under several independent analyses.

### Representational similarity

Standard RNNs, across random seeds and \(g\), exhibit pairwise \(r^2\approx1\); they collapse to the same line attractor, ring attractor, or fixed-point cube [2509.21504]. In contrast, INSD solutions labeled alt-1 and alt-2 have low linear predictivity with respect to standard networks after projection to the readout null-space, often \(r^2<0.2\) [2509.21504]. Dynamical Similarity Analysis embeddings place INSD solutions far from the cluster of standard networks [2509.21504].

### Dynamical systems analysis

In context-dependent integration, the standard solution consists of two context-specific line attractors, described as marginally stable fixed-point lines [2509.21504]. The corresponding alt-1 solution has no slow manifold; instead, it exhibits a web of unstable fixed points with oscillatory eigenmodes, and memory is carried by rotations [2509.21504].

In the 3-Bit Flip-Flop task, the standard solution is a cube of eight stable attractors with edge-aligned saddles [2509.21504]. The alt-1 solution has no stable corners, but instead unstable fixed points plus rotational modes [2509.21504].

In MemoryPro, the standard solution is a ring attractor, whereas the alt-1 solution is characterized by an unstable center together with quasi-periodic rotation preserving angular order [2509.21504].

These comparisons are central because they indicate that INSD does not simply induce superficial decorrelation. The resulting models can shift from attractor-based storage to rotation-based or quasi-periodic mechanisms while continuing to solve the original task [2509.21504].

## 6. Decodability, robustness, and broader implications

The paper reports marked differences in linear decodability between standard and deflated solutions. In standard solutions, task variables such as context, stimulus coherence, and angle are linearly decodable at \(>95\%\) [2509.21504]. In alt-1 solutions, linear decoding of the irrelevant input stream collapses, context often requires RBF or higher-order features, and during memory the instantaneous linear readout vector itself rotates [2509.21504]. This indicates that alternative solutions can encode task structure in a way that is less compatible with static linear readouts, even when output behavior remains correct.

INSD is also associated with improved out-of-distribution behavior in some regimes. For context-dependent integration under higher noise or longer trials, alt-2 sometimes outperforms the standard ensemble by \(5\!-\!10\%\) MSE improvement [2509.21504]. In the flip-flop task at higher noise, alt-2 yields modest gains in classification accuracy. In MemoryPro at extreme memory loads or noise, alt-1 generalizes better, maintaining correct recall when standard networks fail [2509.21504].

The paper frames several practical implications. INSD provides a simple task-agnostic lever, namely \(\lambda\), for harvesting multiple qualitatively distinct dynamical hypotheses from the same task [2509.21504]. These alternatives can be tested directly against neural data to determine whether an attractor-based or oscillation-based motif better matches physiology [2509.21504]. The paper also suggests that mixed models, combining standard and INSD networks, might unite the robustness benefits of diverse dynamics with the interpretability of low-dimensional attractors [2509.21504]. Finally, it proposes broader applicability wherever solution degeneracy is a concern, including policy collapse, fairness constraints, and continual learning [2509.21504].

A plausible implication is that INSD shifts the role of task-trained RNNs in computational neuroscience: instead of producing a single interpretable mechanism favored by optimization bias, they can be used to sample a wider hypothesis space of mechanistically distinct yet task-consistent circuit models. That implication follows from the reported ability to uncover richly varied, functionally valid, dynamically distinct solutions by augmenting the task loss with the linear predictivity penalty
\[
L \;=\; L_{\rm task}\;+\;\lambda\sum_{j< i}r^2\bigl(R_i^\perp,R_j^\perp\bigr)
\]
[2509.21504].

## 7. Relation to prior deflation methods and conceptual boundaries

INSD belongs to a broader tradition of iterative deflation methods that seek multiple solutions by modifying the optimization problem after each success. In [2007.03609], deflation is applied to nonlinear PDEs and BVPs by multiplying the loss with a factor that diverges at known solutions, while a separate structure-probing initialization can bias the search toward oscillatory or radial branches. That framework is reported to recover multiple solution families, including 14 distinct solutions for a 2D Yamabe problem, 11 for a \(d=3\) high-dimensional Yamabe problem, 9 for \(d=6\), and over 100 distinct \((u,v)\) pairs for a 3D reaction–diffusion system on an irregular domain, with residuals below \(10^{-3}\) [2007.03609]. The stated rationale is that singularity at known solutions repels gradient-based optimization from those minima while preserving unknown zeros of the original loss [2007.03609].

The conceptual distinction is therefore sharp. In the earlier PDE setting, deflation is defined at the level of already discovered function-space solutions and their \(L^2\) neighborhoods [2007.03609]. In INSD, deflation is defined at the level of neural representations, specifically the readout-null projected recurrent activity and its linear predictivity relative to prior networks [2509.21504]. One formulation changes the loss landscape by erecting barriers around known solutions; the other changes it by discouraging reuse of previously occupied representational subspaces.

A common misconception would be to treat INSD as merely a generic diversity regularizer. The formulation in [2509.21504] is more specific: it penalizes the fraction of variance of prior null-space activity explained by a linear regression from the current network’s null-space activity, rather than enforcing generic parameter diversity or output diversity. Another potential misunderstanding would be to assume that INSD seeks arbitrary difference at any cost. The projection into the readout null-space makes clear that the method is intended to preserve necessary output-potent directions while deflating only those components not required for the task output [2509.21504].

Taken together, these features place INSD at the intersection of representational analysis, dynamical systems theory, and iterative multi-solution optimization. Its defining contribution is to operationalize “alternative solution” not as a different parameterization of the same latent mechanism, but as a network whose internal activity is not linearly predictive of earlier solutions once output-potent structure has been factored out [2509.21504].

Source: https://www.emergentmind.com/topics/iterative-neural-similarity-deflation-insd