---
title: Pseudo-Inverse Learning Rule Overview
url: https://www.emergentmind.com/topics/pseudo-inverse-learning-rule
type: topic
---

# Pseudo-Inverse Learning Rule Overview

Searching arXiv for the cited papers to ground the article in current records.
Pseudo-Inverse Learning Rule (PILR) denotes a family of non-iterative, non–gradient-descent learning procedures in which neural-network weights are obtained from closed-form linear algebra, typically by solving a least-squares problem with the Moore–Penrose pseudoinverse. In its canonical feedforward form, PILR fixes or otherwise determines hidden representations and computes output weights as a pseudoinverse map from hidden activations to targets; in broader variants, pseudoinverse constructions are also used to set encoder, decoder, recurrent, or associative-memory couplings. The rule was introduced for feedforward neural networks in 1995, later generalized to incremental, deep, associative-memory, and quantum settings, and is closely tied to regularization, SVD-based conditioning analysis, and block or recursive matrix identities [1805.07828].

## 1. Foundational formulation and historical scope

In single-hidden-layer feedforward models, PILR converts training into a linear least-squares problem. If \(H\) denotes the hidden-layer design matrix and \(Y\) the target matrix, the output weights are computed as
\[
\beta^* = H^+ T
\]
or, in equivalent notation,
\[
W = H^+ Y.
\]
With singular value decomposition \(H = U \Sigma V^\top\), the pseudoinverse is
\[
H^+ = V \Sigma^+ U^\top.
\]
In full-column-rank and full-row-rank cases, the standard closed forms
\[
H^+ = (H^\top H)^{-1}H^\top
\quad\text{and}\quad
H^+ = H^\top(HH^\top)^{-1}
\]
apply, respectively [1508.06092].

The 1995 PIL formulation, as reviewed in "A VEST of the Pseudoinverse Learning Algorithm" [1805.07828], presents PIL as a non-iterative, non–gradient-descent training rule for feedforward neural networks. For a single-hidden-layer feedforward network with hidden output matrix \(H\) and targets \(T\), the basic rule is
\[
\min_\beta \|H\beta - T\|_F^2
\quad\Rightarrow\quad
\beta^* = H^+T.
\]
When the output nonlinearity \(\sigma(\cdot)\) is invertible on the target range, the target can be transformed as \(B=\sigma^{-1}(T)\), yielding
\[
\beta^* = H^+B.
\]
The same review emphasizes that PIL did not require differentiable activations; it required nonlinear transforms that raise rank.

The early PIL literature also proposed deterministic and random hidden-layer constructions. One deterministic choice is \(V = X^+\), followed by \(H=g(XV)\). Randomized variants choose hidden weights in a small interval and then solve the output layer analytically. The review further states that PIL advocated \(L=N\) hidden units for exact learning on \(N\) samples, so that the hidden-output matrix \(H\) could be numerically full rank and exactly invertible [1805.07828].

Several later literatures recover the same core rule under different names. The review explicitly argues that Extreme Learning Machine (ELM) is a “Variant crEated by Simple name alTernation (VEST)” of PIL for single-hidden-layer feedforward networks, because ELM likewise fixes the hidden layer and solves the output layer by pseudoinverse [1805.07828]. In Broad Learning System (BLS), the same principle appears as
\[
W = A^+Y,\qquad
A^+ \approx \lim_{\lambda\to 0}(A^\top A + \lambda I)^{-1}A^\top,
\]
with the expanded input matrix \(A\) built by concatenating feature and enhancement nodes [1910.07755].

## 2. Spectral conditioning, regularization, and hidden-layer sizing

A central theme in PILR research is that analytical solvability does not remove numerical pathologies. "An analysis of numerical issues in neural training by pseudoinversion" [1508.06092] studies single-hidden-layer neural networks trained by pseudoinversion and identifies instability through singular value analysis. If
\[
H = U\Sigma V^\top,\qquad
H^+ = V\Sigma^+U^\top,
\]
very small singular values \(\sigma_i\) produce large entries in \(\Sigma^+\), making the solution sensitive to noise and round-off. The same work uses the condition number
\[
\kappa(H)=\sigma_{\max}/\sigma_{\min}
\]
and the ratio
\[
r(H)=\sigma_{\min}/\tau
\]
relative to the numerical threshold \(\tau\) used by SVD implementations. Instability is marked when \(r(H)<1\).

That paper introduces the “critical hidden layer size” as the smallest hidden size \(H\) at which \(r(H)=\sigma_{\min}/\tau\) drops below \(1\). As this critical size is approached, unregularized test error grows sharply; after automatic SVD truncation, the error can decrease again, but overfitting may then dominate. The proposed remedy is Tikhonov regularization:
\[
\beta_\lambda = (H^\top H + \lambda I)^{-1}H^\top Y
\]
or, in the underdetermined case,
\[
\beta_\lambda = H^\top(HH^\top + \lambda I)^{-1}Y.
\]
In SVD form,
\[
\beta_\lambda = V D U^\top Y,\qquad
D_i=\sigma_i/(\sigma_i^2+\lambda),
\]
so that small singular directions are attenuated continuously rather than truncated abruptly [1508.06092].

The same study also proposes an effective hidden-weight scaling rule:
\[
c_{ij}\sim \mathrm{Uniform}(-1/\sqrt{H},\,1/\sqrt{H}),
\]
instead of the ELM-like \(\mathrm{Uniform}(-1,1)\). The stated rationale is that, for sigmoid or tanh, smaller pre-activations keep neurons in the near-linear region, avoid saturation and specialization, reduce overfitting, and stabilize training across hidden-layer sizes. In experiments over Abalone, CPU, Delta Ailerons, Iris, Diabetes, and Landsat, regularized methods usually improved performance. The reported examples include Iris, where Sigm-reg achieved \(0.3\%\) at hidden size \(16\) versus ELM \(2.52\%\) at \(23\), and Diabetes, where Sigm-reg achieved \(20.7\%\) at \(45\) versus ELM \(21.2\%\) at \(69\); Landsat is explicitly identified as an exception where ELM was best, with ELM \(10.76\%\) at \(390\) and singular values never approaching threshold \((r(H)>1)\) [1508.06092].

Regularization is equally prominent in later PILR instantiations. In feedforward training it appears as ridge or Tikhonov damping; in BLS incremental learning it appears as
\[
A^+_\lambda = (A^\top A + \lambda I)^{-1}A^\top;
\]
in PILAE it appears in decoder training as
\[
W_d = XH^\top(HH^\top + \lambda_1 I)^{-1};
\]
and in synergetic two-way pseudoinverse systems it is used throughout decoder, classifier, and fusion solves to “prevent overfitting” and stabilize \(HH^\top+\lambda I\) inversions [1910.07755][1811.01545][2406.18931].

## 3. Incremental, online, and inverse-free formulations

One major line of work replaces batch pseudoinversion by recursive or blockwise updates. "Learning the Pseudoinverse Solution to Network Weights" [1207.3368] treats a random-feature feedforward network with hidden activations \(h_t\), output weights \(W\), and targets \(y_t\). The batch solution is \(W^T = H^+Y\), but the paper derives an online learning rule adapted from Greville’s method. With inhibition matrix \(\Theta_k\), gain vector
\[
k_k = \Theta_{k-1}a_k/(1+a_k^\top \Theta_{k-1}a_k),
\]
output error
\[
e_k = y_k - W_{k-1}a_k,
\]
and new hidden activation \(a_k\equiv h_k\), the update is
\[
W_k = W_{k-1} + e_k k_k^\top.
\]
The stationary inhibition update is
\[
\Theta_k = \Theta_{k-1} - (\Theta_{k-1}a_k)k_k^\top.
\]
An adaptive variant adds
\[
(I/M)(1-e^{-E_k}),
\]
with
\[
E_k = \|y_k - W_{k-1}a_k\|/(1+a_k^\top\Theta_{k-1}a_k)^2,
\]
to increase plasticity under non-stationarity. The paper states that, for stationary data, the algorithm converges to the exact pseudoinverse solution and is significantly more memory-efficient than batch SVD; on MNIST with hidden \(L=5760\), the largest OLP matrix is \(5760\times5760\), whereas batch SVD would require storing matrices on the order of \(5760\times60000\) and two similarly sized factors [1207.3368].

For row-partitioned incremental learning in BLS, "Reducing the Computational Complexity of Pseudoinverse for the Incremental Broad Learning System on Added Inputs" [1910.07755] begins from
\[
\tilde{A}=
\begin{bmatrix}
A\\A_x
\end{bmatrix},
\qquad
\tilde{A}^+ =
\begin{bmatrix}
A^+ - BD^\top & B
\end{bmatrix},
\]
with
\[
D^\top = A_xA^+.
\]
In the common \(C=0\) branch, the paper rewrites
\[
B = A^+D(I + D^\top D)^{-1}
\]
via
\[
(I+PQ)^{-1}P = P(I+QP)^{-1}
\]
into two dimension-aware forms. Defining \(\bar{D}:=A^+D\),
\[
B = \bar{D}(I + A_x\bar{D})^{-1}\qquad (q\le k),
\]
and
\[
B = (I + \bar{D}A_x)^{-1}\bar{D}\qquad (q>k).
\]
The output update is then
\[
{}^{x}W = W + B(Y_a - A_xW).
\]
The paper reports that the proposed and existing algorithms achieve the same testing accuracy, while the speedups in BLS training time of the proposed algorithm over the existing algorithm are \(1.24 - 1.30\). In the detailed experimental summary, MNIST speedups are \(1.35\)–\(1.57\times\) when \(q>k\) and \(1.09\)–\(1.13\times\) when \(q<k\); NORB speedups are \(1.10\)–\(1.35\times\) when \(q>k\) and \(1.07\)–\(1.09\times\) when \(q<k\), with unchanged testing accuracy [1910.07755].

Inverse-free ELM work pushes the same idea toward hidden-node growth and pruning. "Efficient Inverse-Free Algorithms for Extreme Learning Machine Based on the Recursive Matrix Inverse and the Inverse LDL' Factorization" [1911.04856] uses the regularized solution
\[
W = YH^\top(HH^\top + k^2I)^{-1},
\qquad
B = H^\top(HH^\top + k^2I)^{-1},
\]
and updates the Hermitian inverse recursively when a new hidden node is added. One direct update maintains
\[
Q = (HH^\top + k^2I)^{-1}
\]
through
\[
T_l = 1/(h_{l+1}^\top h_{l+1} + k^2 - p_l^\top Q_lp_l),
\qquad
t_l = -T_lQ_lp_l,
\]
followed by a direct output-weight update
\[
w_{l+1} = T_l(Yh_{l+1} - Wp_l).
\]
Algorithm 3 instead updates an inverse \(LDL^\top\) factorization to avoid numerical instabilities after a very large number of iterations. The reported numerical experiments state that standard ELM, the existing inverse-free ELM algorithm, and the proposed Algorithms 1–3 achieve the same performance in regression and classification, while all three proposed algorithms significantly accelerate the existing inverse-free ELM algorithm; on MNIST, Algorithm 2 yields speedups of approximately \(44\)–\(52\times\), and Algorithm 3 yields \(26\)–\(34\times\) [1911.04856].

"Efficient Inverse-Free Incremental and Decremental Algorithms for Multiple Hidden Nodes in Extreme Learning Machine" [2004.13023] generalizes this program to block additions and removals. For \(o\) added hidden nodes,
\[
R_{l+o}=
\begin{bmatrix}
R & P\\
P^\top & F
\end{bmatrix},
\quad
G=(F-P^\top QP)^{-1},
\quad
T=-QPG,
\]
with output-weight update
\[
W_{l+o} =
\big[
W + (YH_\delta^\top - WP)T^\top,\,
(YH_\delta^\top - WP)G
\big].
\]
The decremental counterpart removes multiple redundant nodes in one iteration through the corresponding Schur complement. The paper frames these methods as exact regularized least-squares updates that avoid explicit large-matrix inversion [2004.13023].

## 4. Deep feedforward and two-way pseudoinverse systems

PILR has also been extended beyond shallow random-feature readouts. "PILAE: A Non-gradient Descent Learning Scheme for Deep Feedforward Neural Networks" [1811.01545] uses low-rank approximations of pseudoinverses to construct autoencoder encoders and decoders layer by layer. For input \(X\in\mathbb{R}^{d\times N}\) with SVD \(X=U\Sigma V^\top\), a truncated pseudoinverse
\[
X_k^+ = V_k\Sigma_k^{-1}U_k^\top
\]
is used to define the encoder
\[
W_e = \hat{X}^+,
\qquad
H = f(W_eX).
\]
The decoder is learned analytically by
\[
W_d = XH^+ = XH^\top(HH^\top + \lambda_1 I)^{-1}.
\]
The hidden size is selected by rank-guided heuristics:
\[
p = r + \alpha(d-r),\qquad \alpha\in[0,1],
\]
or
\[
p = \beta d,\qquad \beta\in[0,1],
\]
for near-full-rank noisy data. Stacking proceeds greedily, and the final readout is
\[
W_{\text{out}} = ZY^\top(YY^\top + \lambda I)^{-1} = ZY^+.
\]
An early stopping criterion is
\[
\|H^+H - I\|_F^2 < \varepsilon.
\]
The paper reports, for example, on MNIST and Fashion-MNIST that PILAE with a single-hidden-layer network readout can match or exceed BP-trained counterparts in some configurations: MNIST SHLN \(0.9751\) for PILAE versus \(0.9727\) for BP, and Fashion-MNIST SHLN \(0.8819\) for PILAE versus \(0.8795\) for BP [1811.01545].

"Semi-adaptive Synergetic Two-way Pseudoinverse Learning System" [2406.18931] further expands PILR into a two-way subsystem comprising forward learning, backward learning, and feature concatenation. The forward stack uses tied-weight PILAE-style reconstruction:
\[
W_d^l = H^{l-1}(H^l)^\top(H^l(H^l)^\top + \lambda I)^{-1},
\qquad
W_e^l = (W_d^l)^\top,
\]
with final forward readout
\[
W_o = T F(X)^\top(F(X)F(X)^\top + \lambda I)^{-1}.
\]
Backward learning propagates labels through pseudoinverse and inverse-activation maps:
\[
H_b^L = W_o^\dagger T,
\qquad
H_b^l = (W_e^{l+1})^\dagger \sigma^{-1}(H_b^{l+1}),
\]
and fits backward weights by pseudoinverse at each depth. Forward and backward features are then fused and classified analytically. Depth is determined by early stopping used “as a structural control scheme.”

The empirical summary states that the method achieved the best test accuracy on \(16\) of \(19\) datasets. Illustrative results include MNIST \(98.97\%\) versus \(98.78\%\) for ELM-AE and HELM, NORB \(91.50\%\) versus \(90.46\%\) for PILLS, and F-MNIST \(89.58\%\) versus \(89.60\%\) for BLS; the average rank is reported as \(1.21\) for the proposed method versus \(3.74\) for HELM, \(4.68\) for PILAE, \(3.42\) for ELM-AE, \(3.84\) for PILLS, and \(3.89\) for BLS [2406.18931].

A distinct line, "A New Backpropagation Algorithm without Gradient Descent" [1802.00027], applies local pseudoinverse corrections to neuron-wise affine maps. For a scalar block, the paper solves
\[
[C\;\;1]
\begin{bmatrix}
\Delta w\\
\Delta b
\end{bmatrix}
= x - x_n
\]
by pseudoinverse, then updates weights and biases with a scaling factor \(\alpha\). The method uses inverse activations to define desired pre-activations, but it remains iterative over epochs and therefore occupies a different point in the PILR design space than one-shot output-weight solves [1802.00027].

## 5. Associative-memory pseudoinverse rules

In Hopfield-like networks, PILR has a different but related meaning: the coupling matrix is constructed as a projector onto the stored-pattern subspace. For patterns collected in columns of \(X\in\mathbb{R}^{N\times P}\), the classical rule is
\[
J = X(X^\top X)^{-1}X^\top
\]
or, with explicit normalization,
\[
J = \frac{1}{N}X\big(\tfrac{1}{N}X^\top X\big)^{-1}X^\top.
\]
In components,
\[
J_{ij} = \sum_{\mu,\nu=1}^{P} \xi_i^\mu (C^{-1})_{\mu\nu}\xi_j^\nu,
\qquad
C_{\mu\nu}=\sum_k \xi_k^\mu\xi_k^\nu,
\]
followed by \(J_{ii}=0\). This construction makes the stored patterns stable fixed points by orthogonalizing their contributions via the inverse Gram matrix [2604.19258].

For biased patterns, "Daydreaming algorithm for Biased Patterns" [2604.19258] introduces the centered pseudo-inverse rule. With neuron-wise means
\[
m_i = \frac{1}{P}\sum_\mu \xi_i^\mu,
\qquad
\tilde{\xi}_i^\mu = \xi_i^\mu - m_i,
\]
the centered coupling is
\[
\tilde{J}=\tilde{X}(\tilde{X}^\top\tilde{X})^{-1}\tilde{X}^\top,
\]
or equivalently
\[
\tilde{J} = \frac{1}{N}\tilde{X}\big(\tfrac{1}{N}\tilde{X}^\top\tilde{X}\big)^{-1}\tilde{X}^\top,
\]
with \(\tilde{J}_{ii}=0\). The fixed-point identity is
\[
\tilde{J}\tilde{\xi}^\mu = \tilde{\xi}^\mu,
\qquad
\xi^\mu = \tilde{J}(\xi^\mu - m) + m.
\]
The corresponding centered retrieval dynamics is
\[
s_i(t+1)=\operatorname{sign}\!\left(\sum_{j\neq i}\tilde{J}_{ij}[s_j(t)-m_j] + m_i\right),
\]
with energy
\[
E(\mathbf{s}) = -\tfrac{1}{2}\sum_{i\neq j}\tilde{J}_{ij}[s_i-m_i][s_j-m_j] - \sum_i m_is_i.
\]
The paper compares centered pseudo-inverse and centered Daydreaming at \(N=500\), \(\alpha=P/N=0.4\), \(\tau=256\), and \(p_1\in\{0.5,0.6,0.7,0.8\}\). It reports that centered pseudo-inverse basins shrink as bias increases, while centered Daydreaming yields a robust plateau at \(m_{\text{final}}\approx1\) already for \(m_{\text{init}}\gtrsim0.7\) across all tested biases [2604.19258].

"Storing Cycles in Hopfield-type Networks with Pseudoinverse Learning Rule - Retrievability and Bifurcation Analysis" [1308.5201] extends the same projector logic to cyclic pattern storage. For a cycle
\[
\Sigma=(\xi^{(1)},\xi^{(2)},\dots,\xi^{(p)}),
\qquad
F=\Sigma P
\]
with cyclic permutation matrix \(P\), the pseudoinverse learning rule constructs
\[
J^0 = \Sigma\Sigma^+,
\qquad
J = F\Sigma^+.
\]
Here \(J^0\) stores static patterns and \(J\) enforces one-step transitions. In a delayed continuous-time Hopfield-type network, the paper derives the characteristic equation, classifies Hopf, pitchfork, and Bogdanov–Takens bifurcations, and reports that admissible cycles can be retrieved as attracting limit cycles, unstable periodic solutions, or delay-induced long-lasting transient oscillations, depending on cycle structure and parameters [1308.5201].

A further modification appears in "Inferring Concepts from Noisy Examples in Hopfield-like Neural Networks" [2602.01393]. Instead of using empirical pattern correlations, the model uses an expected block-diagonal correlation structure for families of noisy examples generated around unknown archetypes:
\[
J_{ij} = \frac{1}{N}\sum_{\mu,\nu=1}^{P}(\overline{C}^{-1})_{\mu\nu}\xi_i^\mu\xi_j^\nu,
\qquad
J_{ii}=0.
\]
The paper states that classical pseudo-inverse tends to overfit correlated examples by building separate attractors for each example, whereas the modified rule uses the known family structure to foster attractors closer to archetypes. Replica analysis identifies Fully Symmetric, Class Representant, and Outlier Excluding generalizing states, and the paper states that 1RSB substantially extends the survival of these states to higher loads compared to RS [2602.01393].

## 6. Interpretations, extensions, and recurring debates

Several recurring debates concern what PILR does and does not guarantee. One misconception is that exact algebraic fitting automatically yields stable or generalizable models. The numerical-analysis literature explicitly rejects this simplification: near-singular hidden matrices produce pronounced error spikes, regularization is often essential, and hidden-layer sizing must be conditioned on singular-spectrum behavior rather than exact interpolation alone [1508.06092]. In the Hopfield setting, exact fixed-point storage likewise does not imply large basins of attraction; the biased-pattern study shows that centered pseudo-inverse and centered Daydreaming stabilize stored patterns through different mechanisms and yield markedly different basin structures [2604.19258].

A second debate concerns locality and biological plausibility. Batch SVD computation of a pseudoinverse is global and offline, but the OLP formulation recasts the same least-squares solution into local output-synapse updates and an inhibition matrix. That paper explicitly argues that the resulting learning method is biologically plausible and suitable for neuromorphic constraints, while remaining algebraically equivalent to recursive least squares with unit forgetting in the stationary case [1207.3368].

A third debate concerns nomenclature and priority. The PIL review argues that ELM’s essential method—random hidden parameters plus output-weight computation by pseudoinverse—is methodologically identical to PIL0 for single-hidden-layer feedforward networks, and presents ELM as a VEST of PIL [1805.07828]. That claim is historical and terminological rather than algebraic: both literatures share the same central least-squares pseudoinverse solve, even when they differ in emphasis, initialization, or implementation details.

Recent work also broadens PILR beyond classical neural-network settings. "Efficient quantum machine learning with inverse-probability algebraic corrections" [2601.16665] treats QNN training as a local inverse problem in probability space. With residual \(r=y-p(\theta)\) and Jacobian \(J\) estimated by the parameter-shift rule, the update is
\[
\Delta\theta = (J^\top J + \lambda I)^{-1}J^\top r.
\]
The paper identifies this as a damped Moore–Penrose pseudoinverse or Gauss–Newton step, emphasizes that it does not require learning-rate tuning, and reports faster convergence than gradient descent and Adam in a teacher–student benchmark. With finite-shot sampling, it reports that inverse-probability learning approaches the theoretical \(1/S\) error scaling once \(S\gtrsim 10\), while Adam can outperform only in the extreme low-shot case \(S=1\) [2601.16665].

Across these domains, the unifying feature of PILR is not a single architecture but a single algebraic principle: training is formulated as solving a linearized or exactly linear inverse problem, and the Moore–Penrose pseudoinverse, often regularized or updated recursively, supplies the minimum-norm least-squares solution. What changes from one instantiation to another is the object being inverted—hidden activations, block-augmented design matrices, Jacobians, or pattern Gram matrices—and the accompanying strategy for conditioning, regularization, incrementality, or architectural control [1805.07828][1910.07755][1811.01545][2602.01393].

Source: https://www.emergentmind.com/topics/pseudo-inverse-learning-rule