Pseudo-Inverse Learning Rule Overview

• Pseudo-Inverse Learning Rule is a non-iterative, closed-form method that computes neural network weights by solving a least-squares problem using the Moore–Penrose pseudoinverse.
• Regularization techniques, such as Tikhonov damping, are applied to stabilize the solution against numerical issues arising from small singular values.
• PILR extends to online, deep, and associative-memory frameworks, enabling efficient incremental learning and versatile applications in neural network training.

Searching arXiv for the cited papers to ground the article in current records. Pseudo-Inverse Learning Rule (PILR) denotes a family of non-iterative, non–gradient-descent learning procedures in which neural-network weights are obtained from closed-form linear algebra, typically by solving a least-squares problem with the Moore–Penrose pseudoinverse. In its canonical feedforward form, PILR fixes or otherwise determines hidden representations and computes output weights as a pseudoinverse map from hidden activations to targets; in broader variants, pseudoinverse constructions are also used to set encoder, decoder, recurrent, or associative-memory couplings. The rule was introduced for feedforward neural networks in 1995, later generalized to incremental, deep, associative-memory, and quantum settings, and is closely tied to regularization, SVD-based conditioning analysis, and block or recursive matrix identities (Guo, 2018).

1. Foundational formulation and historical scope

In single-hidden-layer feedforward models, PILR converts training into a linear least-squares problem. If $H$ denotes the hidden-layer design matrix and $Y$ the target matrix, the output weights are computed as

$\beta^* = H^+ T$

or, in equivalent notation,

$W = H^+ Y.$

With singular value decomposition $H = U \Sigma V^\top$, the pseudoinverse is

$H^+ = V \Sigma^+ U^\top.$

In full-column-rank and full-row-rank cases, the standard closed forms

$$H^+ = (H^\top H)^{-1}H^\top \quad\text{and}\quad H^+ = H^\top(HH^\top)^{-1}$$

apply, respectively (Cancelliere et al., 2015).

The 1995 PIL formulation, as reviewed in "A VEST of the Pseudoinverse Learning Algorithm" (Guo, 2018), presents PIL as a non-iterative, non–gradient-descent training rule for feedforward neural networks. For a single-hidden-layer feedforward network with hidden output matrix $H$ and targets $T$, the basic rule is

$$\min_\beta \|H\beta - T\|_F^2 \quad\Rightarrow\quad \beta^* = H^+T.$$

When the output nonlinearity $Y$0 is invertible on the target range, the target can be transformed as $Y$1, yielding

$Y$2

The same review emphasizes that PIL did not require differentiable activations; it required nonlinear transforms that raise rank.

The early PIL literature also proposed deterministic and random hidden-layer constructions. One deterministic choice is $Y$3, followed by $Y$4. Randomized variants choose hidden weights in a small interval and then solve the output layer analytically. The review further states that PIL advocated $Y$5 hidden units for exact learning on $Y$6 samples, so that the hidden-output matrix $Y$7 could be numerically full rank and exactly invertible (Guo, 2018).

Several later literatures recover the same core rule under different names. The review explicitly argues that Extreme Learning Machine (ELM) is a “Variant crEated by Simple name alTernation (VEST)” of PIL for single-hidden-layer feedforward networks, because ELM likewise fixes the hidden layer and solves the output layer by pseudoinverse (Guo, 2018). In Broad Learning System (BLS), the same principle appears as

$Y$8

with the expanded input matrix $Y$9 built by concatenating feature and enhancement nodes (Zhu et al., 2019).

2. Spectral conditioning, regularization, and hidden-layer sizing

A central theme in PILR research is that analytical solvability does not remove numerical pathologies. "An analysis of numerical issues in neural training by pseudoinversion" (Cancelliere et al., 2015) studies single-hidden-layer neural networks trained by pseudoinversion and identifies instability through singular value analysis. If

$\beta^* = H^+ T$0

very small singular values $\beta^* = H^+ T$1 produce large entries in $\beta^* = H^+ T$2, making the solution sensitive to noise and round-off. The same work uses the condition number

$\beta^* = H^+ T$3

and the ratio

$\beta^* = H^+ T$4

relative to the numerical threshold $\beta^* = H^+ T$5 used by SVD implementations. Instability is marked when $\beta^* = H^+ T$6.

That paper introduces the “critical hidden layer size” as the smallest hidden size $\beta^* = H^+ T$7 at which $\beta^* = H^+ T$8 drops below $\beta^* = H^+ T$9. As this critical size is approached, unregularized test error grows sharply; after automatic SVD truncation, the error can decrease again, but overfitting may then dominate. The proposed remedy is Tikhonov regularization: $W = H^+ Y.$0 or, in the underdetermined case,

$W = H^+ Y.$1

In SVD form,

$W = H^+ Y.$2

so that small singular directions are attenuated continuously rather than truncated abruptly (Cancelliere et al., 2015).

The same study also proposes an effective hidden-weight scaling rule: $W = H^+ Y.$3 instead of the ELM-like $W = H^+ Y.$4. The stated rationale is that, for sigmoid or tanh, smaller pre-activations keep neurons in the near-linear region, avoid saturation and specialization, reduce overfitting, and stabilize training across hidden-layer sizes. In experiments over Abalone, CPU, Delta Ailerons, Iris, Diabetes, and Landsat, regularized methods usually improved performance. The reported examples include Iris, where Sigm-reg achieved $W = H^+ Y.$5 at hidden size $W = H^+ Y.$6 versus ELM $W = H^+ Y.$7 at $W = H^+ Y.$8, and Diabetes, where Sigm-reg achieved $W = H^+ Y.$9 at $H = U \Sigma V^\top$0 versus ELM $H = U \Sigma V^\top$1 at $H = U \Sigma V^\top$2; Landsat is explicitly identified as an exception where ELM was best, with ELM $H = U \Sigma V^\top$3 at $H = U \Sigma V^\top$4 and singular values never approaching threshold $H = U \Sigma V^\top$5 (Cancelliere et al., 2015).

Regularization is equally prominent in later PILR instantiations. In feedforward training it appears as ridge or Tikhonov damping; in BLS incremental learning it appears as

$H = U \Sigma V^\top$6

in PILAE it appears in decoder training as

$H = U \Sigma V^\top$7

and in synergetic two-way pseudoinverse systems it is used throughout decoder, classifier, and fusion solves to “prevent overfitting” and stabilize $H = U \Sigma V^\top$8 inversions (Zhu et al., 2019, Guo et al., 2018, Liu et al., 2024).

3. Incremental, online, and inverse-free formulations

One major line of work replaces batch pseudoinversion by recursive or blockwise updates. "Learning the Pseudoinverse Solution to Network Weights" (Tapson et al., 2012) treats a random-feature feedforward network with hidden activations $H = U \Sigma V^\top$9, output weights $H^+ = V \Sigma^+ U^\top.$0, and targets $H^+ = V \Sigma^+ U^\top.$1. The batch solution is $H^+ = V \Sigma^+ U^\top.$2, but the paper derives an online learning rule adapted from Greville’s method. With inhibition matrix $H^+ = V \Sigma^+ U^\top.$3, gain vector

$H^+ = V \Sigma^+ U^\top.$4

output error

$H^+ = V \Sigma^+ U^\top.$5

and new hidden activation $H^+ = V \Sigma^+ U^\top.$6, the update is

$H^+ = V \Sigma^+ U^\top.$7

The stationary inhibition update is

$H^+ = V \Sigma^+ U^\top.$8

An adaptive variant adds

$H^+ = V \Sigma^+ U^\top.$9

with

$$H^+ = (H^\top H)^{-1}H^\top \quad\text{and}\quad H^+ = H^\top(HH^\top)^{-1}$$0

to increase plasticity under non-stationarity. The paper states that, for stationary data, the algorithm converges to the exact pseudoinverse solution and is significantly more memory-efficient than batch SVD; on MNIST with hidden $$H^+ = (H^\top H)^{-1}H^\top \quad\text{and}\quad H^+ = H^\top(HH^\top)^{-1}$$1, the largest OLP matrix is $$H^+ = (H^\top H)^{-1}H^\top \quad\text{and}\quad H^+ = H^\top(HH^\top)^{-1}$$2, whereas batch SVD would require storing matrices on the order of $$H^+ = (H^\top H)^{-1}H^\top \quad\text{and}\quad H^+ = H^\top(HH^\top)^{-1}$$3 and two similarly sized factors (Tapson et al., 2012).

For row-partitioned incremental learning in BLS, "Reducing the Computational Complexity of Pseudoinverse for the Incremental Broad Learning System on Added Inputs" (Zhu et al., 2019) begins from

$$H^+ = (H^\top H)^{-1}H^\top \quad\text{and}\quad H^+ = H^\top(HH^\top)^{-1}$$4

with

$$H^+ = (H^\top H)^{-1}H^\top \quad\text{and}\quad H^+ = H^\top(HH^\top)^{-1}$$5

In the common $$H^+ = (H^\top H)^{-1}H^\top \quad\text{and}\quad H^+ = H^\top(HH^\top)^{-1}$$6 branch, the paper rewrites

$$H^+ = (H^\top H)^{-1}H^\top \quad\text{and}\quad H^+ = H^\top(HH^\top)^{-1}$$7

via

$$H^+ = (H^\top H)^{-1}H^\top \quad\text{and}\quad H^+ = H^\top(HH^\top)^{-1}$$8

into two dimension-aware forms. Defining $$H^+ = (H^\top H)^{-1}H^\top \quad\text{and}\quad H^+ = H^\top(HH^\top)^{-1}$$9,

$H$0

and

$H$1

The output update is then

$H$2

The paper reports that the proposed and existing algorithms achieve the same testing accuracy, while the speedups in BLS training time of the proposed algorithm over the existing algorithm are $H$3. In the detailed experimental summary, MNIST speedups are $H$4–$H$5 when $H$6 and $H$7–$H$8 when $H$9; NORB speedups are $T$0–$T$1 when $T$2 and $T$3–$T$4 when $T$5, with unchanged testing accuracy (Zhu et al., 2019).

Inverse-free ELM work pushes the same idea toward hidden-node growth and pruning. "Efficient Inverse-Free Algorithms for Extreme Learning Machine Based on the Recursive Matrix Inverse and the Inverse LDL' Factorization" (Zhu et al., 2019) uses the regularized solution

$T$6

and updates the Hermitian inverse recursively when a new hidden node is added. One direct update maintains

$T$7

through

$T$8

followed by a direct output-weight update

$T$9

Algorithm 3 instead updates an inverse $$\min_\beta \|H\beta - T\|_F^2 \quad\Rightarrow\quad \beta^* = H^+T.$$0 factorization to avoid numerical instabilities after a very large number of iterations. The reported numerical experiments state that standard ELM, the existing inverse-free ELM algorithm, and the proposed Algorithms 1–3 achieve the same performance in regression and classification, while all three proposed algorithms significantly accelerate the existing inverse-free ELM algorithm; on MNIST, Algorithm 2 yields speedups of approximately $$\min_\beta \|H\beta - T\|_F^2 \quad\Rightarrow\quad \beta^* = H^+T.$$1–$$\min_\beta \|H\beta - T\|_F^2 \quad\Rightarrow\quad \beta^* = H^+T.$$2, and Algorithm 3 yields $$\min_\beta \|H\beta - T\|_F^2 \quad\Rightarrow\quad \beta^* = H^+T.$$3–$$\min_\beta \|H\beta - T\|_F^2 \quad\Rightarrow\quad \beta^* = H^+T.$$4 (Zhu et al., 2019).

"Efficient Inverse-Free Incremental and Decremental Algorithms for Multiple Hidden Nodes in Extreme Learning Machine" (Zhu, 2020) generalizes this program to block additions and removals. For $$\min_\beta \|H\beta - T\|_F^2 \quad\Rightarrow\quad \beta^* = H^+T.$$5 added hidden nodes,

$$\min_\beta \|H\beta - T\|_F^2 \quad\Rightarrow\quad \beta^* = H^+T.$$6

with output-weight update

$$\min_\beta \|H\beta - T\|_F^2 \quad\Rightarrow\quad \beta^* = H^+T.$$7

The decremental counterpart removes multiple redundant nodes in one iteration through the corresponding Schur complement. The paper frames these methods as exact regularized least-squares updates that avoid explicit large-matrix inversion (Zhu, 2020).

4. Deep feedforward and two-way pseudoinverse systems

PILR has also been extended beyond shallow random-feature readouts. "PILAE: A Non-gradient Descent Learning Scheme for Deep Feedforward Neural Networks" (Guo et al., 2018) uses low-rank approximations of pseudoinverses to construct autoencoder encoders and decoders layer by layer. For input $$\min_\beta \|H\beta - T\|_F^2 \quad\Rightarrow\quad \beta^* = H^+T.$$8 with SVD $$\min_\beta \|H\beta - T\|_F^2 \quad\Rightarrow\quad \beta^* = H^+T.$$9, a truncated pseudoinverse

$Y$00

is used to define the encoder

$Y$01

The decoder is learned analytically by

$Y$02

The hidden size is selected by rank-guided heuristics: $Y$03 or

$Y$04

for near-full-rank noisy data. Stacking proceeds greedily, and the final readout is

$Y$05

An early stopping criterion is

$Y$06

The paper reports, for example, on MNIST and Fashion-MNIST that PILAE with a single-hidden-layer network readout can match or exceed BP-trained counterparts in some configurations: MNIST SHLN $Y$07 for PILAE versus $Y$08 for BP, and Fashion-MNIST SHLN $Y$09 for PILAE versus $Y$10 for BP (Guo et al., 2018).

"Semi-adaptive Synergetic Two-way Pseudoinverse Learning System" (Liu et al., 2024) further expands PILR into a two-way subsystem comprising forward learning, backward learning, and feature concatenation. The forward stack uses tied-weight PILAE-style reconstruction: $Y$11 with final forward readout

$Y$12

Backward learning propagates labels through pseudoinverse and inverse-activation maps: $Y$13 and fits backward weights by pseudoinverse at each depth. Forward and backward features are then fused and classified analytically. Depth is determined by early stopping used “as a structural control scheme.”

The empirical summary states that the method achieved the best test accuracy on $Y$14 of $Y$15 datasets. Illustrative results include MNIST $Y$16 versus $Y$17 for ELM-AE and HELM, NORB $Y$18 versus $Y$19 for PILLS, and F-MNIST $Y$20 versus $Y$21 for BLS; the average rank is reported as $Y$22 for the proposed method versus $Y$23 for HELM, $Y$24 for PILAE, $Y$25 for ELM-AE, $Y$26 for PILLS, and $Y$27 for BLS (Liu et al., 2024).

A distinct line, "A New Backpropagation Algorithm without Gradient Descent" (Ranganathan et al., 2018), applies local pseudoinverse corrections to neuron-wise affine maps. For a scalar block, the paper solves

$Y$28

by pseudoinverse, then updates weights and biases with a scaling factor $Y$29. The method uses inverse activations to define desired pre-activations, but it remains iterative over epochs and therefore occupies a different point in the PILR design space than one-shot output-weight solves (Ranganathan et al., 2018).

5. Associative-memory pseudoinverse rules

In Hopfield-like networks, PILR has a different but related meaning: the coupling matrix is constructed as a projector onto the stored-pattern subspace. For patterns collected in columns of $Y$30, the classical rule is

$Y$31

or, with explicit normalization,

$Y$32

In components,

$Y$33

followed by $Y$34. This construction makes the stored patterns stable fixed points by orthogonalizing their contributions via the inverse Gram matrix (Doi et al., 21 Apr 2026).

For biased patterns, "Daydreaming algorithm for Biased Patterns" (Doi et al., 21 Apr 2026) introduces the centered pseudo-inverse rule. With neuron-wise means

$Y$35

the centered coupling is

$Y$36

or equivalently

$Y$37

with $Y$38. The fixed-point identity is

$Y$39

The corresponding centered retrieval dynamics is

$Y$40

with energy

$Y$41

The paper compares centered pseudo-inverse and centered Daydreaming at $Y$42, $Y$43, $Y$44, and $Y$45. It reports that centered pseudo-inverse basins shrink as bias increases, while centered Daydreaming yields a robust plateau at $Y$46 already for $Y$47 across all tested biases (Doi et al., 21 Apr 2026).

"Storing Cycles in Hopfield-type Networks with Pseudoinverse Learning Rule - Retrievability and Bifurcation Analysis" (Zhang et al., 2013) extends the same projector logic to cyclic pattern storage. For a cycle

$Y$48

with cyclic permutation matrix $Y$49, the pseudoinverse learning rule constructs

$Y$50

Here $Y$51 stores static patterns and $Y$52 enforces one-step transitions. In a delayed continuous-time Hopfield-type network, the paper derives the characteristic equation, classifies Hopf, pitchfork, and Bogdanov–Takens bifurcations, and reports that admissible cycles can be retrieved as attracting limit cycles, unstable periodic solutions, or delay-induced long-lasting transient oscillations, depending on cycle structure and parameters (Zhang et al., 2013).

A further modification appears in "Inferring Concepts from Noisy Examples in Hopfield-like Neural Networks" (Benedetti et al., 1 Feb 2026). Instead of using empirical pattern correlations, the model uses an expected block-diagonal correlation structure for families of noisy examples generated around unknown archetypes: $Y$53 The paper states that classical pseudo-inverse tends to overfit correlated examples by building separate attractors for each example, whereas the modified rule uses the known family structure to foster attractors closer to archetypes. Replica analysis identifies Fully Symmetric, Class Representant, and Outlier Excluding generalizing states, and the paper states that 1RSB substantially extends the survival of these states to higher loads compared to RS (Benedetti et al., 1 Feb 2026).

6. Interpretations, extensions, and recurring debates

Several recurring debates concern what PILR does and does not guarantee. One misconception is that exact algebraic fitting automatically yields stable or generalizable models. The numerical-analysis literature explicitly rejects this simplification: near-singular hidden matrices produce pronounced error spikes, regularization is often essential, and hidden-layer sizing must be conditioned on singular-spectrum behavior rather than exact interpolation alone (Cancelliere et al., 2015). In the Hopfield setting, exact fixed-point storage likewise does not imply large basins of attraction; the biased-pattern study shows that centered pseudo-inverse and centered Daydreaming stabilize stored patterns through different mechanisms and yield markedly different basin structures (Doi et al., 21 Apr 2026).

A second debate concerns locality and biological plausibility. Batch SVD computation of a pseudoinverse is global and offline, but the OLP formulation recasts the same least-squares solution into local output-synapse updates and an inhibition matrix. That paper explicitly argues that the resulting learning method is biologically plausible and suitable for neuromorphic constraints, while remaining algebraically equivalent to recursive least squares with unit forgetting in the stationary case (Tapson et al., 2012).

A third debate concerns nomenclature and priority. The PIL review argues that ELM’s essential method—random hidden parameters plus output-weight computation by pseudoinverse—is methodologically identical to PIL0 for single-hidden-layer feedforward networks, and presents ELM as a VEST of PIL (Guo, 2018). That claim is historical and terminological rather than algebraic: both literatures share the same central least-squares pseudoinverse solve, even when they differ in emphasis, initialization, or implementation details.

Recent work also broadens PILR beyond classical neural-network settings. "Efficient quantum machine learning with inverse-probability algebraic corrections" (Seo, 23 Jan 2026) treats QNN training as a local inverse problem in probability space. With residual $Y$54 and Jacobian $Y$55 estimated by the parameter-shift rule, the update is

$Y$56

The paper identifies this as a damped Moore–Penrose pseudoinverse or Gauss–Newton step, emphasizes that it does not require learning-rate tuning, and reports faster convergence than gradient descent and Adam in a teacher–student benchmark. With finite-shot sampling, it reports that inverse-probability learning approaches the theoretical $Y$57 error scaling once $Y$58, while Adam can outperform only in the extreme low-shot case $Y$59 (Seo, 23 Jan 2026).

Across these domains, the unifying feature of PILR is not a single architecture but a single algebraic principle: training is formulated as solving a linearized or exactly linear inverse problem, and the Moore–Penrose pseudoinverse, often regularized or updated recursively, supplies the minimum-norm least-squares solution. What changes from one instantiation to another is the object being inverted—hidden activations, block-augmented design matrices, Jacobians, or pattern Gram matrices—and the accompanying strategy for conditioning, regularization, incrementality, or architectural control (Guo, 2018, Zhu et al., 2019, Guo et al., 2018, Benedetti et al., 1 Feb 2026).