The Forward-Backward Disconnect: State Dynamics, Credit Assignment, and Biological Grounding in Neural Computation
Published 20 Aug 2026 in cs.NE | (2608.19995v1)
Abstract: A recurring pattern in neural computation is the reintroduction of dynamical and biological structure into models originally simplified for scalable optimization. Early feedforward networks reduced biological neurons to threshold or rate-like summation units, an abstraction compatible with global-gradient training at scale. Since then, forward computation has diversified: modern architectures carry recurrent state, retrieve from long contexts through attention and associative memory, compress histories through structured state-space dynamics, evolve in continuous time, settle to implicit equilibria, and communicate through sparse spikes. Training has diversified less. Scalable learning remains concentrated around backpropagation, backpropagation through time, adjoint methods, implicit differentiation, and surrogate-gradient variants. We call this asymmetry the forward-backward disconnect and develop a taxonomy spanning neural model families along three coupled axes: state-dynamics structure, credit-assignment mechanism, and biological grounding. Forward and learning grounding are treated separately, and the unit of analysis is the architecture-learning configuration rather than the architecture name alone. Across static, recurrent, attention-based, state-space, continuous-time, implicit, spiking, biologically plausible, and neuromorphic families, forward dynamics have diversified while the highest demonstrated scales remain concentrated in global or closely gradient-derived error-propagation mechanisms. Closing this disconnect requires better alignment among state dynamics, credit assignment, and computational substrate.
The paper introduces a taxonomy of 32 architecture–learning configurations, showing that forward dynamics span five classes while global or gradient-derived credit assignment dominates, covering 16 configurations.
The survey finds that biologically grounded learning rules such as STDP, e-prop, equilibrium propagation, and predictive coding improve locality but have not yet matched backpropagation at full ImageNet or language-model scale.
The paper argues that progress requires jointly designing architectures, learning rules, and hardware, with benchmarks that measure accuracy, locality, temporal credit assignment, memory, and energy rather than accuracy alone.
Overview and central claim
This survey by Al Mubasher and Awad identifies and names a structural asymmetry in neural computation that the authors call the forward–backward disconnect: forward computation has diversified substantially across state-dynamics classes—recurrent hidden states, attention-based retrieval, structured state-space evolution, continuous-time trajectories, implicit fixed points, and event-driven spikes—while scalable credit assignment remains concentrated around backpropagation (BP), backpropagation through time (BPTT), adjoint methods, implicit differentiation, and surrogate-gradient variants. The paper's contribution is not a new algorithm but a taxonomic reconstruction that makes this asymmetry visible as a categorical pattern rather than a scattered observation. Its unit of analysis is deliberately the architecture–learning configuration rather than the architecture name alone, because the same forward operator occupies different taxonomic positions depending on whether it is trained by global gradients, approximate gradients, local plasticity, or energy-based objectives.
The survey is organized along three coupled axes: a five-class state-dynamics axis (Static, Discrete-time/sequence, Continuous-time, Implicit/fixed-point, Hybrid event-driven), a four-category credit-assignment axis (Global gradient; Approximate or implicit gradient; Local plasticity; Energy/unsupervised), and a split biological-grounding axis applied independently to the forward operator and to the learning rule. The split grounding scheme is one of the paper's more useful methodological moves: it allows configurations such as surrogate-gradient-trained versus STDP-trained spiking networks to be distinguished despite sharing an identical LIF forward operator.
Taxonomy construction
The taxonomy rests on a uniform description of neural models as state-evolution systems, with continuous-time dynamics h˙(t)=F(h(t),x(t);θF), discrete-time recurrences ht+1=F(ht,xt;θF), and implicit states defined by z⋆=F(z⋆,x;θF). Static feedforward maps are treated as the degenerate memoryless case. The credit-assignment axis is further refined by five secondary dimensions—gradient relation, update scope, temporal requirement, objective source, and update timing—which prevent the compact four-row primary from conflating genuinely different mechanisms such as feedback alignment, synthetic gradients, equilibrium propagation, and e-prop.
The audited corpus comprises 32 architecture–learning configurations. The headline quantitative result is stark: Global gradient accounts for 16 of 32 configurations, receives ribbons from four of the five forward classes, and is followed by Approximate or implicit gradient with 7. At the mechanism level, ordinary reverse-mode backpropagation alone accounts for ten configurations, while the Approximate/implicit category fragments across seven individually small mechanisms. The authors are appropriately careful about what these counts mean: they are descriptive of the audited representative set, not estimates of literature prevalence, and are conditioned on a documented search procedure with a June 2026 cutoff.
Biological grounding is assigned ordinally on each axis. On the forward side, tiers run from None/historical inspiration through Functional analogy and Architectural constraint to Mechanistic plausibility; on the learning side, from None through Weak and Local/algorithmic to Mechanistic plasticity. The tier boundaries are adjudicated explicitly—for example, Architectural constraint requires biology to rule out alternative design choices (the Neocognitron's simple/complex cell alternation), whereas the RNN's recurrence would have been adopted independently for sequence modeling and therefore merits only Functional analogy.
Findings across model families
The family-by-family traversal yields several results worth highlighting:
Discrete-time recurrence is where the disconnect first becomes diagnostic. RNNs, LSTMs, and GRUs trained by BPTT sit at Global gradient; classical Hopfield networks with Hebbian storage sit at Local plasticity but pay a capacity ceiling of roughly $0.14N$ patterns; RBMs and DBNs occupy the Energy/unsupervised row via contrastive divergence. The same forward class supports all four credit-assignment categories, and the choice of mechanism—not the forward dynamics—determines whether the model scales.
Attention, modern Hopfield retrieval, and structured SSMs form an analytical neighborhood rather than a single architecture. Modern Hopfield retrieval recovers softmax attention exactly at β=1/dk with exponentially small single-step retrieval error under separation assumptions; state-space duality connects structured SSMs to semiseparable matrices and linear attention. All three families have None/historical forward grounding and None learning grounding—their scaling advantages come precisely from discarding biological structure in favor of gradient-trainable parallelism. Notably, pure-SSM variants still lag Transformers on copying and in-context retrieval tasks.
Continuous-time and implicit models change state representation, not credit assignment.Neural ODEs recover gradients through adjoint ODEs (with Gholami et al.'s result that reverse-time reconstruction can be numerically inconsistent without checkpointing), and DEQs use implicit differentiation through (I−∂f/∂z⋆), achieving constant activation memory—a 172M-parameter DEQ-Transformer matches Transformer-XL perplexity on WikiText-103 at roughly one-third the activation memory. Scalability here is governed by numerical conditioning rather than expressivity.
Spiking networks exhibit the widest learning-axis split. Surrogate-gradient training has driven directly trained SNNs to ImageNet scale—Spikformer reaches 74.81% top-1 at 66.3M parameters with only 4 time steps, and QKFormer reaches 85.65% top-1 on ImageNet-1k—but its learning grounding is only Weak. STDP-trained SNNs achieve full Mechanistic plasticity grounding yet remain at small demonstrated scale (Diehl and Cook's 95% MNIST accuracy with 6,400 neurons). This pair is the clearest demonstration that biological grounding does not transfer automatically from forward operator to learning rule.
Biologically plausible learning and the substrate
The survey enumerates six BP assumptions that alternatives relax: weight transport, global error broadcast, update locking, exact gradients, stored activations, and temporal unrolling. Its synthesis table shows that no surveyed family relaxes the full set while preserving deep-learning scalability. The strongest recent counterweights are quantitative: Weight Mirror and Kolen–Pollack reach ResNet-50 ImageNet top-1 error of 23.4% and 23.9% versus 22.9% for BP; Product Feedback Alignment reaches 69.30% top-1 on ResNet-18 versus 69.69% for BP, closing the gap to under half a percentage point. Equilibrium propagation reaches 11.7% CIFAR-10 test error, and predictive coding admits BP-equivalence proofs under stated limit conditions. E-prop removes BPTT via synapse-local eligibility traces combined with a broadcast signal, demonstrating online temporal credit assignment—but the residual dependence on a top-down broadcast signal is exactly the coordinate the authors identify as unresolved. Dendritic Localized Learning extends dendrite-inspired plasticity to CNNs, ResNets, and Transformers competitively on CIFAR-scale tasks, though not yet at full ImageNet-1k scale.
The substrate analysis applies Hooker's hardware lottery to the disconnect: GPU/TPU fabrics natively support dense matmul, batch synchronization, and global gradient transport, while neuromorphic platforms (Loihi, Loihi 2, SpiNNaker, BrainScaleS-2) natively support sparse events, local state, and online updates. The asymmetry is quantified—Loihi reports up to five orders of magnitude advantage in time-to-solution and six in energy on large sparse-coding workloads—but these gains are workload-specific and conditioned on sparsity. The current bridge between substrates is empirical and asymmetric: large spiking systems are trained off-chip with surrogate gradients and deployed on-chip for inference, with conversion methods such as SpikeZIP-TF achieving 83.82% ImageNet-1k accuracy at approximately 8 time steps. The missing bridge—a substrate-aligned learning rule competitive with BP at deep-network scale—is identified as absent, and neuromorphic hardware does not supply it.
Limitations and open questions
The paper is candid about several constraints. The configuration counts depend on scope rules that exclude active programs organized by orthogonal principles (graph neural networks, diffusion models, capsule networks, world models), and a sparse-region audit found that three of four initially flagged empty taxonomy cells were occupied after reclassification—so empty cells should not be read as research gaps without verification. The DEQ equivalence to infinite-depth weight-tied networks holds only under unique-fixed-point convergence assumptions; predictive-coding BP-equivalence requires unit-variance and output-layer-dominant limits or a fixed-prediction assumption; modern Hopfield's exponential capacity holds under pattern-separation assumptions. These regime dependencies mean the formal bridges among attention, associative memory, and SSMs hold "only under the specific regime conditions stated by their source results," as the authors put it.
The central open problem is stated falsifiably: a strictly local or mechanistic-plasticity rule reaching backpropagation-parity performance at full ImageNet-1k or GPT-class language-model scale under comparable training compute would revise the disconnect claim. Four specific gaps remain: a local rule addressing both spatial and temporal credit assignment without unrolling time into space; compartment-local teaching signals exploiting dendritic structure; on-chip deep learning that trains rather than merely deploys non-trivial models; and configuration-level benchmarks jointly reporting accuracy, locality, temporal behavior, memory, and energy cost.
Conclusion
This survey converts a widely intuited but rarely systematized observation—that neural architectures have diversified far faster than their training rules—into a reproducible taxonomy over 32 architecture–learning configurations, with biological grounding assessed independently on forward and learning axes. Its empirical core is the concentration of scalable credit assignment in global and gradient-derived mechanisms (16 of 32 configurations in Global gradient; ten in plain reverse-mode BP) against a fully populated fan of five forward state-dynamics classes. The paper's strongest claims are carefully bounded: alternatives to backpropagation exist and are improving, hardware alignment removes substrate penalties for event-driven computation, but no surveyed configuration combines locality, temporal credit assignment, task breadth, scalability, and substrate alignment. Whether configuration-level co-design can produce such a combination at contemporary scale remains, by the authors' own framing, an open empirical question.