LocalRNN: Localized Neural Architectures

• LocalRNN is a framework that incorporates locality in parameterization, computation, and learning to enable memory-efficient hierarchical RNN training and mesh-free PDE solutions.
• The approach leverages locally computable losses and randomized features to decouple global gradient flows, achieving high accuracy with reduced computational overhead.
• Local representation alignment in LocalRNN replaces global backpropagation with localized target propagation, offering practical benefits for parallel computation in sequential tasks.

LocalRNN is an umbrella term for methods and architectures that incorporate locality—either in parameterization, computation, or learning—in the design and training of recurrent neural networks (RNNs) or their randomized, mesh-free counterparts. The LocalRNN designation has been used in several research trajectories, encompassing: (1) hierarchical RNNs with locally computable losses for memory-efficient training; (2) mesh-free numerical methods using local sets of randomized features for fast, linear-algebraic solution of partial differential equations (PDEs); and (3) localized updates and target propagation schemes for temporal learning in RNNs. This article reviews the mathematical foundations, architectures, algorithmic principles, convergence properties, and empirical performance of prominent LocalRNN approaches.

1. Hierarchical LocalRNNs with Locally Computable Losses

The LocalRNN framework in hierarchical sequence modeling eliminates memory- and compute-intensive cross-level gradient flows by introducing auxiliary locally computable losses at each level of a hierarchical RNN (HRNN) (Mujika et al., 2019). An HRNN of depth $D$ consists of RNN modules $f^{(l)}$ at each level $l$, with higher levels running at exponentially slower clocks. At time $t$, only the relevant subset of levels is active.

LocalRNNs decouple backpropagation by introducing per-level auxiliary decoders $g^{(l)}$ trained to reconstruct recent local state histories or inputs. The full loss is

$$L = L_{\mathrm{task}} + \sum_{l=1}^{D-1} \beta_l \sum_{t \in \mathrm{Tick}(l)} L_t^{(l)},$$

where $L_{\mathrm{task}}$ is the global task loss, and $L_t^{(l)}$ is the reconstruction loss for level $l$. During training, downward (higher-to-lower) gradient paths are severed; each level is updated by local loss terms and limited-use global gradients, yielding an exponential reduction in memory:

$$M_{\mathrm{LocalRNN}} = O\bigg((D-1)kH + \frac{T}{k^{D-1}}H \bigg).$$

Empirically, LocalRNN matches or slightly lags full hierarchical RNNs using TBPTT, with major memory savings. On copy tasks and Pixel-MNIST, LocalRNN achieves performance equivalent to full HRNN at 1/4 to 1/5 the memory footprint (Mujika et al., 2019).

2. Local Randomized Neural Networks for PDEs

In mesh-free numerical analysis, Local Randomized Neural Network (LRNN) methods approximate PDE solutions by assigning independent randomized-neural-feature bases to non-overlapping subdomains (Sun et al., 2022, Li et al., 2023). For domain $\Omega \subset \mathbb{R}^d$ partitioned into $\{\Omega_i\}$, an LRNN uses

$$u_i(x) = \sum_{j=1}^{M} \alpha_{ij} \, \sigma(w_{ij} \cdot x + b_{ij}), \quad x \in \Omega_i$$

with $w_{ij}, b_{ij}$ drawn randomly and fixed, leaving output weights $\alpha_{ij}$ as the only free parameters. This randomized feature selection renders the solution linear in $\alpha_{ij}$.

Coupling across subdomains is achieved by a discontinuous Galerkin (DG) framework. LRNN-DG forms a global, possibly overdetermined, linear system using interior penalty or collocation constraints for continuity and normal derivatives. The assembly reads:

• Compute local basis functions on each $\Omega_i$.
• Integrate DG forms (or collocational residuals) via quadrature over the domain and interfaces.
• Stack block-rows corresponding to domain interior, interface, and boundary conditions, yielding

$R X = \tilde{G}$

and solve $\min_X \|RX - \tilde{G}\|_2^2$.

This approach exhibits several favorable properties:

• Mesh-free discretization, operable in arbitrary dimension.
• Linear solve phase, entirely replacing non-linear gradient-based training.
• High accuracy per degree of freedom, especially on interface and high-contrast problems (relative $L^2$ error $\sim 10^{-5}$ in seconds for $N \sim 10^4$, $d$ up to $20$) (Li et al., 2023).

3. Local Representation Alignment for RNNs

LocalRNN training via Local Representation Alignment (LRA) replaces global backpropagation with local target-propagation/objective mechanisms (Ororbia et al., 2018, Manchev et al., 18 Apr 2025). The core principle is to decompose the RNN computation graph into shallow subgraphs corresponding to adjacent time steps or layers. For a vanilla RNN with hidden state $h_t$:

$$h_t = \sigma(W_{xh} x_t + W_{hh} z_{t-1} + b_h), \quad z_t = h_t$$

LRA assigns local targets $\hat{z}_t$ to each $z_t$. The update for each parameter block is driven by a local loss $\mathcal{L}(z_t, \hat z_t)$, with local targets propagated backward in time via a single-step target-prop update:

$$\hat{z}_t = \sigma(\hat{h}_t - \gamma \frac{\partial \mathcal{L}(z_{t+1}, \hat{z}_{t+1})}{\partial h_t} )$$

This strictly localizes credit assignment but may not prevent vanishing gradients; gradient regularization (as in (Manchev et al., 18 Apr 2025)) addresses this by explicitly pushing Jacobian norms toward unity.

On synthetic sequence tasks, regularized LRA-RNN matches BPTT up to moderate horizons, but lags target-propagation-through-time (TPTT) at long sequence lengths. Notably, empirical evidence refutes earlier claims that LRA universally resolves vanishing gradients.

4. Practical Algorithms and Pseudocode

The LocalRNN paradigm yields efficient algorithms grounded in local computation:

• Hierarchical LocalRNN: Each level updates its own parameters using gradients restricted to local reconstruction losses and a short BPTT window; no global backward pass across levels is computed (Mujika et al., 2019).
• LocalRNN-DG for PDEs: Randomized features are drawn for each subdomain; global matrices are assembled from DG forms (or collocation residuals) and solved as a single SPD/least-squares problem. Assembly pseudocode and workflow are well specified (Sun et al., 2022).
• LRA for RNNs: Each time slice computes local loss and target, updates parameters locally, and uses optional regularization to enhance signal propagation (Manchev et al., 18 Apr 2025). State correction in representation-alignment approaches can be computed in parallel over time and layers (Ororbia et al., 2018).

5. Empirical Results and Limitations

Experimental results, summarized below, demarcate the strengths and bounds of LocalRNN techniques:

Application Domain	Method	Memory/Compute	Accuracy/Metric (Examples)	Main Limitations
HRNN sequence learning	LocalRNN (aux. losses)	$O(((D-1)k + T/k^{D-1})H)$	Copy: $L_{\max}=108$ at $1/4$ memory of TBPTT HRNN; MNIST: $0.9886$	Requires fixed clock, aux. param. tuning (Mujika et al., 2019)
Mesh-free PDE solvers	LRNN-DG	Single linear solve	Poisson: $10^{-8}$ relative $L^2$ error in 2 s	Dense linear systems, range selection, no adaptive sampling (Li et al., 2023)
RNN training (temporal)	LRA-RNN	$O(TK m^2)$ (with $K$ inner steps)	Up to $T=20$ on Temporal Order (regularized), $T=80$ Random Permutation	Vanishing gradients, overhead, suboptimal vs. TPTT/BPTT (Manchev et al., 18 Apr 2025)

In HRNNs, local objective decoupling yields substantial memory savings at comparable performance. For mesh-free PDEs, the LRNN approach outperforms PINNs and fitted-mesh alternatives in efficiency and accuracy (especially under strong interface jumps and in high $d$), at the cost of increased memory for the dense collocation matrix and sensitivity to random-feature hyperparameters. In RNN sequential learning, LRA methods enable fully local credit assignment and parallelization but lose performance in very-long-range temporal regimes unless significantly regularized.

6. Connections, Extensions, and Future Directions

The LocalRNN paradigm lies at the confluence of randomized neural methods, domain decomposition, local learning algorithms, and memory-efficient training strategies. Key connections include:

• DP-based domain decomposition and collocation for mesh-free PDE inference (Sun et al., 2022, Li et al., 2023).
• Memory-efficient sequence learning by decoupling hierarchical credit assignment (Mujika et al., 2019).
• Biologically plausible learning via local representation alignment and predictive coding (Ororbia et al., 2018).
• Local gradient/target-based alternatives to BPTT for RNNs, with explicit studies of their limitations and interactions with vanishing gradients (Manchev et al., 18 Apr 2025).

Open research directions focus on: (1) robust range selection and adaptive random-feature sampling for LRNN PDE solvers; (2) closure of the empirical performance gap between local and global credit assignment in RNNs for long temporal dependencies; (3) integration of learnable clocks or spatiotemporal adaptivity in HRNNs; and (4) improved algorithmic stability and theoretical analysis for local learning methods in deep or recurrent architectures.

7. Summary

LocalRNN methodologies exemplify the replacement of global, resource-intensive optimization and credit assignment by localized, often linear or parallelizable, algorithms across a variety of settings—from mesh-free PDE solvers to memory-efficient hierarchical RNNs and temporally decomposed RNN learning algorithms. These approaches offer substantial computational and memory benefits, but typically require heuristic parameter tuning and may encounter residual signal propagation issues in extreme regimes. The current research trajectory continues to expand the scope, foundations, and robustness of LocalRNN approaches across scientific computing and sequence modeling domains (Mujika et al., 2019, Sun et al., 2022, Li et al., 2023, Ororbia et al., 2018, Manchev et al., 18 Apr 2025).