ReLaX-Net: Time-Multiplexed Efficiency in PNNs

• The paper introduces ReLaX-Net, a parameter-efficient architecture that leverages time-multiplexed weight sharing to overcome scaling limits in physical neural networks.
• ReLaX-Net transforms L physical weight banks into T effective layers via fast switching, significantly reducing the parameter count by a factor of L/T.
• Empirical results on tasks like SVHN digit classification and Shakespeare language modeling demonstrate improved inference accuracy while balancing hardware constraints.

ReLaX-Net refers to a parameter-efficient architecture for physical neural networks (PNNs) that leverages time-multiplexed weight and bias reuse to overcome the scaling constraints of current hardware-limited PNN implementations. By decoupling the slow update times of physical weight banks from the rapid execution of nonlinear activations, ReLaX-Net achieves an effective network depth far exceeding the number of physically realized layers, enabling large-scale inference with a drastically reduced hardware footprint (Tsuchiyama et al., 28 Oct 2025).

1. Physical Neural Networks and Parameter Bottlenecks

Physical neural networks, including photonic, spintronic, or chemical platforms, implement trainable weights and nonlinear activations using hardware elements such as phase shifters, memristive crossbars, spin oscillators, or chemical reaction circuits. A persistent bottleneck in PNN scaling arises from the mismatch between fast nonlinear forward-pass timescales (nanosecond or faster) and much slower weight/bias reprogramming times (micro- to milliseconds). Traditional deep or recurrent neural architectures require distinct, trainable weight matrices for each layer, making hardware implementation of large networks prohibitive for existing PNN technology.

In digital neural networks, parameter-efficient schemes such as convolutional weight sharing ameliorate hardware constraints. However, PNNs typically provide only a limited set of static weight banks, hampering deployment of larger or deeper architectures using conventional design patterns (Tsuchiyama et al., 28 Oct 2025).

2. Architecture and Time-Multiplexing Mechanism

ReLaX-Net implements a time-multiplexed, weight-tying scheme that transforms L physical weight-bias banks into T effective logical layers (“effective depth”). A fast optical or electronic switch cyclically routes signals through one of the L available weight sets at each time step. The recurrent update is:

$$h[t+1] = \alpha h[t] + f(W_{xh} x[t] + W_{hh}[t] h[t] + b_h[t])$$

where $W_{hh}[t]$ and $b_h[t]$ are selected from $\{\theta^l\}_{l=1}^{L}$ as $\theta[t] = \theta^{(t \bmod L) + 1}$. The nonlinearity $f(\cdot)$ may be realized via device-specific effects (e.g., ReLU, oscillator, saturating photodetector). The scalar $\alpha$ introduces a residual connection for stability.

Although only L physically distinct sets $\theta^l$ exist, the effective network depth achieved is $T = R \times L$ if the sequence of L banks is cycled R times per inference run. This approach trades static layer duplication for high-speed dynamic switching, leveraging PNN platforms’ strengths.

3. Parameter Efficiency and Scaling Behavior

A fully untied $T$-layer MLP with hidden dimension $W_{hh}[t]$0 requires $W_{hh}[t]$1 recurrent parameters. ReLaX-Net reduces this to $W_{hh}[t]$2, giving a parameter reduction factor $W_{hh}[t]$3 for $W_{hh}[t]$4. Table 1 summarizes key scaling properties.

Logical Depth $W_{hh}[t]$5	Physical Layers $W_{hh}[t]$6	Parameter Count	Performance Trend
12	1	$W_{hh}[t]$7	Baseline (stateless RNN)
12	2	$W_{hh}[t]$8	Steep gain over $W_{hh}[t]$9
12	4	$b_h[t]$0	Diminishing returns
12	12	$b_h[t]$1	Fully untied DNN

Performance on SVHN digit classification rises sharply with $b_h[t]$2 (recovering $b_h[t]$3 of total gain) and then saturates for $b_h[t]$4 (error rate drops from $b_h[t]$5 at $b_h[t]$6 to $b_h[t]$7 at $b_h[t]$8, $b_h[t]$9 at $\{\theta^l\}_{l=1}^{L}$0). For NLP tasks (Shakespeare next-token prediction), optimal $\{\theta^l\}_{l=1}^{L}$1 is $\{\theta^l\}_{l=1}^{L}$2 under $\{\theta^l\}_{l=1}^{L}$3, beyond which vanishing gradients degrade results (Tsuchiyama et al., 28 Oct 2025).

4. Hardware Requirements and Integration

ReLaX-Net requires hardware modifications limited to the addition of fast switching elements—such as digital mirror devices, femtosecond fiber couplers, or transistors—to existing PNNs. All trainable weights remain static during inference, switching only when reprogrammed on slow training timescales. Activations and routing switches operate at device-native speeds ($\{\theta^l\}_{l=1}^{L}$4GHz), delivering throughput far beyond what is possible with repeated weight reconfiguration.

During inference, throughput is bottlenecked only by the switching speed and activation propagation, not by statically programmed weight update rates. Training remains decoupled: weight/bias elements are tuned on slow PNN timescales, while inference operations cycle through logical depth at the maximum possible rate (Tsuchiyama et al., 28 Oct 2025).

5. Quantitative Benchmarks and Performance

The SVHN benchmark (image classification, $\{\theta^l\}_{l=1}^{L}$5, $\{\theta^l\}_{l=1}^{L}$6) demonstrates ReLaX-Net’s efficacy:

• $\{\theta^l\}_{l=1}^{L}$7: $\{\theta^l\}_{l=1}^{L}$8 accuracy (baseline, equivalent to a stateless RNN)
• $\{\theta^l\}_{l=1}^{L}$9: $\theta[t] = \theta^{(t \bmod L) + 1}$0 accuracy
• $\theta[t] = \theta^{(t \bmod L) + 1}$1: $\theta[t] = \theta^{(t \bmod L) + 1}$2 accuracy
• $\theta[t] = \theta^{(t \bmod L) + 1}$3: $\theta[t] = \theta^{(t \bmod L) + 1}$4 accuracy
• $\theta[t] = \theta^{(t \bmod L) + 1}$5: $\theta[t] = \theta^{(t \bmod L) + 1}$6 accuracy

Error reduction is most pronounced for small increases in $\theta[t] = \theta^{(t \bmod L) + 1}$7; later gains are marginal. For the Shakespeare task ($\theta[t] = \theta^{(t \bmod L) + 1}$8, $\theta[t] = \theta^{(t \bmod L) + 1}$9), cross-entropy loss falls from $f(\cdot)$0 (at $f(\cdot)$1) to $f(\cdot)$2 (at $f(\cdot)$3). In each case, optimal $f(\cdot)$4 balances parameter count, gradient stability (mitigating vanishing/exploding effects), and hardware resource allocation.

A fixed parameter budget (e.g., $f(\cdot)$5) constrains the interplay of width ($f(\cdot)$6) and depth ($f(\cdot)$7 vs $f(\cdot)$8). For image tasks, wide banks with aggressive time-multiplexing ($f(\cdot)$9) can be superior to many narrow, distinct banks at fixed total parameters.

6. Limitations and Future Directions

ReLaX-Net’s time-multiplexing approach is subject to vanishing/exploding gradients if $\alpha$0 is too small relative to $\alpha$1 and if residual connections ($\alpha$2) are not carefully tuned. Increasing $\alpha$3 at fixed $\alpha$4 gives diminishing expressivity returns; small $\alpha$5 can underutilize physical capacity under a fixed parameter budget. All current studies employ periodic switching; more expressive or adaptive, data-dependent switching is an open avenue. No physical hardware demonstrations currently exist; practical integration of high-speed switches with slow-tunable elements remains a challenge.

Extensions proposed include convolutional ReLaX-Nets, attention-like or reservoir-style probing, and comprehensive energy-throughput scaling analysis. The architecture addresses the “intermediate-scale” PNN regime where per-layer scaling is the limiting factor, rather than overall parameter count (Tsuchiyama et al., 28 Oct 2025).

7. Relation to Other Adaptive Neural Network Schemes

ReLaX-Net sits between a stateless RNN with one physical weight bank and a fully distinct DNN with no weight reuse. It generalizes digital parameter-efficiency motifs to the hardware-constrained neural network paradigm and is grounded in physical timescale separation inherent to PNN platforms. The parameter-tying via fast switching draws inspiration from earlier digital neural architectures while addressing PNN-specific constraints on scale and update speed.

A plausible implication is that as PNN platforms evolve, schemes analogous to ReLaX-Net will be integral in closing the expressivity and performance gap with purely digital architectures, especially where inference throughput and energy efficiency are paramount.