Multimodal Optoelectronic Reservoir Networks

• Multimodal optoelectronic reservoir networks are hybrid systems that use optical front-ends for nonlinear, high-dimensional transformations and electronic back-ends for state storage and readout.
• They enable efficient processing of heterogeneous data modalities—such as images, video, and speech—by leveraging integrated optical and electronic components in unified architectures.
• Fusion mechanisms like spectral slicing, temporalization, and sensor-level dual-mode acquisition enhance performance while reducing training costs and system complexity.

Multimodal optoelectronic reservoir networks are physical reservoir computing systems in which optical components perform part of the sensing, encoding, or high-dimensional transformation, while electronic components perform state storage, synchronization, feedback, digitization, or readout. In the recent literature, the term multimodal spans several distinct cases: multiple optically pre-processed views of one communication waveform, heterogeneous data streams such as image, video, and speech, coherent quadrature-selective measurements, wavelength-parallel channels, and device-level co-processing of voltage and light inputs (Ranzini et al., 2020, Iqbal et al., 28 May 2026, Henaff et al., 2024, Xiang et al., 11 Dec 2025, Boer et al., 27 Aug 2025). The unifying RC principle is that the reservoir dynamics are largely fixed by physical structure and operating point, while only a final linear readout is trained.

1. Definition and conceptual scope

A precise definition appears in the deep binarized photonic RC literature: a multimodal optoelectronic reservoir network is a physical RC system that uses an optical front-end to perform high-dimensional, parallel, and nonlinear transformations of inputs, while an electronic back-end handles state storage and readout; multimodal refers to support for heterogeneous data modalities, and optoelectronic highlights the hybrid nature in which optical modulation and propagation produce reservoir dynamics while electronic sensing and digital logic implement state updates, inter-layer coupling, and readout training (Iqbal et al., 28 May 2026). This definition is broad enough to include systems in which optics dominate the internal transformation as well as systems in which optics merely precondition the input before an electronic reservoir.

The short-reach optical communications receiver demonstrated with spectral slicing is a canonical example of the latter case. Its receiver intentionally partitions processing between optics and electronics, using a wavelength selective switch and multiple photodiodes to create simultaneous inputs from several spectral slices of the same received waveform, and then feeding those slices to a shared electronic echo state network. The system is explicitly optoelectronic because functionality is shared between the optical front-end and the electronic reservoir; it is also explicitly not a photonic reservoir (Ranzini et al., 2020). By contrast, the pulsed femtosecond-laser system with phase encoding and homodyne readout is a delay-based photonic reservoir whose optoelectronic character arises at the readout and feedback interface: optical phase encoding, interferometric quadrature detection, electrical low-pass filtering, delayed feedback, and offline ridge readout are jointly responsible for the computational dynamics (Henaff et al., 2024).

The term multimodal is also used at the sensor-material level. In microscale MAPbI$_3$ perovskite networks, a single volatile device can process voltage-only inputs, light+voltage inputs, or a combined feature vector obtained by concatenating the two transformed state sets (Boer et al., 27 Aug 2025). In all-optically controlled ZnO memristor arrays, multimodal fusion is realized by wavelength-selective bipolar and parallel coding, where different optical channels induce opposite-signed state changes and can encode multiple sources into one reservoir (Hu et al., 13 Feb 2026). Accordingly, the literature does not restrict multimodality to semantically different sensors; it also includes multiple spectral, spatial, temporal, wavelength, or quadrature views of the same underlying signal.

2. Architectural families and physical substrates

The reported platforms span communication receivers, free-space photonic processors, pulsed coherent reservoirs, integrated silicon photonics, optoelectronic memristors, FPGA-electro-optic loops, multicore fibers, and perovskite in-sensor arrays.

Platform	Modalities / channels	Characteristic mechanism
Spectral-slice receiver (Ranzini et al., 2020)	1–4 optical spectral slices	WSS + PD front-end feeding a shared electronic ESN
Deep binarized photonic RC (Iqbal et al., 28 May 2026)	Image, video, speech	DMD modulation, random scattering, CMOS detection, time-multiplexed deep layers
Pulsed phase-encoded reservoir (Henaff et al., 2024)	Temporal pulse modes, optical quadratures	EOM phase encoding, balanced homodyne detection, delayed electronic feedback
Silicon DPRC (Xiang et al., 11 Dec 2025)	WDM channels, skeleton feature branches	MRR nonlinearity, true time-delay lines, all-optical inter-layer coupling
ZnO AOCM reservoir (Hu et al., 13 Feb 2026)	Dual-wavelength bipolar / parallel optical coding	Wavelength-programmable PPPC/NPPC in memristor crossbars
Hybrid EOM + FPGA reservoir (Kumar et al., 2021)	Multi-channel digital injection	MZM nonlinearity plus FPGA delay lines and FIR connectivity
Active multicore-fiber reservoir (Chekhovskoy et al., 16 Jun 2026)	Spatial cores, temporal masks	Inter-core coupling, Kerr nonlinearity, saturable gain, delayed feedback
Perovskite in-sensor network (Boer et al., 27 Aug 2025)	Voltage and light inputs	Ionic drift-diffusion transients in microscale MAPbI$_3$ crossbars

Two architectural poles recur. One pole uses optics primarily as a front-end transform and leaves recurrent state evolution to electronics. The short-reach receiver and the hybrid EOM+FPGA platform belong to this class: the former uses passive spectral slicing followed by a software/FPGA-like ESN, while the latter uses a Mach–Zehnder electro-optic modulator as the nonlinear element and realizes delay, filtering, and connectivity digitally in FPGA fabric (Ranzini et al., 2020, Kumar et al., 2021). The opposite pole uses optical or material physics for both nonlinear mapping and memory. The DMD–diffuser–CMOS system uses optical scattering and intensity detection, the pulsed comb reservoir uses coherent interference and finite-bandwidth coupling, the silicon DPRC uses free-carrier dynamics and true optical delay lines, the multicore-fiber design uses coupled nonlinear propagation with saturable gain, and the memristive and perovskite systems use photo-induced and ionic material dynamics as the reservoir substrate (Iqbal et al., 28 May 2026, Henaff et al., 2024, Xiang et al., 11 Dec 2025, Chekhovskoy et al., 16 Jun 2026, Hu et al., 13 Feb 2026, Boer et al., 27 Aug 2025).

A second axis of variation concerns depth and parallelism. In the free-space DMD platform, depth is virtualized in time by repeatedly reusing the same physical optics for $L$ layers per global update (Iqbal et al., 28 May 2026). In the silicon DPRC, reservoir layers are cascaded optically without O–E conversion, and wavelength-division multiplexing provides parallel channels with shared long-term memory hardware (Xiang et al., 11 Dec 2025). In the multicore-fiber reservoir, parallelism is spatial: each core contributes a distinct nonlinear transformation, reducing reliance on long serial temporal masks (Chekhovskoy et al., 16 Jun 2026).

3. State dynamics, encoding, and readout

Despite their physical diversity, these systems share a common RC decomposition into input encoding, fixed recurrent or fading-memory dynamics, and a trainable linear readout. In the multi-input ESN used for spectral-slice equalization, the reservoir state follows

$$x[n] = \alpha \cdot \tanh\!\big(W^{in} u[n] + W^{res} x[n-1]\big) + (1-\alpha)\cdot x[n-1],$$

with $\alpha = 0.9$, and the output uses skip connections,

$y[n] = W^{out}_{res} x[n] + W^{out}_{in} u[n].$

Only $W^{out}$ is trained; recurrent and input weights remain fixed, with $W^{in} \sim \mathcal{U}(-1,1)$, $W^{res}$ nonzeros drawn from $\mathcal{N}(0,1)$, 98% sparsity, and $_3$0 activation (Ranzini et al., 2020).

In the deep binarized photonic system, the layerwise state is optical in origin but electronically stored and binarized. Its update has the leaky form

$_3$1

with binarized states $_3$2, a linear ridge readout over concatenated layer states, and a leak schedule linearly varying from $_3$3 to $_3$4 across layers. Optical scattering is modeled by $_3$5, photodetection by $_3$6, and hardware-compatible discretization by basket encoding with $_3$7 and $_3$8 (Iqbal et al., 28 May 2026).

The pulsed phase-encoded homodyne reservoir replaces square-law intensity readout with quadrature-sensitive interferometric readout. Its encoded per-node optical field is

$_3$9

and the measured nonlinear state is modeled as

$L$0

Finite detector bandwidth produces coupling between adjacent pulses, so the reservoir graph is controlled not only by delayed feedback but also by the ratio $L$1 (Henaff et al., 2024).

In silicon DPRC, the dominant nonlinearity is neither a software activation nor a detector response but the free-carrier dynamics of silicon microring resonators:

$L$2

This carrier-induced detuning modulates the ring transfer function and provides short-term memory, while true optical delay lines provide longer shared memory. The deep state update is written as

$L$3

Again, the readout is trained by single-shot ridge regression (Xiang et al., 11 Dec 2025).

Material-dynamic reservoirs exhibit the same RC pattern with different state variables. In the ZnO memristor system, a suitable physical-node approximation is

$L$4

capturing persistent photoconductivity, negative persistent photoconductivity, and tunable relaxation (Hu et al., 13 Feb 2026). In microscale perovskites, measured fading memory is fitted by

$L$5

with illumination increasing $L$6 and shifting the dynamics toward slower diffusion-dominated decay (Boer et al., 27 Aug 2025).

Across these implementations, the readout is typically trained in closed form,

$L$7

or by an equivalent ridge-regression formulation; a notable exception is the perovskite study, which uses Adam in PyTorch for the linear classifier while explicitly noting ridge regression as a common RC alternative (Iqbal et al., 28 May 2026, Xiang et al., 11 Dec 2025, Henaff et al., 2024, Kumar et al., 2021, Boer et al., 27 Aug 2025).

4. Modalities and fusion mechanisms

The most elementary multimodal pattern is multi-view fusion of one source. In the short-reach receiver, a wavelength selective switch defines up to four narrowband optical spectral slices positioned relative to the received carrier and sidebands; each slice is detected by its own photodiode, and the shared ESN ingests all slices simultaneously as a multi-input reservoir. Fusion occurs through the random input projections and reservoir dynamics rather than through parallel sub-reservoirs (Ranzini et al., 2020). A related idea appears in multicore-fiber RC, where a scalar input is distributed across cores by a spatial mask and, optionally, across mask positions by a temporal mask; the concatenated per-core intensities serve as the feature vector, and spatial coupling provides cross-channel interaction (Chekhovskoy et al., 16 Jun 2026).

A second pattern is temporalization of non-temporal data. In the DMD-based deep photonic RC, each MNIST image is decomposed into four horizontal strips, HOG features are reduced to 25 components per strip, and the four segments are fed sequentially; video frames are converted to HOG features reduced by PCA to 1000 components and streamed to the reservoir; TI-46 speech is converted to a Lyon cochleagram whose columns are presented sequentially (Iqbal et al., 28 May 2026). This does not merely serialize the data; it deliberately induces a short sequence so that the reservoir’s memory and nonlinearity can act on modality-specific structure.

A third pattern is measurement-space multimodality. In the pulsed comb system, homodyne readout with tunable LO phase selects different quadratures $L$8, enabling amplitude–phase combinations beyond intensity-only detection. The same paper explicitly describes concatenating state vectors measured at multiple LO phases, $L$9, to construct an augmented multimodal state (Henaff et al., 2024). The optoelectronic receiver of (Ranzini et al., 2020) similarly demonstrates that modalities need not be heterogeneous in the semantic sense; partially redundant frequency-localized observables are sufficient if they carry complementary information about the channel distortion.

A fourth pattern is channelized physical parallelism. In silicon DPRC, distinct input streams can be mapped onto separate wavelength carriers and processed concurrently by different MRR reservoirs sharing the same delay hardware; the NTU RGB+D skeleton pipeline uses 15 input feature channels, and the Lorenz experiment uses three wavelengths $$x[n] = \alpha \cdot \tanh\!\big(W^{in} u[n] + W^{res} x[n-1]\big) + (1-\alpha)\cdot x[n-1],$$0 nm, $$x[n] = \alpha \cdot \tanh\!\big(W^{in} u[n] + W^{res} x[n-1]\big) + (1-\alpha)\cdot x[n-1],$$1 nm, and $$x[n] = \alpha \cdot \tanh\!\big(W^{in} u[n] + W^{res} x[n-1]\big) + (1-\alpha)\cdot x[n-1],$$2 nm (Xiang et al., 11 Dec 2025). In the ZnO memristor reservoir, multimodal fusion is realized through dual-wavelength parallel coding: face data are encoded at 405 nm, fingerprint data at 650 nm, and both are projected onto the same devices simultaneously so that the coupled device physics performs feature extraction and fusion in one reservoir (Hu et al., 13 Feb 2026).

A fifth pattern is sensor-level dual-mode acquisition. The MAPbI$$x[n] = \alpha \cdot \tanh\!\big(W^{in} u[n] + W^{res} x[n-1]\big) + (1-\alpha)\cdot x[n-1],$$3 in-sensor network operates with voltage-only, light+voltage, or concatenated multimodal state vectors. For MNIST, voltage-based and light-based mappings are concatenated into a single feature vector; for N-MNIST, each pixel’s four-frame temporal sequence is mapped into measured current states and then classified by a linear readout (Boer et al., 27 Aug 2025). The hybrid FPGA–electro-optic platform describes an analogous system-level route to multimodality: separate masks and input gains can be assigned to distinct modalities, early fusion can occur at the electro-optic modulator input, and late fusion can be obtained by concatenating modality-specific reservoir states at the readout (Kumar et al., 2021).

5. Demonstrated tasks and performance regimes

In short-reach optical communications, multimodal optoelectronic RC has been demonstrated as an equalizer for 32 GBd on–off keying transmitted over up to 80 km of SMF. The single-PD broadband baseline gives acceptable BER only for back-to-back and 5 km SMF. With four spectral slices and four photodiodes, both the RC equalizer and a 2-layer FNN reach 80 km of SSMF with BER below the KP4 FEC threshold $$x[n] = \alpha \cdot \tanh\!\big(W^{in} u[n] + W^{res} x[n-1]\big) + (1-\alpha)\cdot x[n-1],$$4; the FFE baseline with 32 taps at 2 sps is inferior in reach. Partial slicing remains effective: the $$x[n] = \alpha \cdot \tanh\!\big(W^{in} u[n] + W^{res} x[n-1]\big) + (1-\alpha)\cdot x[n-1],$$5 and $$x[n] = \alpha \cdot \tanh\!\big(W^{in} u[n] + W^{res} x[n-1]\big) + (1-\alpha)\cdot x[n-1],$$6 two-slice cases both achieve approximately 55 km reach with BER below KP4 FEC (Ranzini et al., 2020).

In free-space deep photonic RC, a five-layer DMD–scattering–CMOS system reports 96.0% accuracy on KTH action recognition in the S1 scenario with 10,000 neurons, 95.2% on MNIST with 3,500 neurons, and 99.4% on TI-46 spoken digits with 500 neurons. The reported per-layer processing speed is approximately 1000 fps, and the system is described as operating at Gigabit-per-second processing rates (Iqbal et al., 28 May 2026). The pulsed phase-encoded homodyne reservoir addresses different benchmarks: on NARMA-$$x[n] = \alpha \cdot \tanh\!\big(W^{in} u[n] + W^{res} x[n-1]\big) + (1-\alpha)\cdot x[n-1],$$7, performance is evaluated by Pearson correlation and degrades monotonically as $$x[n] = \alpha \cdot \tanh\!\big(W^{in} u[n] + W^{res} x[n-1]\big) + (1-\alpha)\cdot x[n-1],$$8 increases; simulations show that increasing the number of virtual nodes from $$x[n] = \alpha \cdot \tanh\!\big(W^{in} u[n] + W^{res} x[n-1]\big) + (1-\alpha)\cdot x[n-1],$$9 to $\alpha = 0.9$0 significantly reduces error. The same platform predicts both the laser mean power fluctuation $\alpha = 0.9$1 and the central frequency fluctuation $\alpha = 0.9$2 from pump intensity noise, using datasets of 10,000 points with 8,000 used for training and 2,000 for validation (Henaff et al., 2024).

Integrated silicon DPRC extends multimodal optoelectronic RC into deep, all-optical inter-layer pipelines. On NTU RGB+D skeleton recognition, an 8-layer DPRC with $\alpha = 0.9$3 neurons per layer achieves 98.1% top-1 accuracy on Cross-View and 96.7% on Cross-Subject with approximately 1.1M trainable parameters. A prototype with three layers and 75 neurons per layer, evaluated on eight representative actions with 7,534 samples, reaches 95.6% on X-View and 92.2% on X-Sub. Additional reported results include 97.3% train and 97.4% test accuracy on Iris classification with $\alpha = 0.9$4, and Lorenz chaotic-system prediction NMSEs of 0.006, 0.008, and 0.018 for $\alpha = 0.9$5, $\alpha = 0.9$6, and $\alpha = 0.9$7, respectively (Xiang et al., 11 Dec 2025).

Device-dynamic reservoirs show similar multimodal gains. In the ZnO all-optically controlled memristive reservoir, bipolar coding raises four-letter word-recognition accuracy to approximately 93% after 100 epochs, compared with approximately 76% for unipolar coding; the confusion between “LOCK” and “LUCK” falls from 48% to 5%. On Lorenz prediction with 800 points scaled to $\alpha = 0.9$8, the average NRMSE across $\alpha = 0.9$9 is approximately 0.12 for bipolar coding versus approximately 0.31 for unipolar coding. In dual-factor authentication, unipolar-input reservoirs achieve 70% accuracy, bipolar-input reservoirs 91%, and single-reservoir parallel coding 90.5% while using 512 feature values rather than 1024 (Hu et al., 13 Feb 2026).

Hybrid electronic–photonic reservoirs also deliver competitive temporal-processing performance. The electro-optic modulator + FPGA system reports NARMA-10 NRMSEs of 0.142 over 1000 testing steps and 0.148 over 25,000 steps, Santa Fe laser prediction NMSE of $y[n] = W^{out}_{res} x[n] + W^{out}_{in} u[n].$0 with 950 nodes, and isolated spoken-digit testing WER close to 0.34% (Kumar et al., 2021). The active multicore-fiber reservoir, evaluated numerically on Mackey–Glass one-step-ahead prediction, reduces validation NRMSE from 0.5956 for a single-core baseline to 0.0651 in a seven-core configuration at 40 GHz with equal temporal masks; at 1 GHz, spatial-only encoding reaches 0.0323 validation NRMSE in the seven-core case and 0.0157 in the nineteen-core case (Chekhovskoy et al., 16 Jun 2026).

In in-sensor multimodal RC based on halide perovskites, multimodal concatenation of voltage- and light-derived features reaches 92.3 ± 0.1% on MNIST with square mapping, 92.6 ± 0.1% on row mapping, and 95.3 ± 0.1% for Row+Column concatenation. On modified N-MNIST video, the baseline accuracy is 73.2 ± 0.1%, while reservoir networks achieve 79.2 ± 0.1% in voltage mode, 84.3 ± 0.1% in light mode, and 87.8 ± 0.1% in multimodal mode. The multimodal network therefore exceeds the linear classifier baseline by 3.1% on images and 14.6% on video (Boer et al., 27 Aug 2025).

6. Complexity, latency, robustness, and open issues

A central attraction of this class is that multimodality is often obtained without fully training a large nonlinear model. In the short-reach receiver, the RC readout trains approximately $y[n] = W^{out}_{res} x[n] + W^{out}_{in} u[n].$1 parameters plus bias; for $y[n] = W^{out}_{res} x[n] + W^{out}_{in} u[n].$2 slices and $y[n] = W^{out}_{res} x[n] + W^{out}_{in} u[n].$3, that is approximately 504 trained parameters. The FNN baseline, by contrast, must explicitly encode channel memory in a 5-symbol $y[n] = W^{out}_{res} x[n] + W^{out}_{in} u[n].$4 8 sps $y[n] = W^{out}_{res} x[n] + W^{out}_{in} u[n].$5 4-PD input window, leading to approximately 5152 trainable weights. The same study states that optical slicing is passive and linear, adds negligible latency, and that reservoir inference is a one-step recurrent update plus readout (Ranzini et al., 2020). Similar training asymmetry appears across the other platforms: ridge regression in the DMD–scattering system, single-shot ridge in silicon DPRC, linear readout in multicore-fiber RC, and linear classifiers atop experimentally measured perovskite states (Iqbal et al., 28 May 2026, Xiang et al., 11 Dec 2025, Chekhovskoy et al., 16 Jun 2026, Boer et al., 27 Aug 2025).

Latency and throughput vary widely because the physical reservoirs operate on very different timescales. The DMD-based deep photonic processor reports approximately 1000 fps per layer and end-to-end latency $y[n] = W^{out}_{res} x[n] + W^{out}_{in} u[n].$6 for an $y[n] = W^{out}_{res} x[n] + W^{out}_{in} u[n].$7-layer deep step (Iqbal et al., 28 May 2026). Silicon DPRC reports a 50 ps virtual-node interval corresponding to 20 Gbps per reservoir stream, measured memory around 1.45 ns, and a computational density of 334.25 TOPs/mm$y[n] = W^{out}_{res} x[n] + W^{out}_{in} u[n].$8 that remains consistent with depth because inter-layer processing is fully optical (Xiang et al., 11 Dec 2025). By contrast, the perovskite system uses $y[n] = W^{out}_{res} x[n] + W^{out}_{in} u[n].$9 ms in voltage mode and $W^{out}$0 ms in light mode, while the ZnO memristor demonstrations use pulses and sampling on the order of $W^{out}$1–$W^{out}$2 s; these timescales are consistent with near-/in-sensor classification and edge inference rather than ultrafast communications (Boer et al., 27 Aug 2025, Hu et al., 13 Feb 2026).

Robustness is strongly substrate-dependent. The communications receiver reports BER averaged over 10 independent measurements and notes that BER versus distance is not monotonic because OSNR varies across spools, connectors, and EDFA preamp conditions; nevertheless, the reservoir remains effective under those variations and even under partial slice sets (Ranzini et al., 2020). The DMD-scattering system argues that basket encoding preserves similarity in Hamming space and that decorrelated macro-pixel selection mitigates redundancy and drift (Iqbal et al., 28 May 2026). The silicon DPRC prototype reports stable action-recognition accuracies over 5.5 hours and identifies the main constraints as WDM crosstalk, thermal drift, and self-pulsation near 8 mW pump power (Xiang et al., 11 Dec 2025). The ZnO memristor array retains bipolar photoresponse after 9 months under individual, alternating, and synchronous light modes, and repeatability maps show small $W^{out}$3 across repeated parallel-coding combinations (Hu et al., 13 Feb 2026). The perovskite study reports only minor deterioration under realistic measurement noise but also identifies environmental stability and ionic hysteresis as long-term materials issues (Boer et al., 27 Aug 2025).

Several recurring misconceptions are resolved by the published implementations. First, optoelectronic does not imply a photonic reservoir internal state: the short-reach equalizer is explicitly an electronic ESN with optical preconditioning (Ranzini et al., 2020). Second, multimodal does not require distinct sensing domains such as audio and vision; the literature uses the term for spectral slices, quadratures, wavelength channels, spatial cores, and separate voltage/light transforms of the same sample (Henaff et al., 2024, Xiang et al., 11 Dec 2025, Chekhovskoy et al., 16 Jun 2026, Boer et al., 27 Aug 2025). Third, all-optical depth does not eliminate systems concerns: the silicon DPRC still requires precise thermal tuning and crosstalk control, the pulsed homodyne system remains sensitive to LO phase drifts and was limited experimentally to $W^{out}$4 virtual nodes by available low-loss coaxial cable, and the FPGA-electro-optic reservoir remains bounded by ADC/DAC bandwidth, quantization, and EOM bias stability (Xiang et al., 11 Dec 2025, Henaff et al., 2024, Kumar et al., 2021).

The immediate research trajectory suggested by these reports is not a single dominant hardware solution but a convergent design space. One branch pushes more computation into optics through deep photonic layering, WDM, and shared optical memory (Iqbal et al., 28 May 2026, Xiang et al., 11 Dec 2025). Another branch exploits material dynamics for sensor-proximate or edge inference with native multimodal fusion (Hu et al., 13 Feb 2026, Boer et al., 27 Aug 2025). A third branch uses hybrid digital control to make multimodality programmable at the mask, delay, and coupling level (Kumar et al., 2021). Taken together, these systems position multimodal optoelectronic reservoir networks as a technically heterogeneous but conceptually coherent family of low-training-cost architectures for communications equalization, multimedia recognition, dynamical prediction, authentication, and in-sensor spatiotemporal processing.