ElectrodeNet-CS: Deep Learning CI Sound Coding

• The paper introduces a differentiable deep-learning framework that unifies envelope estimation and channel selection via a custom topK layer.
• It employs a four-layer fully connected DNN that produces ACE-compatible electrode patterns, ensuring gradient propagation through selected channels.
• Integration with audio-visual speech enhancement demonstrates significant improvements in objective and subjective intelligibility metrics under adverse noise conditions.

ElectrodeNet-CS (ECS) is a deep-learning-based sound coding strategy for cochlear implants, designed to serve as a fully differentiable surrogate for the envelope detection and channel selection (CS) stages of the Advanced Combination Encoder (ACE) framework. Unlike conventional methods, ECS embeds the top-N channel selection directly within the neural network graph, unifying envelope estimation and CS in an end-to-end optimization process. This approach enables seamless integration with modern speech enhancement (SE) techniques—including multimodal, audio-visual modules—facilitating improved speech intelligibility, especially under adverse noise conditions (Lin et al., 19 Aug 2025, Huang et al., 2023).

1. Motivation and Theoretical Foundations

Traditional cochlear implant sound coding strategies like ACE utilize handcrafted signal processing pipelines comprising bandpass envelope extraction followed by N-of-M channel selection, where only the channels with the largest amplitudes are stimulated at each frame. The original ElectrodeNet framework replaced the envelope extraction stage with a neural network but retained the separate, non-differentiable CS step, potentially introducing envelope–CS mismatches. ElectrodeNet-CS addresses this by folding the CS operation directly into the DNN via a differentiable, learnable “topK” layer. This design enables the model to concentrate learning capacity on channels most informative for speech, producing N-of-M compatible electrode patterns that match or slightly exceed ACE in both objective and subjective intelligibility metrics (Huang et al., 2023).

2. Network Architecture and Mathematical Formulation

ECS employs a fully connected feedforward neural network (DNN) comprising four dense layers with progressively decreasing hidden sizes: [1024, 512, 256, 22]. The input representation is a spectral–temporal matrix $X \in \mathbb{R}^{L\times M}$ (typically $L=65$ STFT frequency bins and $M=22$ channels as in ACE). The flow per time frame $t$ is:

• $h_1 = \mathrm{ReLU}(W_1\, \mathrm{vec}(X_t) + b_1)$, $h_1 \in \mathbb{R}^{1024}$
• $h_2 = \mathrm{ReLU}(W_2 h_1 + b_2)$, $h_2 \in \mathbb{R}^{512}$
• $h_3 = \mathrm{ReLU}(W_3 h_2 + b_3)$, $h_3 \in \mathbb{R}^{256}$
• $L=65$0, $L=65$1 (raw envelope predictions)

A custom TopK (“CS function”) layer computes the binary mask $L=65$2 by selecting the $L=65$3 largest entries (commonly $L=65$4), zeroing others: $L=65$5. All TopK operations are performed in a differentiable manner using e.g., PyTorch’s topk, ensuring gradient propagation through selected channels (Lin et al., 19 Aug 2025, Huang et al., 2023).

3. Training Procedures and Data Regimes

The ECS model is supervised by paired datasets acquired from the clean speech processed by ACE, comprising $L=65$6 across multiple speakers and sentences (e.g., 320 TMHINT sentences for pretraining). During end-to-end training in AVSE-ECS (audio-visual SE + ECS), additional data augmentation includes the injection of 100 non-speech noise types at varied SNR levels ($L=65$7 dB to $L=65$8 dB), and visual features from mouth ROIs extracted by Mediapipe landmarks.

Optimization is performed using the Adam optimizer, typically with a learning rate of $L=65$9 for 100 epochs and batch sizes of 16–128 samples, though precise hyperparameters are not always detailed. During joint training, ECS weights are frozen and only the upstream SE parameters (e.g., UNet with cross-attention for AVSE) are updated, ensuring that the electrodogram loss directly guides the SE module (Lin et al., 19 Aug 2025).

4. Objective Functions and Loss Design

Two principal loss terms are used in joint training:

1. Spectrogram reconstruction loss:

$M=22$0

where $M=22$1 is the reference clean spectrogram.

2. Electrodogram (intelligibility-driven) loss:

$M=22$2

which supervises the ECS mapping toward producing ACE-compatible or optimal stimulation patterns. The overall loss is a weighted sum, $M=22$3 with typical weights $M=22$4, $M=22$5.

In models incorporating AVSE as a front-end to ECS (AVSE-ECS), the joint objective incentivizes not only waveform fidelity but also downstream intelligibility as reflected directly in the electrode patterns (Lin et al., 19 Aug 2025).

5. Quantitative Performance and Comparative Analysis

ECS demonstrates performance at parity with, or slightly above, ACE under both clean and adverse noise conditions. For ECS alone (no SE front-end) on noisy input, STOI is 0.4870, ESTOI is 0.2073, and NCM is 0.3258. When paired with an AVSE front-end (jointly trained), AVSE-ECS achieves substantial improvements (e.g., STOI=0.6305, ESTOI=0.3899, NCM=0.5211 under $M=22$6 to $M=22$7 dB test noise), outperforming all baselines including ACE alone or audio-only SE+ECS. Objective correlations between ECS and ACE exceed 0.999 for STOI and NCM across test conditions (Lin et al., 19 Aug 2025, Huang et al., 2023).

Subjective sentence recognition tests confirm that ECS matches or slightly outperforms ACE, with paired t-tests showing no significant difference (e.g., ACE=13.0% vs. ECS=13.6% at $M=22$8 dB SNR for $M=22$9 selection), indicating maintenance or improvement of perceptual speech intelligibility (Huang et al., 2023).

Method	STOI	ESTOI	NCM
ACE	0.4870	0.2067	0.3262
ECS	0.4870	0.2073	0.3258
AVSE-ECS (joint)	0.6305	0.3899	0.5211

6. Integration with Audio-Visual Speech Enhancement

The AVSE-ECS architecture positions ECS as a differentiable, fixed back-end atop a multimodal SE front-end. Visual features derived from a frozen encoder (TCN + ResNet-18) are input to a cross-attentive NCSN++ UNet, which enhances the spectrogram before reconstruction and envelope extraction. Joint training with $t$0 provides a direct learning signal from the electro-neural space, leveraging lip movements and visual speech cues to improve spectral envelope extraction and thus downstream electrode stimulation patterns. Freezing ECS during such training stabilizes stimulation mapping and focuses learning on the SE module (Lin et al., 19 Aug 2025).

7. Implications and Future Directions

Embedding channel selection within the neural coding model, as in ECS, eliminates post hoc envelope–CS mismatches and extends differentiability across the entire sound coding pipeline. This enables joint optimization with state-of-the-art SE, including audio-visual models, and provides a platform for open-ended, data-driven cochlear implant research. The strong objective and subjective agreement with ACE, even in challenging noise, suggests ECS as a viable, future-ready drop-in for practical CI systems (Huang et al., 2023, Lin et al., 19 Aug 2025). A plausible implication is that this modular differentiable framework may facilitate broader integration of real-world multimodal signals—visual, spatial, contextual—into clinical auditory prosthesis design.