The Differentiable Auditory Loop (DAL): An ML Framework for Hyper-Personalized Hearing Aids

Abstract: Conventional hearing aids rely on fixed, frequency-dependent amplification and compression to manage reduced sensitivity, which often fails to provide sufficient listening support in complex environments, such as situations with multiple speakers (the ``cocktail party'' problem). To more comprehensively address the underlying encoding dysfunctions of hearing loss, we introduce the Differentiable Auditory Loop (DAL), a new open-source framework for personalized hearing aid design and fitting. Our first implementation of DAL incorporates CARFAC, a differentiable model of human cochlear function, which we ported to JAX, to optimize a deep neural network to match impaired auditory neural activity patterns with a normal-hearing reference. To build a hearing aid with the fine-grained spectro-temporal signal processing required, we adopt SEANet, a waveform-to-waveform fully convolutional UNet generator. We fine-tune the network by comparing the outputs of a CARFAC model fitted to normal hearing with that of a CARFAC model fitted to match each subject's individual hearing impairment. The comparison is done using loss functions derived from the respective CARFAC neural activity pattern (NAP) outputs and stabilized auditory images (SAIs), the latter providing a 2D representation that captures phase-insensitive temporal structure in the auditory nerve output. Through gradient descent, the SEANet model learns to both denoise the input and compensate for the hearing loss modelled by the impaired CARFAC model. Across neural-representation and signal-fidelity metrics, the DAL-optimized SEANet model outperformed the tested master hearing aid (MHA) baselines. The DAL framework provides a practical path toward model-based, machine-learning-driven personalization of hearing aid signal processing. Next steps include hardware deployment to enable real-world clinical testing.

• The paper introduces DAL, an ML framework that leverages biophysical cochlear models to optimize hearing aid algorithms with end-to-end differentiability.
• It integrates CARFAC and SEANet to simulate neural activity patterns and minimizes discrepancies between impaired and normal hearing through tailored NAP and SAI losses.
• Experimental results demonstrate that DAL’s approach outperforms conventional methods by enhancing both neural and perceptual fidelity in noisy conditions.

Differentiable Auditory Loop (DAL): Advancing ML-Driven Hyper-Personalized Hearing Aids

Motivation and Framework Architecture

The Differentiable Auditory Loop (DAL) framework addresses the fundamental limitations of conventional hearing aids, which primarily employ fixed-frequency amplification and compression based on pure-tone audiometry. These approaches offer limited compensation for cochlear dysfunctions and often fail under complex acoustic conditions—specifically, in suprathreshold and noisy listening environments. DAL introduces an end-to-end differentiable platform, leveraging sophisticated computational models of inner ear physiology to facilitate the design of hyper-personalized hearing aid algorithms. By optimizing waveform processing to minimize the discrepancy between impaired and reference neural patterns, DAL targets direct remediation of encoding dysfunctions rather than symptomatic threshold compensation.

Figure 1: Core DAL architecture—two cochlear models (normal and impaired) guide optimization of a deep learning hearing aid to minimize discrepancies in neural activity patterns.

Central to the DAL framework is CARFAC, a biophysically grounded and differentiable model of cochlear mechanics and auditory nerve transduction. For ML-driven pre-compensatory audio transformation, DAL utilizes SEANet, a lightweight, causally-constrained waveform-to-waveform convolutional architecture suitable for deployment within stringent latency and compute constraints of hearing aid hardware.

Biophysical Modeling: CARFAC Integration

CARFAC (Cascade of Asymmetric Resonators with Fast-Acting Compression) is designed to emulate the major structures and nonlinear dynamics of the cochlea, including basilar membrane mechanical filtering, outer hair cell feedback-driven compression, and the spiking properties of multiple classes of auditory nerve fibers. DAL extends CARFAC to include explicit multi-fiber innervation of inner hair cells, and ports the model to JAX, enabling efficient automatic differentiation for gradient-based training of neural networks.

Figure 2: CARFAC biophysical model explicitly mirrors cochlear anatomy, yielding detailed neural activity patterns that mirror clinical auditory nerve responses.

The output—termed neural activity patterns (NAPs)—represents spatio-temporal activity along the tonotopic axis. These patterns serve as canonical representations of healthy hearing. For impaired individuals, CARFAC parameters are modified (e.g., simulating decreased outer hair cell undamping) to reflect subject-specific loss profiles as estimated from audiometric and physiological data.

Perceptually-Aligned Losses: NAPs and SAIs

Matching impaired neural responses to those predicted for normal hearing requires objective, differentiable losses. DAL exploits two distinct domains:

• NAP loss: Mean absolute error between healthy and impaired NAPs enforces pointwise temporal and spectral alignment, placing strong constraints on phase fidelity and waveform coherence.
• SAI loss: Recognizing phase misalignment and the potential for temporal jitter in impaired responses, Stabilized Auditory Images (SAIs)—computed via short-term autocorrelation of NAPs—provide phase-insensitive 2D summaries capturing spectro-temporal features salient for perception.

A spectrum of loss functions is evaluated, including L1, partial normalization, and structural similarity metrics on both NAPs and SAIs, as well as hybrid combinations.

Figure 3: Temporal evolution of NAPs and corresponding SAI frames for a voiced segment, illustrating underlying auditory structure captured by the cochlear model.

Experimental Design and Simulation Protocols

A controlled experimental pipeline is established using the LibriSpeech corpus. Mild sensorineural hearing loss is simulated by reducing outer hair cell health within CARFAC, generating a characteristic audiogram with 30+ dB HL in the 3–4 kHz region.

Figure 4: Simulated audiogram demonstrating reduced gain in the impaired CARFAC model, reflecting typical sensorineural loss.

Benchmarking involves comparisons across three conditions:

• Unprocessed (noisy) baseline
• Master Hearing Aid (MHA): multi-band dynamic range compression configured by the NAL-NL2 algorithm or retrained variants
• DAL-trained SEANet: optimized with CARFAC-driven losses

Input signals span calibrated clean and noisy (white noise added at random SNRs from -5 to 10 dB) speech segments. Model evaluation is conducted over 1000 four-second utterances, split by speaker to ensure independence between training and test sets.

Quantitative and Qualitative Results

Neural Pattern Recovery

DAL-trained SEANet models demonstrate robust recovery of auditory neural representations, outperforming both unprocessed and MHA-processed conditions in both impairment-matched and reference-matched NAP/SAI metrics. Notably, MHA often degrades neural representations relative to the noisy baseline, while SEANet optimized with NAP-domain L1 loss achieves the greatest fidelity, reducing L1 distances and elevating correlation coefficients.

Figure 5: Visual comparison of NAPs across conditions—SEANet more closely reconstructs the normal-hearing neural structure in noisy environments than MHA.

Figure 6: Quantitative evaluation of NAP-domain metrics—SEANet consistently improves both L1 loss and Pearson correlation against reference.

Perceptual Representation Fidelity

In the SAI domain, the advantages of DAL-driven SEANet architectures persist. MHA improves correlation but worsens L1 distance. SEANet variants, particularly those trained on SAI-based metrics (e.g., SSIM), attain highest SAI-domain structural similarity and reduced error, indicating superior capture of perceptually relevant features.

Figure 7: SAI-domain quantitative metrics—SEANet with SSIM and partial normalization losses shows clear superiority over both unprocessed and MHA conditions.

Across all domains, improvements in scale-invariant signal-to-distortion ratio (SI-SDR) further confirm enhanced neural signal fidelity, contingent on the use of appropriate training losses.

Practical and Theoretical Implications

DAL represents a substantive advancement over fixed-architecture audiometric compensation, establishing a neuro-computational approach to hyper-personalization in hearing restoration. Its design allows for:

• Direct remediation of specific cochlear dysfunctions: Personalized cochlear models permit targeting of distinct impairments, including but not limited to outer hair cell deficit, inner hair cell loss, and synaptopathy.
• Richer signal transformation via deep learning: SEANet’s waveform-to-waveform architecture circumvents the limitations of conventional filterbank approaches, enabling nonlinear, context-sensitive pre-processing tailored to neural recovery objectives.
• Model-agnostic extensibility: While the framework currently exploits CARFAC and SEANet, it is designed to accommodate alternate cochlear models and DNN configurations as needed for evolving theoretical or hardware constraints.

Clinically, DAL paves the way for hearing aids that closely recapitulate normal auditory nerve signaling, with the prospect for multidimensional fitting beyond pure-tone thresholds—potentially improving speech-in-noise intelligibility and user adaptation in dynamic acoustic settings.

Limitations and Future Directions

The present work applies DAL to simulated OHC-related loss; real-world extension will require expansion to multi-factorial impairments and validation in large-scale clinical cohorts. Establishing correspondence between neural similarity metrics and perceptual outcomes (i.e., subjective intelligibility/quality measures) is essential. The tradeoff between aggressive denoising and preservation of weak, but salient, speech cues remains unresolved and merits systematic user-centered investigation—potentially via preference-tunable algorithms.

Adoption of DAL-based pipelines in commercial devices necessitates continued optimization of computational efficiency, as well as algorithmic robustness across languages, accents, and environmental conditions.

Conclusion

DAL introduces a rigorously biophysically and perceptually grounded ML platform for hearing aid algorithm development, integrating end-to-end differentiability through physiologically accurate cochlear modeling. Experimental evidence confirms substantial advances over conventional hearing aid fitting, particularly in reconstructing impaired neural representations under adverse listening conditions. The framework establishes a foundation for future AI-driven, hyper-personalized auditory prostheses, with broad implications for both research and clinical practice.