Hybrid Real- and Complex-valued Neural Network Architecture

Abstract: We propose a \emph{hybrid} real- and complex-valued \emph{neural network} (HNN) architecture, designed to combine the computational efficiency of real-valued processing with the ability to effectively handle complex-valued data. We illustrate the limitations of using real-valued neural networks (RVNNs) for inherently complex-valued problems by showing how it learnt to perform complex-valued convolution, but with notable inefficiencies stemming from its real-valued constraints. To create the HNN, we propose to use building blocks containing both real- and complex-valued paths, where information between domains is exchanged through domain conversion functions. We also introduce novel complex-valued activation functions, with higher generalisation and parameterisation efficiency. HNN-specific architecture search techniques are described to navigate the larger solution space. Experiments with the AudioMNIST dataset demonstrate that the HNN reduces cross-entropy loss and consumes less parameters compared to an RVNN for all considered cases. Such results highlight the potential for the use of partially complex-valued processing in neural networks and applications for HNNs in many signal processing domains.

• The paper introduces a hybrid neural network architecture that blends real-valued and complex-valued processing via novel domain conversion functions and complex activation functions.
• The architecture leverages an advanced Neural Architecture Search strategy to optimize parameter efficiency and enhance robustness, particularly under noisy conditions.
• Experimental results on the AudioMNIST dataset demonstrate that the hybrid approach outperforms traditional real-valued networks, effectively preserving phase information and managing noise.

Hybrid Real- and Complex-Valued Neural Network Architecture

Introduction

The paper "Hybrid Real- and Complex-valued Neural Network Architecture" (2504.03497) presents a novel approach to neural network design that integrates both real-valued and complex-valued data processing to address complex-valued problems more efficiently. The authors introduce a Hybrid Neural Network (HNN) architecture, systematically marrying real-valued processing's computational efficiency with complex-valued data handling's nuanced capabilities. This combination is managed through novel domain conversion functions and complex-valued activation functions to enhance generalization and parameter efficiency.

Deep learning applications in signal processing domains frequently involve complex-valued data, but conventional real-valued neural networks (RVNNs) struggle with efficiently processing such data due to the intrinsic phase information loss. In response, the authors propose hybrid architectures that leverages complex-valued neural networks (CVNNs) alongside real-valued pathways, optimizing both types of processing paths through systematic architecture search techniques.

Architecture Exploration

The proposed HNN architecture is inspired by RVNN's implicit complex-valued operations observed when handling complex inputs. A key insight derived from RVNN experimentation is that neural networks naturally exhibit inefficient complex-valued convolution patterns under real-valued constraints. The paper establishes that adding true complex-valued processing paths and domain conversion functions can substantially enhance network efficiency.

Figure 1: Non-ordered weights for the first convolutional layer.

Figure 2: Plot of reordered and normalized weights for each layer's convolutions for the RVNN experiment. The arrows indicate how the output of each layer is followed by the input of another. Distinct regions in the weight patterns are highlighted with dashed blue lines.

The architecture depends on building blocks structuring both real- and complex-valued inputs and outputs. The HNN splits input into real and imaginary constituents, processing them via these complex pathways before merging or transitioning across domains using conversion functions like Cartesian or polar mappings and others. Complex activation functions also enhance model expressiveness beyond equivalent real-valued solutions.

Architecture Search Process

The foundation of the proposed architecture is an advanced Neural Architecture Search (NAS) approach using Optuna. This iterative process customizes the HNN according to specific task requirements, striking a balance between model complexity and computational efficiency. The phases include dependency check, block selection, refinement, and optimization of activations and hyperparameters.

Figure 3: HNN architecture search process.

To ensure practicality, the total number of parameters is minimized, and the loss function is optimized. The NAS also introduces diverse domain conversion functions, facilitating information flow between real and complex domains. The resultant network is highly parameter-efficient while maintaining accuracy, especially in signal processing applications.

Experimental Validation

The authors validate their proposed HNN using the AudioMNIST dataset for spoken digit classification. Comparing the performance of the HNN against an RVNN configuration optimized through similar NAS processes, the results favor the HNN, especially under high noise conditions. The ability of the HNN to effectively handle noise and exhibit robustness indicates its superior performance in signal processing tasks.

Figure 4: Optimised HNN Architecture for -5 dB SNR noise. c is output channels, k is kernel, n is number of groups and p is dropout rate.

Figure 5: Optimised RVNN model. c is output channels, k is kernel, n is number of groups and p is dropout rate.

Test results underscore the efficiencies gained through hybrid processing. The HNN showcases better loss reduction and parameter efficiency compared to its RVNN counterpart, particularly in noisy environments. This hybrid approach yields promising results across signal-based domains where phase and magnitude processing are pivotal.

Implications for AI and Future Developments

The HNN's innovative blending of real- and complex-valued domains opens new possibilities in neural network design by engaging complex attribute processing which traditional real-valued models miss. The approach can be generalized across various fields like audio synthesis, radar signal processing, and biomedical signal analysis. New research could further refine domain conversion functions, explore alternate NAS methods, and expand applications to more complex signal domains.

The future of AI will likely see the proliferation of hybrid models, optimizing networks to better mimic natural signal pathways, thereby enhancing both algorithmic understanding and computational performance. As hybrid architectures evolve, they may inspire novel AI solutions offering unparalleled efficiency in processing complex datasets.

Conclusion

The paper "Hybrid Real- and Complex-valued Neural Network Architecture" presents a compelling advancement in neural network design by strategically integrating real- and complex-valued processing pathways. Demonstrating both theoretical innovation and practical value, the HNN suggests a promising direction for future AI-driven signal processing applications. Through robust architecture search and tailored optimization, this paper advances the field by proposing a model that efficiently manages complexity in inherently complex-valued problems.