Analog Alchemy: Neural Computation with In-Memory Inference, Learning and Routing (2412.20848v1)

Abstract: As neural computation is revolutionizing the field of AI, rethinking the ideal neural hardware is becoming the next frontier. Fast and reliable von Neumann architecture has been the hosting platform for neural computation. Although capable, its separation of memory and computation creates the bottleneck for the energy efficiency of neural computation, contrasting the biological brain. The question remains: how can we efficiently combine memory and computation, while exploiting the physics of the substrate, to build intelligent systems? In this thesis, I explore an alternative way with memristive devices for neural computation, where the unique physical dynamics of the devices are used for inference, learning and routing. Guided by the principles of gradient-based learning, we selected functions that need to be materialized, and analyzed connectomics principles for efficient wiring. Despite non-idealities and noise inherent in analog physics, I will provide hardware evidence of adaptability of local learning to memristive substrates, new material stacks and circuit blocks that aid in solving the credit assignment problem and efficient routing between analog crossbars for scalable architectures.

• The paper introduces a novel filament control technique that enhances memristor bit precision, improving on-chip learning dynamics as shown on datasets like MNIST.
• The paper presents an online credit assignment framework in spiking neural networks using the e-prop rule to bolster temporal learning efficiency.
• The paper unveils a Mosaic architecture with reconfigurable halide perovskite memristors that enable energy-efficient in-memory computation and effective spike routing.

Analysis of Neural Computation with In-Memory Inference, Learning, and Routing

The paper "Analog Alchemy: Neural Computation with In-Memory Inference, Learning and Routing" by Yiğit Demirağ introduces an innovative approach to neural computation using memristive devices that integrate memory and computation. This integration seeks to address the energy inefficiencies inherent in traditional von Neumann architectures, which separate processing and memory units, in neural network implementations.

Summary of Key Contributions

1. Enhanced Bit Precision of Memristors: The research highlights challenges with limited precision in binary RRAMs for on-chip learning and suggests a device programming technique that precisely controls filament growth in these devices. This approach augments bit precision, ensuring more effective training dynamics. The method is validated through simulations on standard datasets such as MNIST, showcasing its application potential in tangible AI solutions.
2. Online Credit Assignment in Spiking Neural Networks (SNNs): A framework employing a statistical model of Phase Change Material (PCM) crossbar arrays introduces novel methodologies to tackle credit assignment challenges in SNNs. The adoption of the e-prop learning rule serves as a foundational element, supplemented by the development of PCM-trace for synaptic eligibility traces. This model enhances temporal credit assignment efficiency in recurrent SNNs despite memristor non-idealities.
3. Reconfigurable Memristor Material: The paper discusses the discovery of a halide perovskite memristor capable of switching between volatile and non-volatile modes—a significant advancement in memory technology that facilitates streamlined integration across various computational frameworks. This material offers record endurance performances and supports differentiated dynamic and static neural parameter implementation.
4. Mosaic Architecture for In-Memory Computing: The proposed Mosaic architecture capitalizes on a systolic array structure supporting small-world connectivity. This structure enhances spike routing efficiency compared to existing hardware platforms, underscoring the architecture's potential energy efficiency in edge computing applications. The architecture was validated through real-chip fabrications and benchmarked against various AI tasks.

Implications and Speculations

The implications of this research are profound, both practically and theoretically. Practically, the integration of memristors can lead to highly efficient AI systems with lower energy consumption, potentially redefining applications in edge computing where power resources are constrained. Theoretically, it challenges current paradigms in neuromorphic engineering that rely extensively on digital architectures and calls for a reconsideration of learning algorithms tailored to analog substrates.

Future research might further explore:

• Material Innovations: Optimization of memristor materials for consistency and performance across applications, potentially leveraging novel compounds or fabrication techniques to enhance device stability and precision.
• Algorithmic Co-Design: Development of new learning algorithms specifically designed for analog substrates, extending beyond current adjustments of digital methods such as backpropagation.

Conclusion

Yiğit Demirağ’s dissertation represents a significant step towards achieving energy-efficient neural computation by leveraging the unique properties of memristive devices to unify processing and memory. This work provides a compelling case for rethinking the physical substrates of neural networks, with potential long-term impacts on both neuromorphic computing and practical AI system design. The contributions to hardware design, algorithm development, and materials science unveil new chapters in the quest for efficient, scalable AI solutions.