Alice's Adventures in a Differentiable Wonderland -- Volume I, A Tour of the Land (2404.17625v2)

Abstract: Neural networks surround us, in the form of LLMs, speech transcription systems, molecular discovery algorithms, robotics, and much more. Stripped of anything else, neural networks are compositions of differentiable primitives, and studying them means learning how to program and how to interact with these models, a particular example of what is called differentiable programming. This primer is an introduction to this fascinating field imagined for someone, like Alice, who has just ventured into this strange differentiable wonderland. I overview the basics of optimizing a function via automatic differentiation, and a selection of the most common designs for handling sequences, graphs, texts, and audios. The focus is on a intuitive, self-contained introduction to the most important design techniques, including convolutional, attentional, and recurrent blocks, hoping to bridge the gap between theory and code (PyTorch and JAX) and leaving the reader capable of understanding some of the most advanced models out there, such as LLMs and multimodal architectures.

• The paper presents a comprehensive introduction to differentiable models by simplifying neural network design fundamentals.
• It reviews critical operations such as convolutions, transformers, and graph layers for efficient multi-dimensional data processing.
• The integration of theoretical insights with practical coding examples equips readers to build advanced model architectures.

The paper "Alice's Adventures in a Differentiable Wonderland -- Volume I, A Tour of the Land" is positioned as a comprehensive and self-contained introduction to the design of modern neural networks, albeit referred to as "differentiable models" due to the historical complexities associated with the term "neural." This first volume, spanning 250 pages, seeks to equip readers with the foundational knowledge necessary to understand and construct efficient building blocks for processing multi-dimensional data.

The core areas of focus in this volume include:

1. Convolutions: Understanding how convolutional operations can be utilized to efficiently handle spatial data is essential, given their widespread use in image processing tasks.
2. Transformers: These models have revolutionized the handling of sequential data, particularly in natural language processing, by enabling the modeling of long-range dependencies without recurrent architectures.
3. Graph Layers: The integration of graph-based representations allows models to leverage relationships and structures within data that can be naturally described as graphs, such as social networks or molecules.
4. Modern Recurrent Models: This includes advancements like linearized transformers and structured state-space models, which aim to improve efficiency and performance over traditional recurrent neural networks (RNNs).

The paper attempts to balance theoretical insights with practical coding examples, making it a valuable resource for those with a foundational understanding of machine learning and linear algebra. Although some preliminary concepts are revisited to ensure a comprehensive understanding, the text assumes that the reader is already somewhat familiar with these areas.

Significantly, the paper highlights its evolving nature, indicative of the rapid advancements in the field of machine learning and deep learning. This volume deliberately omits more advanced topics such as generative modeling, explainability, prompting, and intelligent agents, which are intended to be explored in subsequent publications on a companion website.

Moreover, the work originated from a refined compilation of lecture notes from a course taught at Sapienza University, which further underscores its pedagogical roots aimed at educating and equipping data science students and practitioners alike.

Overall, the paper is highly regarded on social media platforms like Twitter, although it hasn't yet garnered citations or significant recognition on other scholarly platforms. This indicates potential for broad interest and discussion among the AI and machine learning communities, particularly among those interested in introductory and intermediate-level material.