Brief Review
Finally we reached one of the final boss in the field. Yes, this book allows you to deep dive even into the floor of Mariana Trench (It even still so hard for me to follow sometimes). This is the book for you if you really want to deep dive into the mathematical foundations behind AI, machine learning, and deep learning.
Here, Calin answer the deeper questions in deep learning architecture. Why can a NN approximate complicated functions? How much information can a NN represent? What actually happens to information when it passes through layers? What kind of geometry is created by a neural network? And mathematically, why do these learning mechanisms work?
The first part builds what I would call the foundation mechanics of neural networks. You will learn deeply about activation functions, cost functions, algorithms for finding minima, abstract neurons, and neural networks themselves. These are things that we often use almost automatically when building a model. We choose ReLU, define a loss function, choose an optimizer, and start training. But here, you actually study the mathematics behind those choices.
Then it goes deeper into approximation theory. This is one of the parts that makes this book very interesting to me. Neural networks are often called universal function approximators, but what does that sentence actually mean mathematically? Calin discusses approximation theorems, universal approximators, one-dimensional learning problems, and even exact learning. Instead of simply accepting that neural networks can approximate almost any function, you start understanding the mathematical reasoning behind that statement.
And then comes probably one of the most unique parts of this book: information processing.
The book treats neural networks not only as function approximators, but also as systems that represent and process information. You will learn about information representation and information capacity assessment. This brings concepts from information theory into the discussion and makes you think differently about what a neural network is actually doing. Instead of seeing a hidden layer as just another matrix multiplication followed by an activation function, you can start asking how much information that layer can carry and how that information is transformed through the network.
After that, things become even more interesting with the geometric theory of neural networks.
Calin discusses output manifolds and something called neuromanifolds. This is already quite far from the usual practical deep learning textbook. Here, neural networks can be studied using geometry. The outputs, parameters, and probability distributions associated with neural networks can form mathematical spaces with their own geometric properties. Concepts such as manifolds, metrics, and information geometry start appearing.
This part was particularly interesting to me because it shows that deep learning is connected with much more mathematics than only calculus and linear algebra. Once you go deep enough, you begin finding connections with geometry, information theory, probability theory, optimization, approximation theory, and other areas of mathematics.
The book eventually brings all of these mathematical foundations back into more familiar deep learning architectures. It discusses pooling, Convolutional Neural Networks, Recurrent Neural Networks, classification systems, generative models, and stochastic networks. But again, the perspective is different. The objective is not simply to tell you how to implement a CNN or RNN. The objective is to mathematically understand what these architectures are doing.
That is probably the biggest difference between this book and many other AI books I have read.
With a practical book, you might write several lines of PyTorch, train the model, see that your validation loss decreases, and be satisfied that the model works. With this book, Calin almost forces you to ask another question: Why does it work?
And then after you understand one answer, you probably find another deeper question underneath it.
There are equations everywhere. Sometimes you may spend more time understanding three pages of this book than reading an entire chapter from a more practical AI book. You might need to derive something yourself, revisit calculus, check a theorem, or open another mathematics book just to understand one section. But I think that is exactly the point of reading this book.
The Target Readers
I definitely would not recommend this as your first book about AI or machine learning. If you are still trying to understand what a neural network is, what gradient descent does, or what backpropagation means, there are much easier places to start. Raschka’s books, introductory machine learning courses, and more practical deep learning books will probably give you a much better foundation first. This book is more suitable for people who already understand machine learning and deep learning but are no longer satisfied with only knowing how to use them.
I think it is particularly valuable if you want to become an AI or machine learning researcher, work on theoretical machine learning, or simply want to understand the deepest mathematical foundations behind neural networks. You should already be relatively comfortable with calculus, linear algebra, probability, and statistics. Knowledge of optimization will also help a lot. And when you reach the geometric sections, some additional mathematics will probably be necessary.
If your objective is only to build AI applications, train existing architectures, fine-tune models, or use an API, this book is probably much deeper than you need. But if you are the type of person who reads that “neural networks are universal approximators” and immediately asks, “Okay, but can you prove it?”, then this book is probably written for you.
For me, this book sits somewhere between computer science and pure mathematics. It does not only teach you another deep learning architecture. It tries to explain the mathematical structure underneath the entire idea of neural networks. And the deeper you go, the more AI stops looking like black magic and starts looking like a huge mathematical system that we are still trying to fully understand.