96753 - Superior Mechanical Statistics

Learning outcomes

At the end of the course the student: has an in-depth understanding of machine learning and inference algorithms for solving inverse problems and learning from data; is able to use tools from statistical mechanics to characterize their typical performance, discuss fundamental limits of inference, and identify computational and statistical phase transitions; is able to analyze the training dynamics of neural networks using dynamical theories within the framework of statistical mechanics.

Course contents

The course introduces some of the main mathematical models used in the theoretical study of modern machine learning through the tools of statistical mechanics. It begins with the classical models that laid the foundations of statistical learning theory and progresses to current research topics in artificial intelligence, highlighting the deep connections between statistical physics, probability theory, and machine learning.

 

The topics covered include:

• Fundamentals of statistical mechanics for disordered systems and inference problems. In particular, the Spiked Wigner model as a paradigmatic framework for the theoretical analysis of Principal Component Analysis (PCA) and of statistical and computational phase transitions.
• Teacher-Student models for linear and logistic regression, with emphasis on the analysis of generalization performance.
• Random Feature models and kernel methods, including the study of their statistical properties in the high-dimensional regime.
• Learning from structured data through Gaussian mixture models, which provide a mathematical description of the latent structure of real-world datasets, including the MNIST dataset.
• Multilayer neural networks: analysis of online learning dynamics using methods from statistical mechanics, leading to the derivation of the time evolution of the generalization error.

Throughout the course, the relationships among these models and their role in the development of the mathematical theory of artificial intelligence will be emphasized, with particular attention to the analysis of performance, generalization capabilities, and the dynamics of learning algorithms.

 

Prerequisites

Students are expected to have a solid background in linear algebra, probability theory, mathematical analysis, and differential calculus.

In addition, it is strongly recommended that students have attended at least one of the following courses:

• Statistical Mechanics (General Curriculum);
• Mathematics for Complex Systems (Applied Curriculum).

These courses provide the statistical mechanics background we will use throughout the course.

Readings/Bibliography

Course notes and selected research papers will be made available throughout the course via the University's Virtuale platform.

Main references:

• H. Nishimori, Statistical Physics of Spin Glasses and Information Processing.
• M. Mézard and A. Montanari, Information, Physics, and Computation.
• A. Engel and C. Van den Broeck, Statistical Mechanics of Learning.
• Selected research articles related to the topics covered in the course.

Suggested reading:

• M. E. J. Newman, Networks: An Introduction. Oxford University Press, 2010.
• A. Decelle, F. Krzakala, C. Moore, and L. Zdeborová, "Asymptotic Analysis of the Stochastic Block Model for Modular Networks and Its Algorithmic Applications," Physical Review E, 84(6), 066106 (2011).
• L. Zdeborová and F. Krzakala, "Statistical Physics of Inference: Thresholds and Algorithms," Advances in Physics, 65(5), 453–552 (2016).
• E. Gardner and B. Derrida, "Optimal Storage Properties of Neural Network Models," Journal of Physics A: Mathematical and General, 21, 271–284 (1988).

Teaching methods

The course consists primarily of blackboard lectures, during which the main mathematical derivations will be presented and the theoretical results will be discussed and interpreted. Selected topics will also be illustrated through numerical examples and computational simulations, providing further insight into the behavior of the models and the validity of the theoretical predictions.

Assessment methods

The assessment consists of an oral examination.

During the examination, students are expected to demonstrate a thorough understanding of the theoretical tools developed throughout the course, the ability to reproduce and explain the key mathematical derivations, and the capacity to critically discuss the main mathematical models presented, including their underlying assumptions, theoretical results, and fields of application. The evaluation will take into account the student's command of the mathematical formalism, clarity of exposition, and ability to establish connections among the different topics covered in the course.

Teaching tools

Course materials and supporting teaching resources will be made available through the University's Virtuale platform.

Office hours

See the website of Federica Gerace