87450 - Models and Numerical Methods in Physics

Learning outcomes

"At the end of the course the student will acquire the tools to build up dynamical models for the evolution of the classical physical systems formed by interacting particles under the influence of external fields. He/she will be able to use numerical techniques for the solution of the corresponding differential equation even in the case of fluctuating fields. In particular, in the limit of a large number of particles the kinetic and the fluid approximations will be developed; in the case of long range interactions the average field equations will be considered, together with self-consistent solutions and collision models based on stochastic processes."

Course contents

"..a large Language Model is just something that compress part of the Internet ....and then it dreams about...."

(Andrej Karpathy, https://youtu.be/zjkBMFhNj_g?si=_CjyJdSOKhvyVZXk)

Entropy and Diffusion in Physics: Theory and Applications

The course will focus on developing the mathematical and computational tools for understanding two main concepts in Physics, along with their applications:

1. Entropy: We will explore the close relationship between Entropy, Information, and Compression, starting from stochastic processes over finite alphabets.
2. Diffusion: We will introduce and explore diffusive processes in physics and their recent remarkable applications in the so-called Probabilistic Diffusion Models (e.g., stable diffusion).

Methods and techniques lie at the intersection of Dynamical Systems, Statistical Mechanics, and Information Theory. The course will combine traditional lectures with numerical investigations (in Python).

This online program will be refined over time, but here is a preliminary list of topics:

· Entropy and Information:

(1) Review of Probability Theory and Dynamical Systems: an overview of the fundamental concepts and mathematical frameworks necessary for understanding entropy and diffusion in physical systems.

(2) Shift Spaces over Finite Alphabets, Ergodicity, and Covering: Examination of the structure and dynamics of shift spaces, ergodic properties, and covering theorems relevant to stochastic processes.

(3) Entropy of a Random Variable: Detailed exploration of entropy, its significance, and its calculation for random variables in various contexts.

(4) The Shannon-McMillan-Breiman (SMB) Theorem: Study of the SMB theorem and its implications, with examples including Shannon’s source coding theorem.

· Coding and Entropy: The Lempel-Ziv Parsing and Coding: Introduction to Lempel-Ziv algorithms for parsing and coding, highlighting their relation to entropy and information theory.

· Relative Entropy and Entropy Production: Investigation of relative entropy, its significance in statistical mechanics, and the concept of entropy production in (irreversible) physical processes. Irreversibility in Physics.

· From Compression to Embedding: Introduction to Variational Auto-Encoders (VAE): Introduction to modern techniques in data compression and representation, specifically focusing on Variational Auto-Encoders and their applications.

· Introduction to Diffusion Models in Physics: Comprehensive introduction to diffusion models in physics, covering Langevin dynamics, Fokker-Planck equations, and diffusion phenomena. This section will also include an exploration of random walks and random processes on graphs or networks, providing a deeper understanding of the mathematical and physical principles behind diffusion.

· Probabilistic Diffusion Models: Examination of the theoretical foundations of probabilistic diffusion models, their implementation, and applications in Artificial Intelligence. We will explore the generative methods based on diffusion, focusing on canonical examples for image generation conditioned on text, and more recent applications to urban data.

Readings/Bibliography

Here just the basic reference books used in the course. All sources, books and papers, will be available to students in digital format:

• Notes "Entropy. Information and Large Language Models", M.- Degli Esposti (2024)
  
• Cover, M.T., and Thomas, A.J.: Elements of Information Theory. John Wiley & and Sons,1991.
• Andrej Karpathy's Lecture (YouTube)

Teaching methods

Lectures and Numerical Simulations (with Python)

Assessment methods

to be defined

Teaching tools

Blackboard and Python

Office hours

See the website of Mirko Degli Esposti