87450 - MODELS AND NUMERICAL METHODS IN PHYSICS

Conoscenze e abilità da conseguire

By the end of the course, the student will have acquired the theoretical and numerical skills for the study of entropic properties of datasets, more or less structured. Theoretical skills will be acquired in the area of intersection between Complex Dynamical Systems Theory, Information Theory and Statistical Mechanics. Please refer to the syllabus for detailed topics. As far as numerical skills are concerned, the student will experiment with Python the numerical implementation of algorithms for the estimation of entropy, relative entropy and entropic production for stochastic processes on finite alphabets. Some applications in the area of natural language, gene sequences and (time permitting) in the area of human mobility data will be explored.

Contenuti

"..a large Language Model is just something that compress part of the Internet ....and then it dreams about...."

(Andrej Karpathy, https://youtu.be/zjkBMFhNj_g?si=_CjyJdSOKhvyVZXk)

Entropy and Diffusion in Physics: Theory and Applications

The course will focus on developing the mathematical and computational tools for understanding two main concepts in Physics, along with their applications:

1. Entropy: We will explore the close relationship between Entropy, Information, and Compression, starting from stochastic processes over finite alphabets.
2. Diffusion: We will introduce and explore diffusive processes in physics and their recent remarkable applications in the so-called Probabilistic Diffusion Models (e.g., stable diffusion).

Methods and techniques lie at the intersection of Dynamical Systems, Statistical Mechanics, and Information Theory. The course will combine traditional lectures with numerical investigations (in Python).

This course is a natural extension of the Complex Systems Physics course, sharing approaches, techniques, and results.

This course will start by recalling the main definitions and results about “Entropy and Information” developed in the course Complex System Physics.


This online program will be refined over time, but here is a preliminary list of topics:

· Entropy, Information and Coding: recall of basic notions from Complex Systems Physics course

· Coding and Entropy: Recall of the main notions of entropy for processes over finite alphabet. Introduction to Lempel-Ziv algorithms for parsing and coding, highlighting their relation to entropy and information theory(8h)

· From Compression to Embedding: Introduction to Variational Auto-Encoders (VAE): Introduction to modern techniques in data compression and representation, specifically focusing on Variational Auto-Encoders and their applications.(8h)

· Numerical Methods (16h):

I. Numerical schemes to integrate ODE: the properties of Runge-Kutta schemes. (2h)

II. Symplectic numerical schemes to integrate Hamiltonian system: expected error for integrable and chaotic systems, properties of symplectic maps (2h).

III. Numerical schemes for dynamical systems on Lie groups with applications (2h).

IV. Chaotic systems and definition of Ljapunov exponents: numerical scheme to evaluate Ljapunov exponents with applications (2h).

V. Numerical integration of stochastic differential equations: Wiener process and Ito integral definition, Ito formula and stochastic equivalence of stochastic process (4h).

VI. Fokker-Planck equation: properties and numerical integration (2h).

VII. Introduction to stochastic processes on manifolds (2h)

· Introduction to Diffusion Models in Physics: Comprehensive introduction to diffusion models in physics, covering Langevin dynamics, Fokker-Planck equations, and diffusion phenomena. This section will also include an exploration of random walks and random processes on graphs or networks, providing a deeper understanding of the mathematical and physical principles behind diffusion. (8h)

· Probabilistic Diffusion Models: Examination of the theoretical foundations of probabilistic diffusion models, their implementation, and applications in Artificial Intelligence. We will explore the generative methods based on diffusion, focusing on canonical examples for image generation conditioned on text, and more recent applications to urban data. (8h)

Testi/Bibliografia

Here just the basic reference books used in the course. All sources, books and papers, will be available to students in digital format:

• Notes "Entropy. Information  and Large Language Models", M.- Degli Esposti (2024)
  
• Cover, M.T., and Thomas, A.J.: Elements of Information Theory. John Wiley & and Sons,1991.
• Andrej Karpathy's Lecture (YouTube)

Metodi didattici

Lectures and Numerical Simulations (with Python)

Modalità di verifica e valutazione dell'apprendimento

to be defined

Strumenti a supporto della didattica

Blackboard and Python

Orario di ricevimento

Consulta il sito web di Mirko Degli Esposti