87450 - MODELS AND NUMERICAL METHODS IN PHYSICS

Conoscenze e abilità da conseguire

By the end of the course, the student will have acquired the theoretical and numerical skills for the study of entropic properties of datasets, more or less structured. Theoretical skills will be acquired in the area of intersection between Complex Dynamical Systems Theory, Information Theory and Statistical Mechanics. Please refer to the syllabus for detailed topics. As far as numerical skills are concerned, the student will experiment with Python the numerical implementation of algorithms for the estimation of entropy, relative entropy and entropic production for stochastic processes on finite alphabets. Some applications in the area of natural language, gene sequences and (time permitting) in the area of human mobility data will be explored.

Contenuti

"..a large Language Model is just something that compress part of the Internet ....and then it dreams about...."

(Andrej Karpathy, https://youtu.be/zjkBMFhNj_g?si=_CjyJdSOKhvyVZXk)


The course will be focused on developing the mathematical and computational tools for understanding the close relation between Entopy, Information and Compression, starting from stochastic processes over finite alphabet, together with a discussion of some concrete applications to Large Language Models and other Generative A.I. Models.

Methods and Techniques lie in the intersection between Dynamical Systems, Statistical Mechanics and Information Theory.The course will combine lectures at the blackboard with numerical investigations (in Python).

This online program will be refined over time, but here a first list of topics:

-Review of probability theory/dynamical systems.

-Shift spaces over finite alphabets. ,Ergodicity and covering

- Entropy of a random variable.

- The Shannon-McMillan-Breiman (SMB) theorem. Example: Shannon’s source coding theorem.

-Entropy and coding–asymptotic optimality and Shannon’s theorem.

-Coding and Entropy: the Lempel-Ziv parsing and coding.

-Relative Entropy and Entropy Production

- Byte Pair Encoding

- Deep Compression: Convolutional and Recurrent (LSTM) networks, variational Auto-Encoder and Transformer (GPT)

- Deep Entropy and applications: GAN (Generative Adversial Network), Probabilistic Diffusion Models

Testi/Bibliografia

Here just the basic reference books used in the course. All sources, books and papers, will be available to students in digital format:

• Notes "Entropy. Information  and Large Language Models", M.- Degli Esposti (2024)
• Shields, P.C. The Ergodic Theory of Discrete Sample Paths. Graduate Studies in Mathematics, AMS 1996.
• Cover, M.T., and Thomas, A.J.: Elements of Information Theory. John Wiley & and Sons,1991.
• Andrej Karpathy's Lecture (YouTube)

Metodi didattici

Lectures and Numerical Simulations (with Python)

Modalità di verifica e valutazione dell'apprendimento

to be defined

Strumenti a supporto della didattica

Blackboard and Python

Orario di ricevimento

Consulta il sito web di Mirko Degli Esposti