B9803 - Machine Learning

Learning outcomes

The course introduces students to some of the most important Machine Learning predictive models such as regularization methods, tree-based methods, Support Vector Machines and Neural Networks, that can potentially contribute to empirical economics. For each topic will we outline its structure, discuss its pros and cons, and focus attention on the issues associated to its empirical application to several economic problems using the software R.

Course contents

Requirements

The course assumes familiarity with linear regression, statistical inference, and basic econometrics. Students are expected to have prior experience with the statistical software R. Basic knowledge of linear algebra and probability is recommended.

MAIN TOPICS

Part 1: Machine learning for prediction

Model selection and regularization

This part discusses model selection techniques and regularization methods for high-dimensional prediction problems, including stepwise selection, Ridge regression, the LASSO, and Elastic Net.

Tree-based methods

We study decision trees and their extensions to ensemble methods. Topics include bagging, Random Forests, boosting, and their application to nonlinear prediction problems and variable selection.

Neural networks and foundation models

This part introduces artificial neural networks and deep learning methods for structured (tabular) data. We discuss the basic architecture of neural networks, deep neural networks, transformer-based models, and recent foundation models for tabular data, emphasizing their predictive capabilities and practical limitations.

Part 2: Machine learning for causal inference

This part examines how machine learning methods can be integrated with modern econometric techniques to estimate causal effects in high-dimensional settings. We discuss the distinction between prediction and causal inference and study methods that combine flexible machine learning algorithms with valid statistical inference.

Post-Double Selection Lasso

We introduce Post-Double Selection Lasso as a method for selecting control variables while preserving valid inference on treatment effects.

Double Machine Learning

We study the Double Machine Learning framework for estimating causal effects using machine learning methods to model nuisance parameters while retaining desirable statistical properties for causal inference.

Readings/Bibliography

James, Witten, Hastie and Tibshirani, An Introduction to Statistical Learning, Springer 2021 (second edition).

Additional material (scientific papers, tutorials, Python notebooks) will be provided during classes.

Teaching methods

Teaching combines the introduction of the theoretical foundations of ML methods with empirical applications implemented in R. During the course, students will apply the techniques discussed in class to real datasets, learning how to train and test ML models, interpret the results, and assess their validity. Attendance is not compulsory but is strongly recommended, as classroom discussions and practical exercises are designed to reinforce students' understanding of the material.

Assessment methods

The exam is joint with Module 1.

The exam tests the ability to apply the methods learnt to simulated or real data, using R, the acquired knowledge of the theoretical concepts and the ability to interpret estimation results in the light of the underlying theory.

The exam consists of a group project plus discussion.

The structure of the final project should be the following:

1. Data description and motivation

State your final objective: outcome, predictors, causal effect of interest, ...

Explore the data: use tools such as plots, PCA, clustering.

2. Setup and assess a prediction model

Prediction: Setup and assess forecasting models

Techniques: Linear/logistic regression, Discriminant analysis, KNN, PCR, PLS, stepwise selection, regularization methods (Ridge, LASSO, Elastic Net, ...), splines, trees, DNN, ...

Include any additional relevant information about the final project.

3. Data

Use your own data and provide it (during the course we will illustrate several interesting data repositories on the Web: Kaggle, UCI Machine Learning, and others). Data must allow exploration and prediction, and possibly the estimation of some causal effect.

Data don’t need to be huge!

4. Methods and techniques

You should use the most important methods and techniques from class (not all). Make sure the data and your goals are compatible.

Work in groups of 3 to 5 students. Let the instructors know the groups’ composition as soon as possible.

The final project must be handed in 5 days before the discussion, including: (i) The data in a format easily readable by R, (ii) The R code that reads the data, does the computations and outputs the results; (iii) The pdf document that illustrates the project.

The file must contain at most 20 pages including tables and figures.

5. Assessment

The final project assessment will take into account the difficulty posed by data cleaning and preparation, and it will also weight the project assessment and the oral discussion.

The maximum possible score is 30 cum laude. The grade is graduated as follows:

<18 failed
18-23 sufficient
24-27 good
28-30 very good
30 e lode excellent

6. Grade rejection

Students can reject the grade obtained at the exam once. To this end, he/she must email a request to the instructors within the date set for registration. The instructors will confirm reception of the request within the same date.

Rejection is intended with respect to the whole exam, whose grade is the weighted average of the grades obtained in the oral discussion and the final project. If the grade is rejected, the student must retake the oral discussion only.

7. Use of AI tools

Students may use artificial intelligence (AI) tools to support the preparation of the project report, including assistance with programming, data analysis, writing, and editing. However, the report submitted must accurately reflect the students' own work and understanding of the project.

During the oral presentation, students are expected to demonstrate a thorough understanding of every aspect of their project, including the methodological choices, implementation of the machine learning techniques, interpretation of the empirical results, and the limitations of the analysis. The use of AI tools does not exempt students from this requirement. Students are therefore responsible for verifying the correctness of any AI-generated content and for ensuring that they can fully explain and justify all elements of their work.

Students are encouraged to explicitely acknowledge any substantial use of AI tools in the report, indicating how such tools contributed to the project.

Teaching tools

We will discuss the different techniques and the output they generate using the statistical software R and the RStudio IDE.

Office hours

See the website of Sergio Pastorello