79213 - Impact Evaluation of Entrepreneurial and Policy Chioces

Learning outcomes

At the end of the course the student is expected to know the approaches and tools used to evaluate economic and social programs or decisions, and the main statistical techniques used in impact evaluation. In particular the student should be able: - to design the evaluation of a program; - to choose the appropriate design for impact evaluation basing on the available data; - to understand the results of impact evaluation.

Course contents

Background

Student are required to have some background on statistical inference and linear model or be willing to learn the necessary tecniques. For these concepts, refer to the textbook "Introduzione all'econometria" (Introduction to econometrics) listed in the bibliography, Chapters 1-5

 

Topics

1)  Introduction to Evaluation: Concepts and objectives, distinguishing between evaluation and monitoring

2) Evaluation Design: Policy design and evaluative questions, implementation evaluation versus impact evaluation, policy logic and objectives, the logic model

3) Impact Evaluation and Counterfactual Approach: Outcome variable and treatment variable, definition of impact and counterfactual, some simple estimators, threats to the validity of impact evaluation

4) The Potential Outcomes Model: Measures of impact (average treatment effect), selection bias and omitted variable bias, strategies to limit the scope of threats to the validity of impact evaluation

5) Experimental Design: Logic of experimental design and estimators, implementation of experimental design (sample size and application methods), limitations in the applicability of experimental design

6) Non-Experimental Designs: Strategies for non-experimental designs, the Difference in Differences (DID) method, estimator and assumptions for impact identification, extensions

7) The Use of Regression in Impact Evaluation: Implementing simple estimators, eliminating baseline differences, improving the precision of estimates in experimental designs, obtaining the DID estimator

8) Other Non-Experimental Designs - Regression Discontinuity Design (RDD): Conditions for application of RDD, RDD estimator and identification assumptions, generalization of the functional form and non-parametric estimation, verifying identification assumptions and estimator robustness

9) Other Non-Experimental Designs - Propensity Score Matching (PSM): Statistical matching, definition and estimation of the Propensity Score (PS), matching algorithms, estimators and identification assumptions

10) Other Non-Experimental Designs - Instrumental Variables (IV): Partial Compliance and Instrumental Variables (IV), definition of an instrumental variable and its properties, IV estimator, Encouragement Design (characteristics and IV estimator).

11) Further Insights (Optional):

• The use of longitudinal data: panel data and extension of the DID method
• Implementation evaluation (overview): what is observed and how evaluative questions are answered

Readings/Bibliography

Reference texts:

A. Martini, M. Sisti (2009), Valutare il successo delle politiche pubbliche, Il Mulino. (ch. 14 excluded; ch.  5, 15 and 16 just reading)

G. Cerulli (2015), Econometric Evaluation of Socio-Economic Programs. Theory and applications, Springer. Ch.1 (1.1, 1.2, 1.3,1.4, 1.5, 1.8), Ch.2 (2.3, 2.7), Ch. 3 (3.4, 3.6), Ch. 4 (4.2, 4.3, 4.4).

Recommended reading:

P. H. Rossi, M. W. Lipsey, H. E. Freeman (2003), Evaluation: A Systematic Approach, Sage, chap. 1, 2, 3, 5, 7

J. H. Stock, M. W. Watson, Introduzione all'econometria, Edizione italiana a cura di F. Peracchi (2005), Pearson, Prentice-Hall

Further reading (optional):

R. Blundell, M. Costa Dias (2002), Alternative approaches to evaluation in empirical microeconomics, Cemmap working paper CWP10/02, The Institute for fiscal studies, Department of economics, Ucl, http://www.cemmap.ac.uk/wps/cwp1002.pdf

Heckman J.J., Smith J.A. (1999), The pre-program earnings dip and the determinants of participation in a social programme. Implications for simple program evaluation strategies, The Economic Journal, 109, 313-348.

James J. Heckman; V. Joseph Hotz (1989), Choosing Among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs: The Case of Manpower Training, Journal of the American Statistical Association, Vol. 84, No. 408, pp. 862-874.

Angrist J.D. and A.B. Krueger (1999), Empirical strategies in labor economics, in O. Ashenfelter and D. Card (eds.), Handbook of Labor Economics, Vol. 3A, Amsterdam, North-Holland, pp.1277-1366.

Becker S.O., Ichino A. (2002), Estimation of average treatment effects based on propensity scores, The Stata Journal, 2(4), 358-377.

Dehejia R.H., Wahba S. (1999), Causal effects in nonexperimental studies: reevaluating the evaluation of training programs, Journal of the America Statistical Association, 94, 1053-1062

Dehejia R.H., Wahba S. (2002), Propensity score matching methods for nonexperimental casual studies, The Review of Economics and Statistics, 84(1), 151-161.

Card and Krueger (1994), Minumum wages and employment: a case study of the fast food industry in New Jersey and Pennsylvania, American Economic Review, 84, 4

D. Bondonio (2000), Statistical methods to evaluate geographically-targeted economic development programs, Statistica applicata, vol. 12, n. 2, pp. 177-204)

T. Lemieux, K. Milligan (2008), Incentive effects of social assistance: A regression discontinuity approach, Journal of Econometrics, 142, pp. 807–828

Teaching methods

The course is structured into lectures that cover theoretical aspects illustrated through examples and case studies, as well as practical activities carried out in the classroom.

The practical activities involve the use of a statistical software (Stata) to apply the methods presented during the theoretical lessons to real data.

The software can be downloaded and installed free of charge by each student on their own laptop. To actively participate in the practical activities, it is therefore necessary to bring your own laptop to class. Students who do not have a laptop available can still work in pairs with a classmate.

Assessment methods

The assessment will consist of a written text lasting 75 minutes, to be taken at the end of the course. The aim is to verify the achievement of the learning objectives:

• the knowledge of the methodological framework, concepts, and methods for impact evaluation;
• the ability to design and interpret an evaluation framework (selection of outcome variables, treatment variable, and appropriate method based on the data);
• the ability to interpret the estimation results;
• the ability to use formal language appropriately.

 

The written text includes two parts:

• a set of 10 multiple-choice questions covering theoretical, definitional, and methodological aspects (worth 15 points in total);
• 4 to 5 open-ended questions covering theoretical, definitional, and methodological aspects, as well as simple impact estimation exercises and interpretation of results from the software output used during the lessons. The ability to use appropriate statistical and formal language will also be assessed (worth 17 points in total).

Examples of the types of questions can be found in the "useful content" section of the teacher website. Toward the end of the course, the teacher will make available on the e-learning platform VIRTUALE a online test, to be completed independently.

During the exam, students may use a calculator, but no materials (personal notes, course materials provided by the instructor, or textbooks) may be consulted.

The exam will be the same for both attending and non-attending students.

Teaching tools

1) On the e-learning platform VIRTUALE  [https://virtuale.unibo.it/ ], the following materials will be made available:

• Lecture slides
• Data sets and software codes for applications
• Additional materials (e.g., articles related to case studies, handouts)

The materials will be provided and updated throughout the course in advance.

 

2) For the in-class practical exercises, it is necessary to install the STATA software, SE version, on your personal computer. UNIBO has activated a Campus license that allows all students to download and install the software for free. You can download the software using the link: [https://www.unibo.it/secure/software-stata/ ] .

Be sure to also download any documentation with instructions for activating the license.

Office hours

See the website of Pinuccia Pasqualina Calia