Methods for Policy Evaluation
Ph.D. in Social and Political Science (Bocconi University, Spring 2025)
Course Description
I offer a modern introduction to empirical strategies in applied micro research in fields such as public policy, development economics, labor economics, education, marketing, and corporate finance. My course focuses on causal reasoning and design-driven identification in the social sciences, offering practical insights and techniques for real-world applications. I emphasize intuition over extensive formal derivation of theoretical aspects of probability and statistics, focusing instead on the essential foundations for assessing causal relationships. I explore a range of methods to assess causal questions effectively, delving into differences between design-based and sampling-based uncertainty. The exploration begins with approximating the conditional expectation function using a linear predictor, employing ordinary least squares. I then discuss the implications of omitted variables and examine the value of quasi-experimental designs versus quasi-experimental methods to replicate the experimental ideal, or thought experiment. All concepts are brought to life with practical applications on real data.
Requirements
Before the semester starts. You are expected to be comfortable with college-level algebra and calculus. A lack of good working knowledge of statistics and probability may make this course more difficult than necessary. You are therefore invited to acquire a good working knowledge of graduate-level mathematical tools, fundamentals of probability, fundamentals of mathematical statistics and matrix algebra before the beginning of classes. After the semester starts. I warmly encourage you to spend time reading course materials and lecture slides before coming to class. You will learn the concepts much more clearly if you spend time struggling with the material before class starts. You are also warmly encouraged to familiarize yourself with the use of statistical software throughout the course. However, my classes are not aimed at developing coding skills.
Course Readings
Lecture slides and reading materials will be made available below before lectures. I will mix traditional approaches to micro-econometrics with more modern tools used in several empirical literatures. For this reason, it is impossible to identify one textbook. Lecture slides are self-contained and may be enough to understand the topics discussed. However, you are strongly invited not to limit your study to the slides, and it is in your interest to familiarize yourself with the additional readings suggested below and during lectures.
Exams
Your final grade will be computed as a weighted average of two components: a final during exam week (70% of the final grade) and a group presentation at the end of the semester (30% of the final grade). The exams may include questions with an analytical component, or problems based on the output of statistical software to test your ability to understand and interpret numbers. There isn't a fixed format for my exams, which I write depending on how our discussion unfolds and your own taste about what I teach. I will detail specific instructions on the group presentation during the semester, and the modality of presentations will be decided depending on class size and how our discussion unfolds.
Self-Assessments
Each file contains a set of questions that I warmly encourage you to engage with by attempting your own solutions. These questions combine material presented in class with ideas and concepts from the suggested readings assigned throughout the course. About a week after each file is posted, I will share a sketch of my proposed solutions. Occasionally, I may also suggest additional readings. These materials serve three main purposes. Self-Assessment. They are designed to help you evaluate how well you have internalized the topics discussed in lectures. This may also help you assess your readiness for the final exam. Understanding the Literature. They offer insight into how the current literature approaches causal inference in various contexts within the social sciences. Food for thought. They are meant to serve as food for thought when considering potential dissertation chapters, regardless of your current research interests. I strongly encourage you to think outside the box and draw on ideas from multiple subfields.
Grading
I grade exams by assessing each question individually rather than grading each exam as a whole. To ensure transparency in grading, I use a detailed checklist which I outline in my solution guide. You can use this guide to self-grade your exam by comparing your answers with the checklist. My grading process involves two readings of the answers for each question. Initially, I read all responses to assess the overall quality and understand common issues. This step helps me determine if the problems stem from the way questions were phrased in the exam or from the clarity of my lectures. In the second reading, I grade each answer using the checklist. I begin with what I considered the best response in the first review, and proceed through all the exams. This method ensures that I have differentiated the quality of responses. After grading, I review each student's exam again to assess their overall understanding and control of the subject matter for each question. Finally, I assign the following letter grades based on percentage scores:
A+, A, A- denote excellent mastery of the subject and outstanding scholarship.
B+, B, B- denote good mastery of the subject and good scholarship.
C+, C, C- denote acceptable mastery of the subject.
D+, D, D- denote borderline understanding of the subject and does not represent satisfactory progress.
F denotes failure to understand the subject and unsatisfactory performance.
I offer a modern introduction to empirical strategies in applied micro research in fields such as public policy, development economics, labor economics, education, marketing, and corporate finance. My course focuses on causal reasoning and design-driven identification in the social sciences, offering practical insights and techniques for real-world applications. I emphasize intuition over extensive formal derivation of theoretical aspects of probability and statistics, focusing instead on the essential foundations for assessing causal relationships. I explore a range of methods to assess causal questions effectively, delving into differences between design-based and sampling-based uncertainty. The exploration begins with approximating the conditional expectation function using a linear predictor, employing ordinary least squares. I then discuss the implications of omitted variables and examine the value of quasi-experimental designs versus quasi-experimental methods to replicate the experimental ideal, or thought experiment. All concepts are brought to life with practical applications on real data.
Requirements
Before the semester starts. You are expected to be comfortable with college-level algebra and calculus. A lack of good working knowledge of statistics and probability may make this course more difficult than necessary. You are therefore invited to acquire a good working knowledge of graduate-level mathematical tools, fundamentals of probability, fundamentals of mathematical statistics and matrix algebra before the beginning of classes. After the semester starts. I warmly encourage you to spend time reading course materials and lecture slides before coming to class. You will learn the concepts much more clearly if you spend time struggling with the material before class starts. You are also warmly encouraged to familiarize yourself with the use of statistical software throughout the course. However, my classes are not aimed at developing coding skills.
Course Readings
Lecture slides and reading materials will be made available below before lectures. I will mix traditional approaches to micro-econometrics with more modern tools used in several empirical literatures. For this reason, it is impossible to identify one textbook. Lecture slides are self-contained and may be enough to understand the topics discussed. However, you are strongly invited not to limit your study to the slides, and it is in your interest to familiarize yourself with the additional readings suggested below and during lectures.
Exams
Your final grade will be computed as a weighted average of two components: a final during exam week (70% of the final grade) and a group presentation at the end of the semester (30% of the final grade). The exams may include questions with an analytical component, or problems based on the output of statistical software to test your ability to understand and interpret numbers. There isn't a fixed format for my exams, which I write depending on how our discussion unfolds and your own taste about what I teach. I will detail specific instructions on the group presentation during the semester, and the modality of presentations will be decided depending on class size and how our discussion unfolds.
Self-Assessments
Each file contains a set of questions that I warmly encourage you to engage with by attempting your own solutions. These questions combine material presented in class with ideas and concepts from the suggested readings assigned throughout the course. About a week after each file is posted, I will share a sketch of my proposed solutions. Occasionally, I may also suggest additional readings. These materials serve three main purposes. Self-Assessment. They are designed to help you evaluate how well you have internalized the topics discussed in lectures. This may also help you assess your readiness for the final exam. Understanding the Literature. They offer insight into how the current literature approaches causal inference in various contexts within the social sciences. Food for thought. They are meant to serve as food for thought when considering potential dissertation chapters, regardless of your current research interests. I strongly encourage you to think outside the box and draw on ideas from multiple subfields.
- Self-assessment one and solutions.
- Self-assessment two and solutions.
- Self-assessment two and solutions.
Grading
I grade exams by assessing each question individually rather than grading each exam as a whole. To ensure transparency in grading, I use a detailed checklist which I outline in my solution guide. You can use this guide to self-grade your exam by comparing your answers with the checklist. My grading process involves two readings of the answers for each question. Initially, I read all responses to assess the overall quality and understand common issues. This step helps me determine if the problems stem from the way questions were phrased in the exam or from the clarity of my lectures. In the second reading, I grade each answer using the checklist. I begin with what I considered the best response in the first review, and proceed through all the exams. This method ensures that I have differentiated the quality of responses. After grading, I review each student's exam again to assess their overall understanding and control of the subject matter for each question. Finally, I assign the following letter grades based on percentage scores:
A+, A, A- denote excellent mastery of the subject and outstanding scholarship.
B+, B, B- denote good mastery of the subject and good scholarship.
C+, C, C- denote acceptable mastery of the subject.
D+, D, D- denote borderline understanding of the subject and does not represent satisfactory progress.
F denotes failure to understand the subject and unsatisfactory performance.
Course Contents (slides will be uploaded weekly)
LECTURE 1 (april 1, 2025). ECONOMIC REASONING, STATISTICAL LEARNING AND MODERN MICROECONOMETRICS
- The Identification Zoo (Structural, Reduced Form, and Causal Parameters)
- Thought Experiments vs Real (Natural) Experiments
- Internal Validity, External Validity, and Statistical Validity
- Potential Outcomes
- Causal Estimands and Policy Relevant Probability
- Potential Outcomes and Selection
- Design-based Uncertainty vs Sampling-based Uncertainty
Lecture slides (updated April 1, 2025).
Most of the concepts discussed in this part can be found in the following references: :
- Angrist Joshua D., and Pischke Jorn-Steffen (2009). Mostly Harmless Econometrics. An Empiricist’s Companion, Princeton University Press. [Chapters 1 and 2]
- Efron, Bradley, and Trevor Hastie (2016). Computer Age Statistical Inference: Algorithms, Evidence and Data Science, Cambridge University Press. [Chapter 1]
- Manski, Charles F. (2000). Analog Estimation Methods in Econometrics, Chapman & Hall. [Chapter 1]
- Rohatgi, Vijay K., and A.K. Md. Ehsanes Saleh (2015). An Introduction to Probability and Statistics (3rd Edition), Wiley. [Chapter 1]
- Wooldridge, Jeffrey M. (2010). Econometric Analysis of Cross Section and Panel Data (2nd Edition), MIT Press. [Chapters 1 and 2]
- Abadie, Alberto, Susan Athey, Guido W. Imbens, and Jeffrey M. Wooldridge (2020). Sampling-Based versus Design-Based Uncertainty in Regression Analysis. Econometrica, Volume 88(1), pp. 265-296.
- Chetty, Raj, Nathaniel Hendren, Maggie R. Jones, and Sonya Porter (2020). Race and Economic Opportunity in the United States: An Intergenerational Perspective. Quarterly Journal of Economics, 135(2), pp. 711-783.
- Heckman, James J., Jeffrey Smith, and Nancy Clements (1997). Making the Most Out of Programme Evaluations and Social Experiments: Accounting for Heterogeneity in Programme Impacts. The Review of Economic Studies, 64(4), pp. 487-535.
- Lewbel, Arthur (2019). The Identification Zoo: Meanings of Identification in Econometrics. Journal of Economic Literature, 57(4), pp. 835-903.
- Manski, Charles F. (1993). Identification Problems in the Social Sciences. Sociological Methodology, 23, pp. 1-56.
- Meyer, Bruce D., and James X. Sullivan (2012). Identifying the Disadvantaged: Official Poverty, Consumption Poverty, and the New Supplemental Poverty Measure. Journal of Economic Perspectives, 26(3), pp. 111-36.
- Tamer, Elie (2019). The ET Interview: Professor Charles Manski. Econometric Theory, 35(2), pp. 233-294.
LECTURE 2 (april 8, 2025). EXOGENOUS (OR RANDOMIZED) "TREATMENT"
- Population Regression for Dummies
- Omitted Variable Bias and the “Partialling Out” Theorem
- Long vs Short Population Regression
- Taxonomy of Research Designs (DAGs)
- Rethinking the Thought Experiment
Lecture slides (updated April 8, 2025).
Most of the terminology and concepts discussed in this part can be found in the following readings:
- Efron, Bradley (2010). Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction. Cambridge University Press. [Chapter 6].
- Efron, Bradley, and Trevor Hastie (2016). Computer Age Statistical Inference: Algorithms, Evidence and Data Science, Cambridge University Press. [Chapter 4]
- Hernán, Miguel A., and Robins, James M. (2020). Causal Inference: What If. Boca Raton: Chapman & Hall/CRC. [Chapters 2, 3 and 10]
- Imbens, Guido W., and Donald B. Rubin (2015). Causal Inference for Statistics, Social, and Biomedical Sciences, Cambridge University Press. [Chapters 5, 6, 7, 9 and 10]
- Rosembaum, Paul R. (2017). Observation and Experiment, Harvard University Press. [Chapters 1 and 2]
- Athey, Susan, and Imbens, G. W. (2017). The Econometrics of Randomized Experiments. In A. V. Banerjee and E. Duflo, eds, Handbook of Field Experiments, Vol. 1 of Handbook of Economic Field Experiments, North-Holland, pp. 73-140.
- Abadie, Alberto (2020). Statistical Nonsignificance in Empirical Economics. American Economic Review: Insights, 2(2), pp. 193-208.
- Abadie, Alberto, Susan Athey, Guido W. Imbens, and Jeffrey M. Wooldridge (2020). Sampling-Based versus Design-Based Uncertainty in Regression Analysis. Econometrica, 88: 265-296.
- Deaton, Angus (2020). Randomization in the Tropics Revisited: a Theme and Eleven Variations. NBER Working Paper No. 27600.
- Goldsmith-Pinkham, Paul, Hull, Peter, and Kolesár, Michal (2022). Contamination Bias in Linear Regressions. American Economic Review,114 (12): 4015–51.
- Imbens, Guido W. 2020. Potential Outcome and Directed Acyclic Graph Approaches to Causality: Relevance for Empirical Practice in Economics. Journal of Economic Literature, 58 (4): 1129-79.
- Lewbel, Arthur, Yingying Dong, and Thomas Tao Yang (2012). Comparing features of convenient estimators for binary choice models with endogenous regressors. The Canadian Journal of Economics / Revue Canadienne d'Economique,
45(3), pp. 809-829.
- Cunningham, S. (2021). Directed Acyclic Graphs. Causal Inference: The Mixtape.
LECTURE 3 (april 15, 2025). ENDOGENOUS "TREATMENT" AND INSTRUMENTAL VARIATION
- The Old School View
- The 'As Good As Random' and 'Exclusion Restriction' Conditions
- Two-Stage Least Squares Mechanics
- Heterogenous Policy Effects
- Putting Monotonicity in (Policy) Context
Lecture slides (updated April 12, 2025).
Most of the terminology and concepts discussed in this part can be found in the following readings:
- Angrist, Joshua D., and Alan B. Krueger (1991). Does Compulsory School Attendance Affect Schooling and Earnings? The Quarterly Journal of Economics, 106(4), pp. 979-1014.
- Angrist, Joshua D., Erich Battistin, and Daniela Vuri (2017). In a Small Moment: Class Size and Moral Hazard in the Italian Mezzogiorno. American Economic Journal: Applied Economics, 9(4), pp. 216-49.
- Imbens, Guido W., and Donald B. Rubin (2015). Causal Inference for Statistics, Social, and Biomedical Sciences, Cambridge University Press. [Chapters 23 and 24]
- Angrist, Joshua D., Victor Lavy, Jetson Leder-Luis, and Adi Shany (2019). Maimonides' Rule Redux. American Economic Review: Insights, 1 (3): 309-24.
- Mogstad, Magne, and Alexander Torgovitsky (2018). Identification and Extrapolation of Causal Effects with Instrumental Variables. Annual Review of Economics, 10(1), pp. 577-613.
- Mogstad, Magne, Alexander Torgovitsky, and Christopher R. Walters (2021). The Causal Interpretation of Two-Stage Least Squares with Multiple Instrumental Variables. American Economic Review, 111(11), pp. 3663-98.
LECTURE 4 (april 29, 2025). "TREATMENT" EFFECTS USING LONGITUDINAL VARIATION
- Static versus Dynamic Treatment Effects
- Data Structures and Research Designs to Retrieve Counterfactuals
- How Difference-in Differences Works, and Why it Works
- Staggered (and More General) Designs
- Back to the Future: Making Staggered Designs Make Sense
- Beyond Staggered Designs
- Synthetic Controls Methods
Lecture slides (updated April 14, 2025).
Most of the terminology and concepts discussed in this part can be found in the following readings:
- Abadie, Alberto (2021). Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects, Journal of Economic Literature, 59(2), pp. 391-425.
- de Chaisemartin, Clément, and Xavier d’Haultfoeuille (2020). Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects, American Economic Review, 110(9), pp. 2964-96.
- de Chaisemartin, Clément and Xavier d'Haultfoeuille (2025), Credible Answers to Hard Questions: Differences-in-Differences for Natural Experiments. Forthcoming, Princeton University Press.
- Abadie, Alberto, and Jérémy L'Hour (2021). A penalized synthetic control estimator for disaggregated data, Unpublished manuscript, MIT.
- Arkhangelsky, Dmitry, Susan Athey, David A. Hirshberg, Guido W. Imbens, and Stefan Wager (2020). Synthetic Difference in Differences, American Economic Review, forthcoming.
- Ben-Michael, Eli, Avi Feller, and Jesse Rothstein (2021). Synthetic Controls with Staggered Adoption. NBER Working Paper 28886.
- Borusyak, Kirill, Xavier Jaravel, and Jann Spiess (2021). Revisiting Event Study Designs: Robust and Efficient Estimation, Unpublished manuscript, University College London.
- Callaway, Brantly, and Pedro H.C. Sant’Anna (2020). Difference-in-Differences with multiple time periods, Journal of Econometrics, forthcoming.
- de Chaisemartin, Clément, and Xavier d’Haultfoeuille (2018). Fuzzy differences-in-differences, The Review of Economic Studies, 85(2), pp. 999-1028.
- Doudchenko, Nikolay, and Guido W. Imbens (2016). Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis, NBER Working Paper 22791.
- Freyaldenhoven, Simon, Christian Hansen and Jesse M. Shapiro (2019). Pre-event Trends in the Panel Event-Study Design, American Economic Review, 109(9), pp. 3307-38.
- Goodman-Bacon, Andrew (2021), Difference-in-differences with variation in treatment timing, Journal of Econometrics,
forthcoming. - Sun, Liyang, and Sarah Abraham (2020). Estimating Dynamic Treatment Effects in Event Studies with Heterogeneous Treatment Effects, Journal of Econometrics, forthcoming.
- Chamberlain Seminar (2020). Symposium on Synthetic Control Methods.
- Imbens, Guido W. (2021). RES 2021: Sargan Lecture - Causal Panel Data Models.
- McKenzie, David (2020). Revisiting the Difference-in-Differences Parallel Trends Assumption: Part I. Pre-Trend Testing. World Bank Blogs.
- McKenzie, David (2020). Revisiting the Difference-in-Differences Parallel Trends Assumption: Part II. What happens if the parallel trends assumption is (might be) violated? World Bank Blogs.
LECTURE 5 (may 6, 2025). "TREATMENT" EFFECTS BY CONDITIONING
LECTURE 6 (may 13, 2025). "TREATMENT" EFFECTS WITH JUMPS AND KINKS
Some notes on RD based on the class size example.
