External validity and policy adaptation: From impact evaluation to policy design
With the growing number of rigorous impact evaluations worldwide, the question of how best to apply this evidence to policymaking processes has arguably become the main challenge for evidence-based policymaking. How can policymakers predict whether a policy will have the same impact in their context as it did elsewhere, and how should this in influence the design and implementation of policy?
This paper introduces a simple and flexible framework to address these questions of external validity and policy adaptation. The author shows that all failures of external validity arise from an interaction between a policy's theory of change and a dimension of the context in which it is being implemented. He develops a method of "mechanism mapping" that maps a policy's theory of change against salient contextual assumptions to identify external validity problems and suggest appropriate policy adaptations.
In deciding whether and how to adapt a policy in a new context, Martin Williams demonstrates that there is a fundamental informational trade-off between the strength and relevance of evidence on the policy from other contexts and the policymaker's knowledge of the local context. This trade-off can guide policymakers' judgments about whether policies should be copied exactly from elsewhere, adapted, or invented anew.
See also by the same author: Policy Memo: External validity and policy adaptation
About the author
Martin J. Williams is Associate Professor in Public Management, Blavatnik School of Government, University of Oxford