External validity and policy adaptation

External validity and policy adaptation: From impact evaluation to policy design

BSG-WP-2017/019

With the growing number of rigorous impact evaluations worldwide, the question of how best to apply this evidence to policymaking processes has arguably become the main challenge for evidence-based policymaking. How can policymakers predict whether a policy will have the same impact in their context as it did elsewhere, and how should this in influence the design and implementation of policy?

This paper introduces a simple and flexible framework to address these questions of external validity and policy adaptation. The author shows that all failures of external validity arise from an interaction between a policy's theory of change and a dimension of the context in which it is being implemented. He develops a method of "mechanism mapping" that maps a policy's theory of change against salient contextual assumptions to identify external validity problems and suggest appropriate policy adaptations.

In deciding whether and how to adapt a policy in a new context, Martin Williams demonstrates that there is a fundamental informational trade-off between the strength and relevance of evidence on the policy from other contexts and the policymaker's knowledge of the local context. This trade-off can guide policymakers' judgments about whether policies should be copied exactly from elsewhere, adapted, or invented anew.

See also by the same author: Policy Memo: External validity and policy adaptation

About the author

Martin J. Williams is Associate Professor in Public Management, Blavatnik School of Government, University of Oxford

Documents: