External validity and policy adaptation: From impact evaluation to policy design

Blavatnik School working paper
Abstract

With the growing number of rigorous impact evaluations worldwide, the question of how best to apply this evidence to policymaking processes has arguably become the main challenge for evidence-based policymaking. How can policymakers predict whether a policy will have the same impact in their context as it did elsewhere, and how should this in influence the design and implementation of policy?

This paper introduces a simple and flexible framework to address these questions of external validity and policy adaptation. The author shows that all failures of external validity arise from an interaction between a policy's theory of change and a dimension of the context in which it is being implemented. He develops a method of "mechanism mapping" that maps a policy's theory of change against salient contextual assumptions to identify external validity problems and suggest appropriate policy adaptations.

In deciding whether and how to adapt a policy in a new context, Martin Williams demonstrates that there is a fundamental informational trade-off between the strength and relevance of evidence on the policy from other contexts and the policymaker's knowledge of the local context. This trade-off can guide policymakers' judgments about whether policies should be copied exactly from elsewhere, adapted, or invented anew.

About the author

Martin Williams is Associate Professor in Public Management at the Blavatnik School of Government.

BSG-WP-2018/019