With the growing number of rigorous impact evaluations worldwide, the question of how best to apply this evidence to policymaking processes has arguably become the main challenge for evidence-based policymaking. How can policymakers predict whether a policy will have the same impact in their context as it did elsewhere, and how should this influence the design and implementation of policy?

In this policy memo, Dr Martin Williams shows that failures of external validity arise from an interaction between a policy’s theory of change and a dimension of the context in which it is being implemented, and develops a five-step method of mechanism mapping that maps a policy’s theory of change against salient contextual assumptions to identify external validity problems and suggest appropriate policy adaptations. How to create a mechanism map in five steps:

1. Map out the policy’s intended theory of change (i.e. mechanism).
2. Below this, map the contextual assumptions that must hold for the theory of change to work.
3. Map the actual characteristics of the context and compare them to the assumptions.
4. Adapt the policy to address any mismatches between assumptions and characteristics.
5. Repeat steps 1-4 for the adapted policy, iterating in more detail until satisfied that all major policy design decisions fit the local context.