Using evidence for better policymaking
Picture this.
It's 1932, the height of the Great Depression in the United States. The unemployment rate is over 20% and growing. American citizens are looking to the government for answers. Franklin Delano Roosevelt is running for the US presidency against incumbent President Herbert Hoover.
On the campaign trail, FDR remarks: "The country needs, and, unless I mistake its temper, the country demands bold persistent experimentation. It is common sense to take a method and try it: If it fails, admit it frankly and try another. But above all, try something."
While many would have expected candidates to campaign on very specific policies, FDR decided instead to campaign on an alternative approach — experimentation and learning. His message was: we don't know what works, so let's try things.
We can figure out what works and what doesn't along the way and adjust our policies accordingly. We can measure both successes and failure and this gathered evidence can and should inform policy design.
In your role as a policymaker, you can implement and pilot policies in ways that allow for learning and adjust policy accordingly. Below is an example of how policymakers can inject a method of experimentation and evaluation into work that can drive positive results.
Improving educational attainment around the world
The past two decades have seen massive improvements in school enrolment around the world. This is partly due to the United Nations Millennium Development Goals (MDGs), which mobilised action in the period 2000-2015 before being superseded by the Sustainable Development Goals (SDGs) in 2015. One of the MDGs was to achieve universal primary education for all girls and boys. Some of the achievements under this goal include:
- An 8 percentage-points increase in the primary school net enrolment between the years 2000 to 2015.
- Nearly 50% decrease in the number of out-of-school children, globally, from 100 million to around 57 million in the same period.
- Remarkable improvement in primary education in Sub-Saharan Africa since the establishment of the MDGs.
But as critics have pointed out since, the MDGs targeted increasing enrolment only, not quality. Hence these global improvements in school enrolment have not been matched by a similar progress in learning.
For example, in India, where school enrolment is high, only about 44% of Grade 8 students can do a division equation. Similarly, only about 75% of Grade 8 students can read a Level-2 book. Level 2 is defined as “reading with help” where engaging stories, longer sentences, and language play are used for developing readers who still need some help. This has led to calls for improving learning in schools.
We see a similar trend in other countries around the world, but few have picked up the mantle of experimentation like Kenya.
Trying to identify solutions
In an attempt to improve learning in schools in western Kenya, policymakers in the early 2000’s collaborated with researchers to implement policies that they expected to work, and when they didn’t, they tried new things:
Solution 1: Flipcharts
Poor education may be due, in part, to the lack of proper school materials.
The intervention: Provide schools with flip charts covering mathematics, geography, health, and two charts for science (agriculture and general science), as well as a teacher’s guide for science. They could be used in more than one classroom each day.
Findings: Despite a large sample size and two years of follow-up data, the estimated impact of flipcharts on student test scores was found to be close to zero.
Solution 2: Textbooks
Again, poor education may be due, in part to the lack of proper school materials. This time: textbooks.
The intervention: Provide schools with additional official government textbooks (English, Maths and Science). Sharing textbooks is common in Kenya, and two or three students typically share a workspace. Hence, a 0.6 textbook per pupil ratio was provided in English and science, and a 0.5 ratio was provided in math, giving nearly all students shared access to a textbook.
Solution 3: Extra teachers
Poor education may be due, in part, to large class sizes. when primary education was made free, it caused an influx of new students with little or no preparation, which affected the overall quality and achievements of all students as teachers have to spend more time trying to bring everyone to the same level.
The intervention: The ETP programme provided funding to hire a local contract teacher to address classroom overcrowding. School committees were responsible for hiring the contract teachers and were free to replace or keep the original contract teacher based on performance. The contract teachers were paid approximately one-quarter of the salary of regular civil service teachers but had similar educational qualifications.
Findings: Providing school committees with funds to hire an extra teacher on a short-term contract had a positive effect on learning. Test scores of students in ETP schools were on average higher than those of students in comparison schools.
Evidence is vital
The above example is not so much about ways to improve education, but rather about a mindset to use evidence and evaluate policy to understand what works where. As the example illustrates, policymakers often have a hard time knowing in advance what will work and what won't, which is why building evidence and evaluation into the policymaking process is crucial for success.
FDR’s policy of experimentation helped the US recover from the worst depression in living memory. At the University of Oxford’s Blavatnik School of Government, we believe evidence is imperative to address the complex challenges facing policymakers in today’s world, which is why we have launched a new online short course on Evidence in Public Policy to equip policymakers with the knowledge and tools to implement this mindset of experimentation.
This article was originally published on Apolitical on 21 October 2020.
Rafat Al-Akhali is Fellow of Practice for Strategic Projects at the Blavatnik School of Government, University of Oxford.