The Implementation Feasibility Trap

The bigger lesson from a New York Times article evaluating 30 education interventions

The New York Times surveyed dozens of academics, including myself, asking us to rank 30 education interventions on their effectiveness and costs. A clear consensus winner emerged, and it wasn’t even on the list.

The article will surely reignite debates among different academic and political camps about the most effective path forward for education reform. Advocates of tutoring, the science of reading, or universal pre-K will be thrilled to see their preferred interventions rise to the top. Champions of private school vouchers or eliminating tracking will inevitably decry the rating process as biased or unscientific. Many others will assail the article for leaving better ideas off the list. All of this hand-wringing will miss the broader point.

To their credit, the Times journalists illustrate how rankings change when we scale effectiveness relative to costs. This “value” ratio is essentially telling us the Return On Investment (ROI) of an intervention, although I wish they had not required an above-average effectiveness rating to be on the main list. Sometimes undertaking a collection of small, inexpensive interventions with high ROI is a key step for generating momentum for larger, bigger-ticket reforms.

Evaluating ROI gets us much closer to the real world decision process that school boards and district leaders confront when choosing how to allocate scarce resources. But this two-dimensional rating system misses the most critical dimension of all for assessing the policy relevance of an education reform — feasibility of implementation.

Dig a little bit deeper into the article and you will uncover this gem:

“If there was one thing our expert panel agreed on, it was that how well ideas are carried out matters.”

In some ways this insight is both obvious and circular. Of course an intervention doesn’t work if you don’t implement it well. But pausing to reflect on and predict, however imprecisely, whether the necessary conditions, resources, and will to implement an education intervention exist would almost be a revolutionary improvement in the education reform process.

Effectiveness might be viewed as the internal validity of the evidence for an intervention. How well did it work when evaluated using rigorous methods capable of identifying cause and effect. Implementation feasibility asks about the external validity of the evidence. Would it replicate or scale with fidelity in other settings?

Some promising interventions such as investing in modern HVAC systems, which I wish had made the cut for the survey, are VERY easy to implement with fidelity. They are technical solutions that require very little human behavioral change. Other promising interventions, like tutoring, are much harder to scale because they require hiring or training personnel, organizational change, and difficult resource tradeoffs.

The critical importance of assessing the feasibility of implementing and scaling an intervention is something we know very little about. With limited research, we are left with an almost vibes-based approach for assessing the one thing most scholars appear to agree on. I see this as a blind spot in academic research. We want to find the next big solution but eschew more scientific study of the messy work of implementation.

Avoiding the Trap

Ignoring implementation feasibility comes at a high price for education reform and budgets. Reforms that flounder because of implementation challenges erode buy-in for future efforts and are a drain on scarce resources.

In my 2020 article on interpreting effect sizes, I proposed a schema for assessing the policy importance of education interventions similar to ones evaluated in this New York Times article. It seeks to elevate this third dimension of implementation feasibility - which I labeled as scalability. The two-dimensional rendition of this three-dimensional decision making process leaves something to be desired, but the insight remains evergreen.

Asking whether an intervention can be pulled off is itself a valuable exercise in reflection even if the answer is elusive. The process can illuminate key steps to take that might increase the possibility of an intervention’s success such as bringing stakeholders into the design phase early on or building networks of practice.

The lesson here should not be to avoid any reforms that are hard to implement. One of the many mottoes I repeat to my kids at bedtime is, “We can do hard things in our family.”

Instead, it is that we should be clear-eyed about the importance of implementation feasibility. Interventions with modest ROIs but that are very feasible to implement should be seen as attractive.

Interventions with higher ROI but that are harder to get right require a tougher decision. Take the plunge and stay the course for several years with a team dedicated to troubleshooting—or turn to other reforms that are more feasible.

The worst outcome of all is pursuing high ROI reforms, while failing to consider and prepare for their implementation challenges. This leads to the all-too-familiar pattern of high hopes and dashed expectations where reformers are quick to cut bait with little to show for their efforts.