Measuring COVID learning loss

Parents, educators and policymakers have faced rising concerns about what students have lost academically during a year of school closures and online learning. Until recently, however, they've lacked concrete evidence about what exactly those losses look like.

One recent study, conducted by University of Virginia faculty member Jim Soland and colleagues, has started to fill in the gap.

As an assistant professor in the School of Education and Human Development with a background in policy work, Soland's interests lie in where measurement, practice and policy intersect. He is particularly driven to understand how seemingly minor decisions about educational measures—like how educators score and use tests and surveys—ultimately affect students.

"A lot of the reason I got into testing policy was because the seemingly arcane decisions made by statisticians in back rooms actually have really profound implications for equity and fairness," he said. "It's pretty technical work, but it's always driven by these questions of equity in the education system."

In addition to his faculty role, Soland holds a position as a research fellow at NWEA, an education nonprofit that administers a widely used test called MAP Growth. MAP Growth was designed to measure how students progress in math and reading skills. Three times a year, about one out of every 10 schools in the country give MAP Growth tests to third- through eighth-graders, providing a rich dataset for researchers.

Last year, Soland and colleagues at NWEA dug into past data on learning loss—including studies on school absences and closures caused by natural disasters—to estimate the potential impacts of COVID on student learning. They predicted learning losses in both reading and math, with steeper losses in math.

Now, they finally have the data to see how real results compared to their predictions. Using data from fall testing of more than 4 million students in public schools across the country, Soland and his colleagues looked at how much students' learning has grown since schools closed in March, as well as how students performed relative to a typical school year.

The results were mixed. On the bright side, the data showed that students did make learning gains on average during the pandemic, and reading scores in 2020 were mostly comparable to students in the same grade in 2019. However, scores on math achievement were more concerning. On average, students across the board scored about five to 10 percentile points lower compared to the prior year.

"Math definitely seems to be where people should … be most worried," Soland said.

The data are preliminary, and much is still unknown about the extent of learning loss and how it may vary within different communities. Critically, researchers are concerned about signs that many students have dropped out of the education system altogether—meaning some of the most at-risk students likely aren't showing up in the data at all. The less severe drops than anticipated in reading could, for example, be related as much to who is being tested as to how those students are performing.

Still, Soland said the early analysis is a valuable first step toward recovery.

"I think, deep down, everybody knows and suspects that [the pandemic] hasn't been good for student learning," he said. "But having concrete data that show just how big the drop-off has been in math is important not just for parents and teachers to know, but also for policymakers as they're debating how to support schools."