A user-friendly, step-by-step guide to conducting comparative product evaluations

April 26, 2018, Massachusetts Institute of Technology
MIT D-Lab research scientist and CITE research affiliate Eric Verploegen (right) installs sensors on an evaporative refrigerator with Ousmane Sanogo of the World Vegetable Center in Mali. Credit: Lauren McKown/MIT D-Lab

According to the World Bank, over 1.1 billion people have lifted themselves from extreme poverty since 1990. But even as the global outlook on extreme poverty improves, billions of people continue to struggle to access basic human needs, like water, food, shelter, health care and energy. In response to these challenges, innovators around the world have developed a preponderance of cost-effective, locally implemented solutions, from solar lanterns and water filters to improved cookstoves and refugee shelters.

With such a dizzying array of products on the market, development professionals often struggle to cut through the hype associated with novel technologies, and many are hesitant to pursue innovative approaches to stubborn development challenges, given the high stakes of working with economically vulnerable populations.

MIT researchers are now seeking to help development professionals overcome these challenges by using design thinking, together with a methodology for comparative technology evaluation that is five years in the making.

A Practitioner's Guide to Technology Evaluation in Global Development offers a user-friendly, step-by-step framework to help organizations identify development solutions that are most likely to succeed in a given context. Co-authored by the MIT Comprehensive Initiative on Technology Evaluation (CITE), a program supported by the U.S. Agency for International Development (USAID), and the Technology Exchange Lab, a Cambridge-based non-governmental organization, the Practitioner's Guide builds upon five years of research and over 12 comparative evaluations conducted by CITE across eight countries.

A methodology for all

CITE was founded in 2013 as a leading member of USAID's Higher Education Solutions Network, a coalition of seven universities seeking to leverage the talent of students, researchers, and faculty towards solving major global development challenges. Since then, CITE has evaluated "hardware solutions"—like , solar-powered water pumps, and water test kits—while also evaluating systems-level solutions, such as how distribution models affect the uptake of malaria-diagnostics, and how food-aid packaging impacts the quality and quantity of international food assistance through complex supply chains.

Throughout this work, CITE researchers from across the Institute—from MIT D-Lab, the Department of Urban Studies and Planning (DUSP), Center for Transporation and Logistics (CTL), and Sociotechnical Systems Research Center (SSRC) — developed and iterated upon a multi-disciplinary evaluation methodology known as '3S' framework, which evaluates technologies from three vantages including: suitability (how well products perform technically), scalability (how effective products are at reaching consumers at scale), and sustainability (how products are adopted and used over time).

While CITE effectively applied its methodology across multiple sectors, past evaluations included rigorous lab testing, and support from faculty, graduate students and additional partners. According to CITE Associate Director Joanne Mathias, "through the Practitioner's Guide we aim to empower practitioners and smaller-scale NGOs with the tools required to find solutions that work, regardless of the resources or facilities they have at their disposal."

From lab to field

Translating five years of academic research into a user-friendly toolkit is no small task. By teaming up with the Technology Exchange Lab (TEL) to co-author the guide, CITE sought out a partner capable of articulating their evaluation methodology to a non-academic audience, while also weaving concepts of human-centered design into the evaluation process. Founded by two MIT Sloan School of Management alumni, TEL works with community-based organizations around the world to implement innovative solutions to problems of poverty.

"Partnering with CITE to develop the Practitioner's Guide was a natural fit," says TEL Programs Director Brennan Lake.  "So much emphasis is made on driving innovation and research through universities, which is fantastic, but there is a bottleneck when it comes to putting research into practice, and making innovative approaches and development solutions as accessible as possible to communities on the ground."

Indeed, the Practitioner's Guide includes real-world examples of everyday challenges faced by development practitioners—such as how to make data-driven trade-offs between a product's quality, affordability, and time to implement—as well as case studies based off of past CITE evaluations. The Practitioner's Guide was also designed to be modular, so that organizations at various stages of project could make use of CITE's methodology.

"Most aid agencies and international NGOs already have strict procurement protocols in place," Lake notes. "The Practitioner's Guide provides program officers with discrete tools for evidence-based decision making, while also offering a more comprehensive framework for NGOs and community-based organizations looking to build programs from the ground up."

Explore further: Researchers release evaluation of solar pumps for irrigation and salt mining in India

Related Stories

Making smarter decisions about classroom technologies

September 30, 2016

In the 21st century, the proliferation of digital media and technology has fundamentally changed the way we learn. More than ever, children carry computers in their pockets and ever-expanding internet connectivity promises ...

Reducing spoilage in food aid shipments

January 6, 2017

The U.S. Agency for International Development (USAID) ships out food aid worth more than $1 billion every year—over a million tons of grains, soybeans, and other staples sent to dozens of countries around the world. Even ...

Recommended for you

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.