Using 'counterfactuals' to verify predictions of drug safety

How can we be sure machine learning is accurate? : NewsCenter
Overview of MMACE. The input is a molecule to be predicted. Chemical space is expanded and clustered. Counterfactuals are selected from clusters to find succinct explanation of base molecule prediction. Credit: Chemical Science (2022). DOI: 10.1039/D1SC05259D

Scientists rely increasingly on models trained with machine learning to provide solutions to complex problems. But how do we know the solutions are trustworthy when the complex algorithms the models use are not easily interrogated or able to explain their decisions to humans?

That trust is especially crucial in , for example, where is used to sort through millions of potentially toxic compounds to determine which might be safe candidates for .

"There have been some high-profile accidents in where a model could predict things quite well, but the predictions weren't based on anything meaningful," says Andrew White associate professor of chemical engineering at the University of Rochester, in an interview with Chemistry World.

White and his lab have developed a new "counterfactual" method, described in Chemical Science, that can be used with any -based machine learning model to better understand how the model arrived at a conclusion.

Counterfactuals can tell researchers "the smallest change to the features that would alter the prediction," says lead author Geemi Wellawatte, a Ph.D. student in White's lab. "In other words, a counterfactual is an example as close to the original, but with a different outcome."

Counterfactuals can help researchers quickly pinpoint why a model made a prediction, and whether it is valid.

The paper identifies three examples of how the new method, called MMACE (Molecular Model Agonistic Counterfactual Explanations), can be used to explain why:

  • a molecule is predicted to permeate the blood-brain barrier
  • a small molecule is predicted to be soluble
  • a molecule is predicted to inhibit HIVs

The lab had to overcome some major challenges in developing MMACE. They needed a method that could be adapted for the wide array of machine-learning methods that are used in chemistry. In addition, searching for the most-similar molecule for any given scenario was also challenging because of the sheer number of possible candidate molecules.

How can we be sure machine learning is accurate? : NewsCenter
From left: PhD student Geemi Wellawatte, Andrew White, an associate professor of chemical engineering, and Aditi Seshadri ’22 in Wegmans Hall. White’s lab has developed a way to verify the predictions of machine learning models used in drug discovery by using counterfactuals. Credit: University of Rochester/J. Adam Fenster

Coauthor Aditi Seshadri in White's lab helped solve that problem by suggesting the group adapt the STONED (Superfast traversal, optimization, novelty, exploration, and discovery) algorithm developed at the University of Toronto. STONED efficiently generates similar molecules, the fuel for counterfactual generation. Seshadri is an undergraduate researcher in White's lab and was able to help on the project via a Rochester summer research program called "Discover."

White says his team is continuing to improve MMACE, by trying other databases in their search for most similar molecules, for example, and refining the definition of molecular similarity.

More information: Geemi P. Wellawatte et al, Model agnostic generation of counterfactual explanations for molecules, Chemical Science (2022). DOI: 10.1039/D1SC05259D

Journal information: Chemical Science

Citation: Using 'counterfactuals' to verify predictions of drug safety (2022, May 2) retrieved 20 May 2024 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Explore further

AI technique narrowed to only propose candidate molecules that can be produced in a lab


Feedback to editors