Scientists doubt that DeepMind's AI is as good for fractional-charge systems as it seems
In their paper published in Science in December 2021, a DeepMind team showed how neural networks can be used to describe electron interactions in chemical systems more accurately than existing methods. A team of researchers from Skoltech, Zelinsky Institute of Organic Chemistry, HSE University, Yandex, and Kyungpook National University show in their comment in Science that DeepMind AI's ability to generalize the behavior of such systems does not follow from the published results and requires revisiting.
Knowing where the electrons are within a molecule can go a long way to explaining its structure, its properties, and its reactivity. Chemists use density functional theory (DFT) methods, approximations to the Schrödinger's equation, to make accurate and computationally efficient models of molecules and materials. But there are well-known circumstances where DFT tools fail. One is predicting how atoms share electrons; in one famous example, DFT methods incorrectly predict that even when a chlorine and a sodium atom are infinitely far apart, the chlorine atom retains a fraction of one of the sodium atom's electrons.
Errors like that arise because DFT equations are only approximations of the physical reality. Researchers from DeepMind machine learning project say that their neural network eliminates that part-of-an-electron error and makes more accurate predictions than traditional DFT methods
"At its core, DFT is a method for solving the Schrödinger equation. Its accuracy is determined by its exchange-correlation part, which is, unfortunately, unknown. To date, over 400 distinct approximations for this part were proposed," says Petr Zhyliaev, senior research scientist at Skoltech.
"One way to build a good exchange-correlation part is to transfer information about it from more 'advanced' numerical methods than the density functional theory, which are, however, orders of magnitude less computationally efficient. In their work, DeepMind used a neural network as a universal interpolator to learn the exchange-correlation part of the functional. Their attempt was not the first by far, but is one of the most ambitious."
DeepMind constructed a neural network based density functional designated as DM21, trained on fractional-electron systems, such as a hydrogen atom with half an electron. To prove its superiority, the authors have tested DM21 on a set of stretched dimers (called BBB set), e.g., two hydrogen atoms at a large distance with a total of one electron.
Expectedly, the DM21 functional shows an excellent performance on the BBB test set, surpassing by far all the tested classic DFT functionals and DM21m, trained identically to DM21 but without the fractional-electron systems in the training set.
Although this may look like DM21 has understood the physics behind the fractional-electron systems, a closer look shows that all the dimers in the BBB set become very similar to the systems in the train set. Indeed, by virtue of electroweak interactions locality, atomic interactions are strong only at short distances, outside of which the two atoms behave essentially as if they were not interacting (see figure above).
"In some ways, neural networks are just like humans: they prefer to get the right answer for the wrong reason, then the other way around. Therefore, it is not so hard to train a neural network, as it is to prove that it has learned the physical laws instead of memorizing the right answers. Testing a neural network on systems it has seen during training is akin to examining a schoolboy with a task he has seen teacher solving just five minutes ago," explains Michael Medvedev, the leader of Group of Theoretical Chemistry at Zelinsky Institute of Organic Chemistry of Russian Academy of Sciences.
Thus, the BBB test set is not a proper one: it does not test DM21 understanding of the fractional -electron systems: DM21 can easily get away with memorizing. A thorough analysis of the other four evidences of DM21 handling of such systems did not lead to a decisive conclusion, either: only its good accuracy on the SIE4x4 set may be reliable, although even there a clear trend of errors growth with the distance suggests that DM21 is not completely void of the trouble with fractional-electron systems.
The usage of fractional-electrons systems in the training set is not the only novelty in the work by DeepMind. Their idea of introducing the physical constraints into a neural network via the training set, as well as the approach for imposing physical sense through training on the correct chemical potential, are likely to be widely used in construction of neural network DFT functionals in the future.