Studies of electron density lead to discovery of disturbing trends in modern computational chemistry

When chemists include some theory in their papers, it usually means that they will perform some quantum-chemical calculations of the participating molecules. This generally means solving a series of extremely complex equations describing the motion of electrons around atomic nuclei to target the energies of the starting molecules and the products.

To understand a reaction, chemists must understand the energies of the participating molecules and the products; and to know the energy of each step leading from one molecule to another is to truly know everything about a process. Similarly, in materials science, knowing the energies of 3-D structures is necessary for the prediction of new materials and the study of their properties.

Due to the difficulty of finding the exact solution of the underlying Schrödinger's equation for real systems, approximations are necessary. However, results from approximate descriptions of electron motion are either too simple to reproduce the behavior of many actual molecules (as in the Hartree-Fock method) or too difficult to compute for large systems (as in the case of the Møller-Plesset perturbation theory and the "heavier" methods).

Density functional theory (DFT) and the Hohenberg-Kohn theorems comprised a breakthrough for computational methods in quantum chemistry. The theorems basically say that the average number of electrons located at any one point in space—otherwise called the electron density distribution—contains all the information needed to determine the energy. There is, however, a catch: These theorems do not provide the method for extracting the energy from the electron density. They just say that such a method, the exact functional, exists.

The DFT first appeared in 1927 in the form of Thomas-Fermi model, gained legitimacy in 1964 with the Hohenberg-Kohn proof, and became the method of choice for materials sciences after 1980, when the Generalized Gradient Approximation (GGA) was introduced. Axel D. Becke implemented hybrid functionals in 1993. Today, there are DFT functionals allowing for a very wide range of the computational cost / accuracy ratio. The DFT boosted chemistry and materials science in many ways, and basically handed the power of quantum-chemical modeling to users without a strong physics background. In recognition of this fact, Walter Kohn shared the 1998 Nobel Prize with John A. Pople, who developed many computational methods and was the principal author of an extremely popular Gaussian computational chemistry program.

Dr. Bushmarinov says, "In our lab, we work a lot with electron density-based approaches to study chemical bonding. My colleague Michael Medvedev, currently a Ph.D. student in our lab, had an idea to test how well different functionals reproduce the electron density. Not in the vague 'let's test it' sense—he decided to check the electron density for the smallest 'difficult' systems out there. These systems are atoms and atomic ions with two, four or 10 electrons. We also chucked some anions in initially for good measure. Well, you might ask a question along the lines of 'exactly which insight do you expect to gain from modeling F5+?' This particle has much more in common with plasma physics than with chemistry! I asked such questions myself, I must admit."

Medvedev, calculated the electron densities and their derivatives for the atomic species using almost every "named" functional available in major modern computational chemistry codes, as well as many combinations of those, more than 100 in total. He noticed that the quality of the electron density (ED) seemed to worsen over the years despite the improvements in energies reported in the literature, which raised major questions about the state of the modern DFT. Dr. Bushmarinov developed a rigorous method to compare the produced electron densities to the exact ones and to produce a rating of the functionals sorted by the quality of their densities.

However, the Honenberg-Kohn theorem does not say that the energy of a system can be extracted from just any electron distribution; it states that the exact energy should arise from the exact density.

Medvedev says, "All functionals currently in use are approximations of the exact functional; so how on Earth can they provide better energies from worse densities? On a philosophical level, this seems to contradict the fundamental principle that an algorithm should not produce correct results from faulty data."

At this point, the researchers contacted Professor John P. Perdew of Temple University for an expert opinion and a collaboration. Prof. Perdew pointed out a fundamental flaw in the tests initially used—it turned out the anions should not have been used after all—and helped with all theoretical aspects of the paper. Dr. Jianwei Sun performed the necessary computations using the latest methods developed by J.P.P. and him.

Dr. Bushmarinov says, "After the actual systematic errors were weeded out, the data became beautiful. Just from the RMSD of the electron density, you can see the whole history of DFT in a plot. Basically, until early 2000s, the densities improved along with the theoretical advances. You can even see the functional hierarchy, for which John Perdew in 2001 coined a term 'Jacob's ladder'—functionals become more complex and describe everything better as you go from LDA to GGA to mGGA to hGGA. Until, well, something happens that makes a large fraction of the modern functionals worse than the 1974 functionals."

The actual lists of the worst and the best functionals provided an answer to this apparent conundrum. The best functionals all happened to be derived from solid theoretical approaches—using several routes taken by different groups.

The worst methods, however, are either the pre-1985 methods, as the theory was not yet ready, or they were fully obtained by so-called parameterization. Most of those were developed by taking a flexible functional equation and tuning all ca. 20 parameters with the aim of obtaining the best energies and geometries on a given dataset (usually low hundreds of molecules). This approach provided excellent energies and geometries, in particular on the test sets normally used for functionals' comparison.

The chemical space is, however, vast, and a method performing well on hundreds of molecules can fail when tested on something it was not trained to reproduce—like the small, simple atoms in the current study. It should be noted that some of these "misbehaving" methods are actually very popular. Since they contradict the basics of the theory—they yield "good" energies from "bad" densities—the authors concluded that these methods most likely suffer from some internal problems. And the density functional theory will stray further from the exact functional if this approach to functional developments is not put in check.