When artificial intelligence evaluates chess champions

The ELO system, which most chess federations use today, ranks players by the results of their games. Although simple and efficient, it overlooks relevant criteria such as the quality of the moves players actually make. To overcome these limitations, Jean-Marc Alliot of the Institut de recherche en informatique de Toulouse (IRIT - CNRS/INP Toulouse/Université Toulouse Paul Sabatier/Université Toulouse Jean Jaurès/Université Toulouse Capitole) demonstrates a new system, published on 24 April 2017 in the International Computer Games Association Journal.

Since the 1970s, the system designed by the Hungarian, Arpad Elo, has been ranking chess players according to the result of their games. The best players have the highest ranking, and the difference in ELO points between two players predicts the probability of either player winning a given game. If players perform better or worse than predicted, they either gain or lose points accordingly. This method does not take into account the quality of the moves played during a game and is therefore unable to reliably rank players who have played at different periods. Jean-Marc Alliot suggests a direct ranking of players based on the quality of their actual moves.

His system computes the difference between the move actually played and the move that would have been selected by the best chess program to date, Stockfish. Running on the OSIRIM supercomputer, this program executes almost perfect moves. Starting with those of Wilhelm Steinitz (1836-1900), all 26,000 games played since then by chess world champions have been processed in order to create a probabilistic model for each player. For each position, the model estimates the probability of making a mistake, and the magnitude of the mistake. These models can then be used to compute the win/draw/lose probability for any given match between two players. The predictions have proven not only to be extremely close to the actual results when players have played concrete games against one another, they also fare better than those based on ELO scores. The results demonstrate that the level of chess players has been steadily increasing. The current world champion, Magnus Carlsen, tops the list, while Bobby Fischer is third.

Under current conditions, this new ranking method cannot immediately replace the ELO system, which is easier to set up and implement. However, increases in computing power will make it possible to extend the new method to an ever-growing pool of players in the near future.