New artificial intelligence index tracks the emerging field

January 9, 2018 by ​andrew Myers, Stanford University
“In many ways, we are flying blind in our discussions about artificial intelligence and lack the data we need to credibly evaluate activity.” Credit: Unsplash/Paul Smith

Since the term "artificial intelligence" (AI) was first used in print in 1956, the one-time science fiction fantasy has progressed to the very real prospect of driverless cars, smartphones that recognize complex spoken commands and computers that see.

In an effort to track the progress of this emerging field, a Stanford-led group of leading AI thinkers called the AI100 has launched an index that will provide a comprehensive baseline on the state of artificial intelligence and measure in the same way the and the S&P 500 track the U.S. economy and the broader stock market.

"The AI100 effort realized that in order to supplement its regular review of AI, a more continuous set of collected metrics would be incredibly useful," said Russ Altman, a professor of bioengineering and the faculty director of AI100. "We were very happy to seed the AI Index, which will inform the AI100 as we move forward."

The AI100 was set in motion three years ago when Eric Horvitz, a Stanford alumnus and former president of the Association for the Advancement of Artificial Intelligence, worked with his wife, Mary Horvitz, to define and endow the long-term study. Its first report, released in the fall of 2016, sought to anticipate the likely effects of AI in an urban environment in the year 2030.

Among the key findings in the new index are a dramatic increase in AI startups and investment as well as significant improvements in the technology's ability to mimic human performance.

Baseline metrics

The AI Index tracks and measures at least 18 independent vectors in academia, industry, open-source software and public interest, plus technical assessments of progress toward what the authors call "human-level performance" in areas such as speech recognition, question-answering and computer vision – algorithms that can identify objects and activities in 2-D images. Specific metrics in the index include evaluations of academic papers published, course enrollment, AI-related startups, job openings, search-term frequency and media mentions, among others.

"The goal of the AI Index is to provide a fact-based measuring stick against which we can chart progress and fuel a deeper conversation about the future of the field," Shoham said.

Shoham conceived of the index and assembled a steering committee including Ray Perrault from SRI International, Erik Brynjolfsson of the Massachusetts Institute of Technology and Jack Clark from OpenAI. The committee subsequently hired Calvin LeGassick as project manager.

"The AI Index will succeed only if it becomes a community effort," Shoham said.

Although the authors say the AI Index is the first index to track either scientific or technological progress, there are many other non-financial indexes that provide valuable insight into equally hard-to-quantify fields. Examples include the Social Progress Index, the Middle East peace index and the Bangladesh empowerment index, which measure factors as wide-ranging as nutrition, sanitation, workload, leisure time, public sentiment and even public speaking opportunities.

Intriguing findings

Among the findings of this inaugural index is that the number of active AI startups has increased 14-fold since 2000. Venture capital investment has increased six times in the same period. In academia, publishing in AI has increased a similarly impressive nine times in the last 20 years while course enrollment has soared. Enrollment in the introductory AI-related machine learning course at Stanford, for instance, has grown 45-fold in the last 30 years.

In technical metrics, image and speech recognition are both approaching, if not surpassing, human-level performance. The authors noted that AI systems have excelled in such real-world applications as object detection, the ability to understand and answer questions and classification of photographic images of skin cancer cells.

Shoham noted that the report is still very U.S.-centric and will need a greater international presence as well as a greater diversity of voices. He said he also sees opportunities to fold in government and corporate investment in addition to the venture capital funds that are currently included.

In terms of human-level performance, the AI Index suggests that in some ways AI has already arrived. This is true in game-playing applications including chess, the Jeopardy! game show and, most recently, the game of Go. Nonetheless, the authors note that computers continue to lag considerably in the ability to generalize specific information into deeper meaning.

"AI has made truly amazing strides in the past decade," Shoham said, "but computers still can't exhibit the common sense or the general intelligence of even a 5-year-old."

Explore further: Artificial intelligence index tracks emerging field

Related Stories

Artificial intelligence index tracks emerging field

December 1, 2017

Since the term "artificial intelligence" (AI) was first used in print in 1956, the one-time science fiction fantasy has progressed to the very real prospect of driverless cars, smartphones that recognize complex spoken commands ...

High fatty liver index tied to colorectal adenomas

December 18, 2017

(HealthDay)—The fatty liver index may be an accurate predictor of colorectal adenomas among an average-risk population, according to a study published in the January issue of Diseases of the Colon & Rectum.

Carnegie Mellon University launches carbon emissions index

March 20, 2017

Mitsubishi Hitachi Power Systems (MHPS) and Carnegie Mellon University (CMU) today announced the creation of a new index that will measure carbon dioxide emissions from the U.S. electrical power generation sector. The Carnegie ...

Indonesia launches eco-friendly investment index

June 9, 2009

The Indonesian stock exchange and a biodiversity foundation have launched an investment index to raise awareness of companies' environmental track records, an environmental group said.

Recommended for you

Permanent, wireless self-charging system using NIR band

October 8, 2018

As wearable devices are emerging, there are numerous studies on wireless charging systems. Here, a KAIST research team has developed a permanent, wireless self-charging platform for low-power wearable electronics by converting ...

Facebook launches AI video-calling device 'Portal'

October 8, 2018

Facebook on Monday launched a range of AI-powered video-calling devices, a strategic revolution for the social network giant which is aiming for a slice of the smart speaker market that is currently dominated by Amazon and ...

Artificial enzymes convert solar energy into hydrogen gas

October 4, 2018

In a new scientific article, researchers at Uppsala University describe how, using a completely new method, they have synthesised an artificial enzyme that functions in the metabolism of living cells. These enzymes can utilize ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.