Most people overlook artificial intelligence despite flawless advice

January 30, 2019, The Army Research Laboratory
A team of researchers including Dr. James Schaffer, scientist for RDECOM's Army Research Laboratory, the Army's corporate research laboratory (ARL) stationed at ARL West in Playa Vista, California, recently discovered that most people overlook artificial intelligence despite flawless advice. AI-like systems will be an integral part of the Army's strategy over the next five years, so system designers will need to start getting a bit more creative in order to appeal to users. (U.S. Army Graphic by Jhi Scott) Credit: U.S. Army Graphic by Jhi Scott

If you were convinced you knew the way home, would you still turn on your GPS?

Army scientists recently attempted to answer a similar question due to an ongoing concern that artificial intelligence, which can be opaque and frustrating to many people, may not be helpful in battlefield decision making.

"The U.S. Army continues to push the modernization of its forces, with notable efforts including the development of smartphone-based software for real-time information delivery such as the Android Tactical Assault Kit, or ATAK, and the allocation of significant funding towards researching new AI and machine learning methods to assist command and control personnel," said Dr. James Schaffer, scientist for RDECOM's Army Research Laboratory, the Army's corporate research laboratory (ARL), at ARL West in Playa Vista, California.

According to Schaffer, despite these advances, a significant gap in basic knowledge about the use of AI still exists, and it is unknown which factors of AI will or will not help military decision-making processes.

University and corporate research has made significant headway into solving this problem for applications like movie and restaurant recommendations, but the findings do not exactly translate to the military world.

"For instance, many research studies and A/B testing, such as those performed by Amazon, have experimented with different forms of persuasion, argumentation and user interface styles to determine the winning combination that moves the most product or inspires the most trust," Schaffer said. "Unfortunately, there are big gaps between the assumptions in these low-risk domains and military practice."

The Army's research, which was a collaboration between Army scientists and university researchers at the University of California, Santa Barbara, hypothesizes that many people trust their own abilities far more than that of a computer, which will affect their judgment when pressured to perform.

According to Schaffer, this implies that even if flawless AI could be created, some people would not listen to the AI's advice.

The researchers constructed an abstract similar to the Iterated Prisoner's Dilemma a game where players must choose to cooperate with or defect against their co-players in every round—in order to control all relevant factors.

The Iterated Prisoner's Dilemma has been used in regards to several real-world problems, such as military arms races, public sharing of resources and international politics.

An online version of the game was developed by the research team, where players obtained points by making good decisions in each round.

An AI was used to generate advice in each round, which was shown alongside the game interface, and made a suggestion about which decision should be made by the player.

Researchers had an opportunity to design an AI that always recommended the optimal course of action.

However, just like in real life, players were required to access the AI's advice manually, just as you must manually switch on GPS, and were free to accept or ignore its suggestion.

The researchers also presented different versions of this AI—some were deliberately inaccurate, some required game information to be entered manually, and some justified their suggestions with rational arguments.

All variations of these AI treatments were tested so that interaction effects between AI configurations could be studied.

People were invited to play the game online and researchers collected a profile of each player and monitored their behavior.

For each player, researchers asked about their familiarity with the game while also measuring their true competency.

Additionally, a test was given halfway through playing that measured awareness of gameplay elements.

"What was discovered might trouble some advocates of AI two-thirds of human decisions disagreed with the AI, regardless of the number of errors in the suggestions," Schaffer said.

The higher the player estimated their familiarity with the game beforehand, the less the AI was used, an effect that was still observed when controlling for the AI's accuracy. This implies that improving a system's accuracy will not be able to increase system adoption in this population.

"This might be a harmless outcome if these players were really doing better but they were in fact performing significantly worse than their humbler peers, who reported knowing less about the game beforehand," Schaffer said. "When the AI attempted to justify its suggestions to players who reported high familiarity with the game, reduced awareness of gameplay elements was observed a symptom of over-trusting and complacency."

Despite these findings, a corresponding increase in agreement with AI suggestions was not observed.

This presents a catch-22 for system designers: incompetent users need the AI most of all, but are the least likely to be swayed by rational justifications, Schaffer said.

Incompetent users were also the most likely to say that they trusted the AI, which was studied through a post-game questionnaire.

"This contrasts sharply with their observed neglect of the AI's suggestions, demonstrating that people are not always honest, or may not always be aware of their own behavior," Schaffer said.

For Schaffer and the team, this research highlights ongoing issues in the usability of complex, opaque systems such as AI, despite continued advances in accuracy, robustness and speed.

"Rational arguments have been demonstrated to be ineffective on some people, so designers may need to be more creative in designing interfaces for these systems," Schaffer said.

Schaffer said this could be accomplished through appealing to emotions or competitiveness, or even by removing presence from the AI, such that users do not register its presence and thus do not anchor on their own abilities.

"Despite challenges in human-computer interaction, AI-like systems will be an integral part of the Army's strategy over the next five years," Schaffer said. "One of the principle challenges facing military operations today is rapid response from guerilla adversaries, who often have shorter command chains and thus can act and react more rapidly than the U.S. Armed Forces. Complex systems that can rapidly react to a changing environment and expedite information flow can improve response times and help maintain op-tempo but only if given sufficient trust by its users."

The research group continues to experiment with different interfaces for AI systems so that all types of people can benefit from increasingly effective automated knowledge.

This research will appear in the proceedings of the ACM's 2019 conference on Intelligent User Interfaces ( ).

Explore further: Does technology really enhance our decision-making ability?

Related Stories

Does technology really enhance our decision-making ability?

September 7, 2018

An Army scientist recently won a best paper award at the Association for Computing Machinery's 26th Conference on User Modeling, Adaptation and Personalization for discovering that most people cannot distinguish between liking ...

China's army develops 'online war game'

June 29, 2011

After setting up its own cyber-warfare team, China's military has now developed its first online war game aimed at improving combat skills and battle awareness, state press said Wednesday.

Recommended for you

Archaeologists discover Incan tomb in Peru

February 16, 2019

Peruvian archaeologists discovered an Incan tomb in the north of the country where an elite member of the pre-Columbian empire was buried, one of the investigators announced Friday.

Where is the universe hiding its missing mass?

February 15, 2019

Astronomers have spent decades looking for something that sounds like it would be hard to miss: about a third of the "normal" matter in the Universe. New results from NASA's Chandra X-ray Observatory may have helped them ...

What rising seas mean for local economies

February 15, 2019

Impacts from climate change are not always easy to see. But for many local businesses in coastal communities across the United States, the evidence is right outside their doors—or in their parking lots.

The friendly extortioner takes it all

February 15, 2019

Cooperating with other people makes many things easier. However, competition is also a characteristic aspect of our society. In their struggle for contracts and positions, people have to be more successful than their competitors ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.