Artificially intelligent game bots pass the Turing test on Turing's centenary

Sep 26, 2012
UT^2 game bot faces off against an opponent. Credit: Jacob Schrum

An artificially intelligent virtual gamer created by computer scientists at The University of Texas at Austin has won the BotPrize by convincing a panel of judges that it was more human-like than half the humans it competed against.

The competition was sponsored by and was set inside the virtual world of "Unreal Tournament 2004," a first-person shooter video game. The winners were announced this month at the IEEE Conference on and Games.

"The idea is to evaluate how we can make game bots, which are nonplayer characters (NPCs) controlled by AI algorithms, appear as human as possible," said Risto Miikkulainen, professor of computer science in the College of Natural Sciences. Miikkulainen created the bot, called the UT^2 game bot, with doctoral students Jacob Schrum and Igor Karpov.

The bots face off in a tournament against one another and about an equal number of humans, with each player trying to score points by eliminating its opponents. Each player also has a "judging gun" in addition to its usual complement of weapons. That gun is used to tag opponents as human or bot.

This video is not supported by your browser at this time.
Under heavy fire from a human judge, UT^2 manages to fight his way to a nearby weapon and obliterate his opponent. Credit: Courtesy of Jacob Schrum

The bot that is scored as most human-like by the human judges is named the winner. UT^2, which won a warm-up competition last month, shared the honors with MirrorBot, which was programmed by Romanian computer scientist Mihai Polceanu.

The winning bots both achieved a humanness rating of 52 percent. Human players received an average humanness rating of only 40 percent. The two winning teams will split the $7,000 first prize.

The victory comes 100 years after the birth of mathematician and computer scientist , whose "Turing test" stands as one of the foundational definitions of what constitutes true machine intelligence. Turing argued that we will never be able to see inside a machine's hypothetical consciousness, so the best measure of machine sentience is whether it can fool us into believing it is human.

"When this 'Turing test for game bots' competition was started, the goal was 50 percent humanness," said Miikkulainen. "It took us five years to get there, but that level was finally reached last week, and it's not a fluke."

This video is not supported by your browser at this time.
This is a combat clip, which is seen from the perspective of a human judge (whose game name is Miguel) as he's killed by the UT^2 game bot (game name Ty). Credit: Courtesy of Jacob Schrum

The complex gameplay and 3-D environments of "Unreal Tournament 2004" require that bots mimic humans in a number of ways, including moving around in 3-D space, engaging in chaotic combat against multiple opponents and reasoning about the best strategy at any given point in the game. Even displays of distinctively human irrational behavior can, in some cases, be emulated.

"People tend to tenaciously pursue specific opponents without regard for optimality," said Schrum. "When humans have a grudge, they'll chase after an enemy even when it's not in their interests. We can mimic that behavior."

In order to most convincingly mimic as much of the range of human behavior as possible, the team takes a two-pronged approach. Some behavior is modeled directly on previously observed human behavior, while the central battle behaviors are developed through a process called neuroevolution, which runs artificially intelligent neural networks through a survival-of-the-fittest gauntlet that is modeled on the biological process of evolution.

Networks that thrive in a given environment are kept, and the less fit are thrown away. The holes in the population are filled by copies of the fit ones and by their "offspring," which are created by randomly modifying (mutating) the survivors. The simulation is run for as many generations as are necessary for networks to emerge that have evolved the desired behavior.

"In the case of the BotPrize," said Schrum, "a great deal of the challenge is in defining what 'human-like' is, and then setting constraints upon the neural networks so that they evolve toward that behavior.

"If we just set the goal as eliminating one's enemies, a bot will evolve toward having perfect aim, which is not very human-like. So we impose constraints on the bot's aim, such that rapid movements and long distances decrease accuracy. By evolving for good performance under such behavioral constraints, the bot's skill is optimized within human limitations, resulting in behavior that is good but still human-like."

Miikkulainen said that methods developed for the BotPrize competition should eventually be useful not just in developing games that are more entertaining, but also in creating virtual training environments that are more realistic, and even in building robots that interact with humans in more pleasant and effective ways.

Explore further: UT Dallas professor to develop framework to protect computers' cores

Related Stories

In search of machines that play at being human

Oct 14, 2009

Researchers at Carlos III University (Spain) have taken part in an international contest whose objective is to improve artificial intelligence utilized in virtual worlds. The challenge for the participants ...

Chatbot Eugene put to Turing test wins first prize

Jun 27, 2012

(Phys.org) -- Billed as the biggest Turing test ever staged, a contest took place on June 23 in the UK, yet another event commemorating the 100th anniversary of the birth of Alan Turing. The twist is that ...

Touch typists could help stop spammers in their tracks

Jul 15, 2009

(PhysOrg.com) -- Computer scientists at Newcastle University are about to give office workers a perfect excuse to play games: it's all in the name of research. Dr Jeff Yan, together with his PhD student Su-Yang ...

Spot a bot to stop a botnet

May 01, 2012

Computer scientists in India have developed a two-pronged algorithm that can detect the presence of a botnet on a computer network and block its malicious activities before it causes too much harm. The team ...

AI's Emmy award goes to Swarmanoid robot book heist

Aug 16, 2011

(PhysOrg.com) -- Robots removing a book off a shelf? Hardly what you would consider worthy of a video award. Not unless you’re talking about a special platoon of robots known as a swarmanoid. Their book ...

Recommended for you

User comments : 20

Adjust slider to filter visible comments by rank

Display comments: newest first

Lurker2358
1 / 5 (3) Sep 26, 2012
I can tell from the video that the Judge sucks at FPS games, and the bot would be DEAD if I was the judge, definitely if I was 20 years old anyway.

Anything parking for that long dies against half decent FPS video game players in their prime.

If you really want to test a bot, try making a bot that can play Starcraft:Broodwar like a human, or Starcraft 2 like a human.

Here's where you stand:

I am considered C in player vs player. Profession is A and is 3 leagues above me.

I can beat 1vs7 computers in Broodwar and 1vs5 insane computers in Starcraft 2.

When you make a computer that can consistently beat me 1vs1 in a best of 5 matchup, or even a best of 3 matchup, then you have an A.I.
Tangent2
3.7 / 5 (3) Sep 26, 2012
The winning bots both achieved a humanness rating of 52 percent. Human players received an average humanness rating of only 40 percent.


With that said, it seems like there is either a problem with the way we perceive "humanness" or there is a problem with our preconceived notion of "humanness", as one would expect the human players to score better than the bots for "humanness" on average.
sirchick
not rated yet Sep 26, 2012
Why use such ancient gaming tech to do this... no physics in the bullets at all there... most games have bullet drop and bullet spread which you have to take into account when shooting.
Magus
not rated yet Sep 26, 2012
If you really want to to seem more human don't give it the physics data. Let it figure it out by pattern and depth analysis the AI visual perspective.
El_Nose
3 / 5 (2) Sep 26, 2012
i do not believe winning or killing was the evaluation for humanness ... i believe it had different metrics such as behaving as a human might... such as going for a more powerful weapon but only after you move away from it
TehDog
not rated yet Sep 26, 2012
Not impressed with the quality of those vids, the bots seem adequate.

"Anything parking for that long dies against half decent FPS video game players in their prime."

Camping a high value position is a well understood strategy for online players, do it myself a lot.
Best bots I ever played against were the Omega bots for Quake 1. Smart mouthed route learning bastards they were. Written by the guy who did the bots in Q3. Could even adapt to new maps. Did they seem human? In play, yes, but their chat was heavily limited.
klawy
1 / 5 (3) Sep 26, 2012
Mohaha ridiculous... "controlled by AI algorithms"

Can't we just start programming a normal computer with quantum algorithms and whoops we have our quantum computer!!!

Stop researching and start programming apparently you can do anything with it! If you can program AI you can program a quantum computer!
StarGazer2011
2.6 / 5 (5) Sep 26, 2012
This isnt the Turing test; its nothing like the Turing test. The Turing test required a computer to CONVERSE with a human and fool the human; not play FPS'!
This is impressive tech and will likely improve video games, but its nothing to do with the Turing test.
Shabs42
not rated yet Sep 27, 2012
"When humans have a grudge, they'll chase after an enemy even when it's not in their interests. We can mimic that behavior."


Sounds like a horrible computer virus of the future.
alfie_null
not rated yet Sep 27, 2012
It would be interesting if significant advances in AI end up being accomplished in support of games (as opposed to all the other "serious" areas of computer science research).

I, for one, welcome our new NPC overlords.
roboferret
not rated yet Sep 27, 2012
All a game bot needs to appear human is to randomly call people "fags" and complain about lag.
Jimbaloid
5 / 5 (1) Sep 27, 2012
When you make a computer that can consistently beat me 1vs1 in a best of 5 matchup, or even a best of 3 matchup, then you have an A.I.


The idea wasn't to make the best or most successful bot player, but to make the most believably human-like bot player.

Perhaps some of the real human players were just a bit too good, even too analytical, such that they actually made themselves appear less human to the judges. Maybe the scoring had more to do with the judges expectation for how a bot might behave. Matching player and bot ability would be tricky.

You could say they made it suck at the game play by just enough, in just the right ways, against these opponents.
CreepyD
not rated yet Sep 27, 2012
Part of the problem is, the better a player, then the more bot-like it will appear.
A good player will ahve everything perfectly timed down to the second, just as a bot will.
It will aim almost perfectly, again just as a good bot would.
The differences would be very subtle - if a bot can show learning and anticipation over the course of a single game, then they've cracked it. I highly doubt that's been acheived.
antialias_physorg
5 / 5 (1) Sep 27, 2012
It's an interesting approach. The more 'real life' a Turing test is (texting, speech, full person interaction, ...) the harder it is for a bot/AI to pass the test.
So they took the approach of not tweaking the AI up but tweaking the system down to a level where a bot could pass.

It's pretty neat because now we have a known passing entity (for that level) which can be built upon to pass higher levels.

Why use such ancient gaming tech to do this... no physics in the bullets at all there..

Because there's no added advantage of having better physics with respect to the test. This is not a game. The main focus is on finding a platfom on which implementing the required adaptations is easy to do.

seb
not rated yet Sep 27, 2012
Pretty sure the default game bots of UT would fool these judges..
TrinityComplex
not rated yet Sep 27, 2012
Why use such ancient gaming tech to do this... no physics in the bullets at all there..


UT2004 has a well established, easy to adapt code for virtually any test that it might be used for. Reprogramming a newer game with better physics introduces more complexity that would not be worth whatever return on the results they would get.
xeoroex
not rated yet Sep 27, 2012
These types of 'Turing' tests i think greatly under-represent the true test/goal of AI. the test shouldnt be to be precieved as a human doing X , but also being only given the same resources as the humans.

IF these bots using only visual input and keyboard/mouse output could learn these maps and play at a human level then it would be impressive.

instead they dumb down existing bots that use gameserver side information(getting raw info of player locations and collision traces).

In a similar way the Chatbot test is also not useful , because too much focus has been on just FAKING they can have a conversation to be believable for the brief time of being judged. none of them have ever been convincing past the first few minutes, and they should be judged after atleast an hour.

on the positive side , developing this along side something like elderscroll's Radiant Ai to create a more natural NPC intereaction would be great.
sirchick
not rated yet Sep 29, 2012
Why use such ancient gaming tech to do this... no physics in the bullets at all there..


UT2004 has a well established, easy to adapt code for virtually any test that it might be used for. Reprogramming a newer game with better physics introduces more complexity that would not be worth whatever return on the results they would get.


Course it is - we humans adjust from things like bullet drops not from calculating them out but by adjusting what we visually see. AI needs to also realise things like bullet drop and adjust by observing the bullet didn't go exactly where they planned it to.

That would be more "human" in nature, unless the AI uses maths to figure it out. No soldier has the time for that, they adjust their instincts.
Urgelt
5 / 5 (1) Oct 01, 2012
Uh, no.

The Turing Test wasn't about controlling avatars in a simplistic combat game. It was about conversation between machines and humans, where humans can't tell if the conversationalist is a machine or not. Nobody is anywhere close to programming a machine to pass that test.

The 'bots didn't pass the Turing Test. Their designers didn't even *try* to pass the Turing Test. They made up their own test and passed it, but Alan Turing's ideas had nothing to do with what they did.
antialias_physorg
not rated yet Oct 01, 2012
Nobody is anywhere close to programming a machine to pass that test.

Don't be so sure
http://cleverbot.com/human

The score is 59.3% for the bot and 63.3% for the human. That's mighty close.

More news stories

Health care site flagged in Heartbleed review

People with accounts on the enrollment website for President Barack Obama's signature health care law are being told to change their passwords following an administration-wide review of the government's vulnerability to the ...

Airbnb rental site raises $450 mn

Online lodging listings website Airbnb inked a $450 million funding deal with investors led by TPG, a source close to the matter said Friday.

Researchers uncover likely creator of Bitcoin

The primary author of the celebrated Bitcoin paper, and therefore probable creator of Bitcoin, is most likely Nick Szabo, a blogger and former George Washington University law professor, according to students ...