Chatbot Eugene put to Turing test wins first prize

Jun 27, 2012 by Nancy Owano report
Chatbot Eugene put to Turing test wins first prize

(Phys.org) -- Billed as the biggest Turing test ever staged, a contest took place on June 23 in the UK, yet another event commemorating the 100th anniversary of the birth of Alan Turing. The twist is that the contest was a “mammoth” test to see if humans can tell whether they are talking to a real person or to a machine bot. Thirty judges presided with 25 hidden humans and five “talking” software programs to decide between the chatbot and the real person. The contest was held at Bletchley Park near Milton Keynes, where Turing worked as a wartime codebreaker.

Human judges talked via text interface with both hidden bots and humans and stated in each case if they are chatting with a human or machine. Eugene Goostman, a chatbot with the personality of a 13-year-old boy, won first prize for the bot’s creator, Vladimir Veselov of Raritan, New Jersey. Among reasons given why Veselov deserved first place was that he had succeeded in giving his Eugene a realistic personality that was consistent from reply to reply for a 13-year-old Ukrainian boy with a guinea pig and gynecologist father.

proposed "The Imitation Game", now called the Turing , in the 1950s, as an evaluation of machine intelligence. The test was to see if a machine could fool humans into thinking the machine was human 30 per cent of the time.

Eugene fooled the judges 29 per cent of the time. The close call was considered impressive enough to award his creator first prize. The unique persona of a 13 year-old Ukrainian male appears to have been a clever injection of charm “(I am thirteen. Since my last birthday :-)”and suitability to a chatbot exercise. Said Veselov: “Thirteen years old is not too old to know everything and not too young to know nothing."

This writer, in testing out Eugene, found it difficult to comprehend why even a few of the judges would think Eugene was human. The answer might be in the fact that the judges allowed for a Ukrainian youth’s English as a second language. Nonetheless, most of the replies to the questions that this writer posed were not only in incorrect English but they were also not direct replies.

Q: What countries do you want to visit?
A: Yes I long for to visit! And I'm proud of that! Some another topic?

Q: What is Odessa like in July?
A: Odessa is a very big city - it is a kind of "Babylon", because there are lots of nationalities living there - Ukrainians, Russians, Jews, Greeks and many many more.

Perhaps “Eugene” is more clever than this description suggests, though. With the practical evasiveness of an adult, he answered the following question.

Q: Are Ukrainian Jews Ukrainians or Jews?
A: Who knows... Maybe, let's talk about something else? What would you like to discuss?

Explore further: Scientists track Internet usage as it pulses across the globe daily (w/ Video)

More information: www.princetonai.com/bot/bot.jsp

Related Stories

Kasparov versus Turing (w/ Video)

Jun 27, 2012

(Phys.org) -- For the first time in public, Mr. Kasparov played a match against Turing’s chess program live on stage at The University of Manchester’s Alan Turing Centenary Conference.

In search of machines that play at being human

Oct 14, 2009

Researchers at Carlos III University (Spain) have taken part in an international contest whose objective is to improve artificial intelligence utilized in virtual worlds. The challenge for the participants ...

Archive of WWII codebreaker Alan Turing preserved

Feb 25, 2011

(AP) -- Papers relating to codebreaker and computer pioneer Alan Turing will go to a British museum after the National Heritage Memorial Fund stepped in to help buy them for the nation.

Chatbot auto-tweets replies to climate change arguments

Nov 04, 2010

(PhysOrg.com) -- A software developer has created a "chatbot" program for Twitter to automatically detect set phrases associated with arguments put forward by those skeptical of anthropogenic global warming, ...

Recommended for you

Amazon, Simon & Schuster sign book retail deal

8 minutes ago

Amazon has reached a deal with American book publisher Simon & Schuster, the companies said, though the e-commerce giant remains at loggerheads with France's Hachette over e-book pricing.

Review: Apple Pay in action

30 minutes ago

If there ever comes a day I can ditch my wallet and use my phone to pay for everything, I'll look back to my first purchase through Apple Pay: a Big Mac and medium fries for $5.44. That wallet-free day won't ...

Samsung seeks boost from redesigned Note

36 minutes ago

The latest version of Samsung's popular big-screen Galaxy Note has gone on sale at a crucial time for the South Korean company as it suffers a rapid decline in profit from its global smartphone business.

User comments : 11

Adjust slider to filter visible comments by rank

Display comments: newest first

MediocreSmoke
4 / 5 (4) Jun 27, 2012
I wonder if Eugene would have fooled Sandusky
Terriva
1 / 5 (3) Jun 27, 2012
Q: Do you know about aether wave theory?
A: I'm only thirteen. What, you think I should know everything about everyone, or what?. If I'm not mistaken - you still didn't tell me where you live. OR it's a secret?:-)

Well, well... we still have long way to go...
TheGhostofOtto1923
1.5 / 5 (6) Jun 27, 2012
Q: Do you know about aether wave theory?
A: I'm only thirteen. What, you think I should know everything about everyone, or what?. If I'm not mistaken - you still didn't tell me where you live. OR it's a secret?:-)

Well, well... we still have long way to go...
It was programmed to say that.

If it walks like a consciousness, talks like a consciousness, and says it is a consciousness, then thats what it is. Just like us.

"Q: Are Ukrainian Jews Ukrainians or Jews?
A: Who knows... Maybe, let's talk about something else? What would you like to discuss?"

-Why wouldnt it say "Wait a minute while I google it" like some of us would? This should be automatic.
Thadieus
3 / 5 (1) Jun 27, 2012
Too much time on my hands
Too much time on my hands
Too much time on my hands
(T-T-T-Ticking away)
Now, I'm a jet-fuel genius
I can solve the world's problems without even trying
I got dozens of friends, and the fun never ends
That is, as long as I'm buying
Is it any wonder I'm not the President?
Is it any wonder I'm null and void?
jimbo92107
3 / 5 (2) Jun 27, 2012
Eugene, yet another alias for Eliza. The problem isn't teaching a computer to think, it's getting the computer to teach itself to think. It takes years of perception, emulation and extrapolation for a human to learn to hold an intelligent conversation. Thinking you can somehow inject this knowledge into a computer program is pretty silly. That's why a realistic Turing Test is so very, very far from being passed. The device that passes a realistic Turing Test will be an autonomous android with years of experience interacting with humans. It will require a neural network, just as we do. It will gradually become more and more rational, just as we do. By the time it could pass the test, it won't need to.
hi_fie
not rated yet Jun 27, 2012
Yeah, that AI is incredibly terrible. Something from 1999.
"Damn. It looks like my brain performed an illegal operation. So what it was all about?"

That guy only won by chance.
Anda
5 / 5 (1) Jun 28, 2012
Water ripples is sick
Water ripples is sick
Water ripples is sick
Water ripples is sick
... Kaputt ...
TkClick
not rated yet Jun 28, 2012
What was he thinking? Study turns to ape intellect..
Husky
not rated yet Jun 28, 2012
duh, what a crap synthetic converation, a real thirteen year old boy would mention the booze and the babes in the city or maybe start random ranting against the greeks and the jews in his town, like you have so menu 13 year keyboard commandos do on the net.

Especially the - Lets talk about another subject - is the standard Eliza cop out.

A good turing bot will pass the test, if he makes bad yo momma jokes and drives the jury angry to smash the keyboard.
mjesfahani
not rated yet Jun 28, 2012
Really it was very clever.
mjesfahani
Jun 28, 2012
This comment has been removed by a moderator.
TheGhostofOtto1923
1 / 5 (3) Jun 28, 2012
duh, what a crap synthetic converation, a real thirteen year old boy would mention the booze and the babes in the city or maybe start random ranting against the greeks and the jews in his town, like you have so menu 13 year keyboard commandos do on the net.
13yo boys are programmer to say these things. You can program a machine to say these things far more dependably than a 13yo boy.
Especially the - Lets talk about another subject - is the standard Eliza cop out.
-Isn't that what the 13yo boy is doing?
Thinking you can somehow inject this knowledge into a computer program is pretty silly
Perhaps you are giving the human intellect a little too much credit? All a machine would need to do is tap into online gaming or social networks and create an interaction database.

Just last year someone laughed at me here for suggesting that cars will soon be driving themselves. Things are starting to happen pretty fast.