Chatbot Eugene put to Turing test wins first prize

(Phys.org) -- Billed as the biggest Turing test ever staged, a contest took place on June 23 in the UK, yet another event commemorating the 100th anniversary of the birth of Alan Turing. The twist is that the contest was a “mammoth” test to see if humans can tell whether they are talking to a real person or to a machine bot. Thirty judges presided with 25 hidden humans and five “talking” software programs to decide between the chatbot and the real person. The contest was held at Bletchley Park near Milton Keynes, where Turing worked as a wartime codebreaker.

Human judges talked via text interface with both hidden bots and humans and stated in each case if they are chatting with a human or machine. Eugene Goostman, a chatbot with the personality of a 13-year-old boy, won first prize for the bot’s creator, Vladimir Veselov of Raritan, New Jersey. Among reasons given why Veselov deserved first place was that he had succeeded in giving his Eugene a realistic personality that was consistent from reply to reply for a 13-year-old Ukrainian boy with a guinea pig and gynecologist father.

Alan Turing proposed "The Imitation Game", now called the Turing Test, in the 1950s, as an evaluation of machine intelligence. The test was to see if a machine could fool humans into thinking the machine was human 30 per cent of the time.

Eugene fooled the judges 29 per cent of the time. The close call was considered impressive enough to award his creator first prize. The unique persona of a 13 year-old Ukrainian male appears to have been a clever injection of charm “(I am thirteen. Since my last birthday :-)”and suitability to a chatbot exercise. Said Veselov: “Thirteen years old is not too old to know everything and not too young to know nothing."

This writer, in testing out Eugene, found it difficult to comprehend why even a few of the judges would think Eugene was human. The answer might be in the fact that the judges allowed for a Ukrainian youth’s English as a second language. Nonetheless, most of the replies to the questions that this writer posed were not only in incorrect English but they were also not direct replies.

Q: What countries do you want to visit?
A: Yes I long for to visit! And I'm proud of that! Some another topic?

Q: What is Odessa like in July?
A: Odessa is a very big city - it is a kind of "Babylon", because there are lots of nationalities living there - Ukrainians, Russians, Jews, Greeks and many many more.

Perhaps “Eugene” is more clever than this description suggests, though. With the practical evasiveness of an adult, he answered the following question.

Q: Are Ukrainian Jews Ukrainians or Jews?
A: Who knows... Maybe, let's talk about something else? What would you like to discuss?

© 2012 Phys.Org