Jump to content

Loebner Prize

From Wikipedia, the free encyclopedia

This is an old revision of this page, as edited by Fobos222 (talk | contribs) at 21:00, 4 December 2016 (→Contests). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

The Loebner Prize is an annual competition in artificial intelligence that awards prizes to the chatterbot considered by the judges to be the most human-like. The format of the competition is that of a standard Turing test. In each round, a human judge simultaneously holds textual conversations with a computer program and a human being via computer. Based upon the responses, the judge must decide which is which.

The contest was launched in 1990 by Hugh Loebner in conjunction with the Cambridge Center for Behavioral Studies, Massachusetts, United States. It has since been associated with Flinders University, Dartmouth College, the Science Museum in London, University of Reading and Ulster University, Magee Campus, Derry, UK City of Culture. In 2004 and 2005, it was held in Loebner's apartment in New York City. Within the field of artificial intelligence, the Loebner Prize is somewhat controversial; the most prominent critic, Marvin Minsky, called it a publicity stunt that does not help the field along.[1]

Prizes

Originally, $2,000 was awarded for the most human-seeming chatterbot in the competition. The prize was $3,000 in 2005 and $2,250 in 2006. In 2008, $3,000 was awarded.

In addition, there are two one-time-only prizes that have never been awarded. $25,000 is offered for the first chatterbot that judges cannot distinguish from a real human and which can convince judges that the human is the computer program. $100,000 is the reward for the first chatterbot that judges cannot distinguish from a real human in a Turing test that includes deciphering and understanding text, visual, and auditory input. Once this is achieved, the annual competition will end.

Competition rules and restrictions

The rules have varied over the years and early competitions featured restricted conversation Turing tests but since 1995 the discussion has been unrestricted.[2]

For the three entries in 2007, Robert Medeksza, Noah Duncan and Rollo Carpenter,[3] some basic "screening questions" were used by the sponsor to evaluate the state of the technology. These included simple questions about the time, what round of the contest it is, etc.; general knowledge ("What is a hammer for?"); comparisons ("Which is faster, a train or a plane?"); and questions demonstrating memory for preceding parts of the same conversation. "All nouns, adjectives and verbs will come from a dictionary suitable for children or adolescents under the age of 12." Entries did not need to respond "intelligently" to the questions to be accepted.

For the first time in 2008 the sponsor allowed introduction of a preliminary phase to the contest opening up the competition to previously disallowed web-based entries judged by a variety of invited interrogators. The available rules do not state how interrogators are selected or instructed. Interrogators (who judge the systems) have limited time: 5 minutes per entity in the 2003 competition, 20+ per pair in 2004–2007 competitions, 5 minutes to conduct simultaneous conversations with a human and the program in 2008-2009, increased to 25 minutes of simultaneous conversation since 2010.

Criticisms

The prize has long been scorned by experts in the field,[4] for a variety of reasons.

It is regarded by many as a publicity stunt.[5][6] Marvin Minsky scathingly offered a "prize" to anyone who could stop the competition.[7] The criticism was reinforced when Loebner, resorting to word-play, claimed that Minsky's offering a prize to stop the competition made him a co-sponsor!

The rules of the competition have encouraged poorly qualified judges to make rapid judgements. Interactions between judges and competitors was originally very brief, for example effectively 2.5 mins of questioning, which permitted only a few questions.[5] Questioning was initially restricted to "whimsical conversation",[4] a domain suiting standard chatbot tricks.[8]

Competition entrants do not aim at understanding or intelligence but resort to basic ELIZA style tricks,[5][9] and successful entrants find deception and pretense is rewarded.[10]

Reporting of the annual competition often confuses the imitation test with intelligence,[11] a typical example being The Atlantic's introduction[12] stating that "in the race to build computers that can think like humans, the proving ground is the Turing Test".

Contests

2006

In 2006, the contest was organised by Tim Child (CEO of Televirtual) and Huma Shah.[13][14] On August 30, the four finalists were announced:

  • Rollo Carpenter
  • Richard Churchill and Marie-Claire Jenkins
  • Noah Duncan
  • Robert Medeksza

The contest was held on 17 September in the VR theatre, Torrington Place campus of University College London. The judges included the University of Reading's cybernetics professor, Kevin Warwick, a professor of artificial intelligence, John Barnden (specialist in metaphor research at the University of Birmingham), a barrister, Victoria Butler-Cole and a journalist, Graham Duncan-Rowe. The latter's experience of the event can be found in an article in Technology Review.[15][16] The winner was 'Joan', based on Jabberwacky, both created by Rollo Carpenter.

2007

The 2007 competition was held on October 21 in New York City. The judges were: computer science professor Russ Abbott, philosophy professor Hartry Field, psychology assistant professor Clayton Curtis and English lecturer Scott Hutchins.[17]

No bot passed the Turing test, but the judges ranked the three contestants as follows:

  • 1st: Robert Medeksza from Zabaware, creator of Ultra Hal Assistant
  • 2nd: Noah Duncan, a private entry, creator of Cletus
  • 3rd: Rollo Carpenter from Icogno, creator of Jabberwacky

The winner received $2,250 and the annual medal. The runners-up received $250 each.

2008

The 2008 competition was organised by professor Kevin Warwick, coordinated by Huma Shah and held on October 12 at the University of Reading, UK.[18] After testing by over one hundred judges during the preliminary phase, in June and July 2008, six finalists were selected from thirteen original entrants - artificial conversational entity (ACE). Five of those invited competed in the finals:

  • Brother Jerome, Peter Cole and Benji Adams
  • Elbot, Fred Roberts / Artificial Solutions
  • Eugene Goostman, Vladimir Veselov, Eugene Demchenko and Sergey Ulasen
  • Jabberwacky, Rollo Carpenter
  • Ultra Hal, Robert Medeksza

In the finals, each of the judges was given five minutes to conduct simultaneous, split-screen conversations with two hidden entities. Elbot[19] of Artificial Solutions[20] won the 2008 Loebner Prize bronze award, for most human-like artificial conversational entity, through fooling three of the twelve judges who interrogated it (in the human-parallel comparisons) into believing it was human. This is coming very close to the 30% traditionally required to consider that a program has actually passed the Turing test. Eugene Goostman[21] and Ultra Hal[22] both deceived one judge each that it was the human.

Will Pavia, a journalist for The Times, has written about his experience; a Loebner finals' judge, he was deceived by Elbot and Eugene.[23] Kevin Warwick and Huma Shah have reported on the parallel-paired Turing tests.[24]

2009

The 2009 Loebner Prize Competition was held September 6, 2009 at the Brighton Centre, Brighton UK in conjunction with the Interspeech 2009 conference. The prize amount for 2009 was $3,000.

Entrants were David Levy, Rollo Carpenter, and Mohan Embar, who finished in that order.

The writer Brian Christian participated in the 2009 Loebner Prize Competition as a human confederate, and described his experiences at the competition in his book The Most Human Human.

2010

The 2010 Loebner Prize Competition was held on October 23 at California State University, Los Angeles. The 2010 competition was the 20th running of the contest.

2011

The 2011 Loebner Prize Competition was held on October 19 at the University of Exeter, Devon, United Kingdom. The prize amount for 2011 was $4,000.

The four finalists and their chatterbots were Bruce Wilcox (Rosette), Adeena Mignogna (Zoe), Mohan Embar (Chip Vivant) and Ron Lee (Tutor), who finished in that order.

That year there was an addition of a panel of junior judges, namely Jean-Paul Astal-Stain, William Dunne, Sam Keat and Kirill Jerdev. The results of the junior contest were markedly different from the main contest, with chatterbots Tutor and Zoe tying for first place and Chip Vivant and Rosette coming in third and fourth place, respectively.

2012

The 2012 Loebner Prize Competition was held on the 15th of May in Bletchley Park in Bletchley, Buckinghamshire, England, in honor of the Alan Turing centenary celebrations. The prize amount for 2012 was $5,000. The local arrangements organizer was David Levy, who won the Loebner Prize in 1997 and 2009.

The four finalists and their chatterbots were Mohan Embar (Chip Vivant), Bruce Wilcox (Angela), Daniel Burke (Adam), M. Allan (Linguo), who finished in that order.

That year, a team from the University of Exeter's computer science department (Ed Keedwell, Max Dupenois and Kent McClymont) conducted the first-ever live webcast of the conversations.[25]

2013

The 2013 Loebner Prize Competition was held, for the first time on the Island of Ireland, on September 14 at the Ulster University, Magee College, Derry, Northern Ireland, UK.

The four finalists and their chatbots were Steve Worswick (Mitsuku), Dr. Ron C. Lee (Tutor), Bruce Wilcox (Rose) and Brian Rigsby (Izar), who finished in that order.

The judges were Professor Roger Schank (Socratic Arts), Professor Noel Sharkey (Sheffield University), Professor Minhua (Eunice) Ma (Huddersfield University, then University of Glasgow) and Professor Mike McTear (Ulster University).

For the 2013 Junior Loebner Prize Competition the chatbots Mitsuku and Tutor tied for first place with Rose and Izar in 3rd and 4th place respectively.

2014

The 2014 Loebner Prize Competition was held at Bletchley Park, England, on Saturday 15 November 2014. The event was filmed live by Sky News. The guest judge was television presenter and broadcaster James May.

After 2 hours of judging, 'Rose' by Bruce Wilcox was declared the winner. Bruce will receive a cheque for $4000 and a bronze medal. The ranks were as follows:

Rose - Rank 1 ($4000 & Bronze Medal); Izar - Rank 2.25 ($1500); Uberbot - Rank 3.25 ($1000); and Mitsuku - Rank 3.5 ($500).

The Judges were Dr Ian Hocking, Writer & Senior Lecturer in Psychology, Christ Church College, Canterbury; Dr Ghita Kouadri-Mostefaoui, Lecturer in Computer Science and Technology, University of Bedfordshire; Mr James May, Television Presenter and Broadcaster; and Dr Paul Sant, Dean of UCMK, University of Bedfordshire.

2015

The 2015 Loebner Prize Competition was again won by 'Rose' by Bruce Wilcox.[26]

The judges were Jacob Aaron, Physical sciences reporter for New Scientist; Rory Callan, Jones Technology correspondent for the BBC; Brett Marty, Film Director and Photographer; Ariadne Tampion, Writer.

2016

The 2016 Loebner Prize was held at Bletchley Park on 17th September 2016. After 2 hours of judging the final results were announced. The ranks were as follows:

  • 1st place: Mitsuku
  • 2nd place: Tutor
  • 3rd place: Rose

Winners

Official list of winners.[27]

Year Winner Program
1991 Joseph Weintraub PC Therapist
1992 Joseph Weintraub PC Therapist
1993 Joseph Weintraub PC Therapist
1994 Thomas Whalen TIPS
1995 Joseph Weintraub PC Therapist
1996 Jason Hutchens HeX
1997 David Levy Converse
1998 Robby Garner Albert One
1999 Robby Garner Albert One
2000 Richard Wallace Artificial Linguistic Internet Computer Entity (A.L.I.C.E.)
2001 Richard Wallace Artificial Linguistic Internet Computer Entity (A.L.I.C.E.)
2002 Kevin Copple Ella
2003 Juergen Pirner Jabberwock
2004 Richard Wallace Artificial Linguistic Internet Computer Entity (A.L.I.C.E.)
2005 Rollo Carpenter George (Jabberwacky)
2006 Rollo Carpenter Joan (Jabberwacky)
2007 Robert Medeksza Ultra Hal
2008 Fred Roberts Elbot
2009 David Levy Do-Much-More
2010 Bruce Wilcox Suzette
2011 Bruce Wilcox Rosette[28]
2012 Mohan Embar Chip Vivant[29]
2013 Steve Worswick Mitsuku[30]
2014 Bruce Wilcox Rose
2015 Bruce Wilcox Rose
2016 Steve Worswick Mitsuku

See also

References

  1. ^ Artificial stupidity, Salon.com, 16 February 2003
  2. ^ 2007 rules, 2008 rules and 2009 rules
  3. ^ 17th Annual Loebner Prize for Artificial Intelligence 21 October 2007 New York City
  4. ^ a b Powers, David. "The Total Turing Test and the Loebner Prize". Retrieved 29 May 2016.
  5. ^ a b c Floridi, Luciano; Taddeo, Mariarosaria; Turilli, Matteo (2009). "Turing's Imitation Game: Still an Impossible Challenge for All Machines and Some Judges––An Evaluation of the 2008 Loebner Contest". Minds & Machines (19): 145–150. doi:10.1007/s11023-008-9130-6.
  6. ^ Sundman, John. "Artificial stupidity". Salon. Retrieved 29 May 2016.
  7. ^ Minsky, Marvin. "Annual Minsky Loebner Prize Revocation Prize 1995 Announcement". Retrieved 29 May 2016.
  8. ^ Fisher, Richard (16 May 2012). "Chatbots fail to convince despite Loebner Prize win". NewScientist. Retrieved 29 May 2016.
  9. ^ Serck, ZLinda. "Could a computer think?". BBC. Retrieved 29 May 2016.
  10. ^ Stephens, Kenneth R. "What Has the Loebner Contest Told Us About Conversant Systems?" (PDF). www.behavior.org. Operant WebSites, Inc. Retrieved 29 May 2016.
  11. ^ Floridi, Luciano. "Humans have nothing to fear from intelligent machines". Financial Times. Retrieved 29 May 2016.
  12. ^ Christian, Brian. "Mind vs. Machine". No. March 2011. The Altantic. Retrieved 29 May 2016.
  13. ^ Loebner Prize 2006 Information
  14. ^ Lobner 2006
  15. ^ How To Be Human, Technology Review, 20 September 2006
  16. ^ Loebner prize 2006, loebner.net
  17. ^ 17th Annual Loebner Prize for Artificial Intelligence, loebner.net
  18. ^ 18th Annual Loebner Prize for Artificial Intelligence 12 October 2008 University of Reading, Reading, UK
  19. ^ Elbot
  20. ^ Artificial Solutions
  21. ^ Eugene Goostman
  22. ^ Ultra Hal
  23. ^ Machine takes on man at mass Turing Test
  24. ^ parallel-paired Turing tests
  25. ^ "2012 Loebner Prize Webcast". Retrieved 15 May 2012.
  26. ^ "Chatbot Rose wins 2015's Loebner artificial intelligence prize". BBC. Retrieved 29 May 2016.
  27. ^ Winners of Previous Contests (section), Loebner Prize's official page
  28. ^ "Read About the Loebner Award Winning Rosette - A Chatbot By Bruce Wilcox". Retrieved 29 October 2011.
  29. ^ "Chip Vivant - by Mohan Embar".
  30. ^ "Mitsuku Chatbot".
Retrieved from "https://en.wikipedia.org/w/index.php?title=Loebner_Prize&oldid=753035766"