The Winograd schema challenge (WSC) is a test of machine intelligence proposed in 2012 by Hector Levesque, a computer scientist at the University of Toronto.
[1] On the surface, Winograd schema questions simply require the resolution of anaphora: the machine must identify the antecedent of an ambiguous pronoun in a statement.
But the exact nature of the test Turing proposed has come under scrutiny, especially since an AI chatbot named Eugene Goostman claimed to pass it in 2014.
[5] Turing's original proposal was what he called the imitation game, which involves free-flowing, unrestricted conversations in English between human judges and computer programs over a text-only channel (such as teletype).
Levesque identifies several major issues,[2] summarized as follows:[7] The key factor in the WSC is the special format of its questions, which are derived from Winograd schemas.
Levesque[2] argues that knowledge plays a central role in these problems: the answer to this schema has to do with our understanding of the typical relationships between and behavior of councilmen and demonstrators.
In 2016 and 2018, Nuance Communications sponsored a competition, offering a grand prize of $25,000 for the top scorer above 90% (for comparison, humans correctly answer to 92–96% of WSC questions[10]).
The organizing committee included Leora Morgenstern (Leidos), Theodore Patkos (The Foundation for Research & Technology Hellas), and Robert Sloan (University of Illinois at Chicago).
[16] In 2017, a neural association model designed for commonsense knowledge acquisition achieved 70% accuracy on 70 manually selected problems from the original 273 Winograd schema dataset.
In 2019 a score of 90.1%, was achieved on the original Winograd schema dataset by fine-tuning of the BERT language model with appropriate WSC-like training data to avoid having to learn commonsense reasoning.