LAUSR

amples) for which the best system’s accuracy in 2019 was 79.1%, considerably below human level. This effect, to prevent AI systems achieving human performance levels in the absence of genuine reasoning ability, was a desired goal. What is the long-term impact? A year later, the Choi team’s UNICORN can solve WINOGRANDE problems with an almost human-level 91.28% accuracy, as indicated by the WINOGRANDE leaderboard. AI systems will likely soon solve WINOGRANDE at human level— without necessarily having made real progress on the underlying task of CSR. Arguably, this indicates that solving either the WSC or WINOGRANDE does not indicate CSR ability. The contributions of WINOGRANDE, however, go far beyond performance on specific datasets. Importantly, the methodologies introduced in the paper are independent of the WINOGRANDE dataset. Methods used to help MTs generate large-scale corpora can be adapted to create other corpora. The filtering algorithm introduced here can be modified to filter bias and other sources of error more aggressively. These techniques will remain useful, whether AI systems prematurely achieve human-level performance on any of the multiple corpora that researchers currently target.