CommonsenseQA 2.0: Exposing the Limits of AI through Gamification

Evaluating models for natural language understanding is a hard task. While state-of-the-art models achieve human parity, they still struggle with out-of-domain or adversarial examples.

A recent paper published on arXiv.org proposes gamification as a framework for data creation that can tap into human intelligence more effectively.

In the suggested game, the player is asked to author a yes/no question, is then shown the answer from the AI, and then marks whether the AI was correct or not. The player earns points for authoring questions where the AI is incorrect. Also, points are attributed for using particular phrases in the question. That enables the control of the distribution of questions towards topics. The collected data were used to evaluate models.

The evaluation showed that humans substantially outperform current models.

Constructing benchmarks that test the abilities of modern natural language understanding models is difficult – pre-trained language models exploit artifacts in benchmarks to achieve human parity, but still fail on adversarial examples and make errors that demonstrate a lack of common sense. In this work, we propose gamification as a framework for data construction. The goal of players in the game is to compose questions that mislead a rival AI while using specific phrases for extra points. The game environment leads to enhanced user engagement and simultaneously gives the game designer control over the collected data, allowing us to collect high-quality data at scale. Using our method we create CommonsenseQA 2.0, which includes 14,343 yes/no questions, and demonstrate its difficulty for models that are orders-of-magnitude larger than the AI used in the game itself. Our best baseline, the T5-based Unicorn with 11B parameters achieves an accuracy of 70.2%, substantially higher than GPT-3 (52.9%) in a few-shot inference setup. Both score well below human performance which is at 94.1%.

Research paper: Talmor, A., “CommonsenseQA 2.0: Exposing the Limits of AI through Gamification”, 2022. Link: https://arxiv.org/abs/2201.05320

You can offer your link to a page which is relevant to the topic of this post.