Dataset Card for GEM/ART

Link to Main Data Card

You can find the main data card on the GEM Website .

Dataset Summary

Abductive reasoning is inference to the most plausible explanation. For example, if Jenny finds her house in a mess when she returns from work, and remembers that she left a window open, she can hypothesize that a thief broke into her house and caused the mess, as the most plausible explanation. This data loader focuses on abductive NLG: a conditional English generation task for explaining given observations in natural language.

You can load the dataset via:

import datasets
data = datasets.load_dataset('GEM/ART')

The data loader can be found here .

website

Website

paper

OpenReview

authors

Chandra Bhagavatula (AI2), Ronan Le Bras (AI2), Chaitanya Malaviya (AI2), Keisuke Sakaguchi (AI2), Ari Holtzman (AI2, UW), Hannah Rashkin (AI2, UW), Doug Downey (AI2), Wen-tau Yih (AI2), Yejin Choi (AI2, UW)

Dataset Overview

Where to find the Data and its Documentation

Webpage

Website

Download

Google Storage

Paper

OpenReview

BibTex

@inproceedings{
Bhagavatula2020Abductive,
title={Abductive Commonsense Reasoning},
author={Chandra Bhagavatula and Ronan Le Bras and Chaitanya Malaviya and Keisuke Sakaguchi and Ari Holtzman and Hannah Rashkin and Doug Downey and Wen-tau Yih and Yejin Choi},
booktitle={International Conference on Learning Representations},
year={2020},
url={https://openreview.net/forum?id=Byg1v1HKDB}
}

Contact Name

Chandra Bhagavatulla

Contact Email

chandrab@allenai.org

Has a Leaderboard?

no

Languages and Intended Use

Multilingual?

no

Covered Languages

English

Whose Language?

Crowdworkers on the Amazon Mechanical Turk platform based in the U.S, Canada, U.K and Australia.

License

apache-2.0: Apache License 2.0

Intended Use

To study the viability of language-based abductive reasoning. Training and evaluating models to generate a plausible hypothesis to explain two given observations.

Primary Task

Reasoning

Credit

Curation Organization Type(s)

industry

Curation Organization(s)

Allen Institute for AI

Dataset Creators

Chandra Bhagavatula (AI2), Ronan Le Bras (AI2), Chaitanya Malaviya (AI2), Keisuke Sakaguchi (AI2), Ari Holtzman (AI2, UW), Hannah Rashkin (AI2, UW), Doug Downey (AI2), Wen-tau Yih (AI2), Yejin Choi (AI2, UW)

Funding

Allen Institute for AI

Who added the Dataset to GEM?

Chandra Bhagavatula (AI2), Ronan LeBras (AI2), Aman Madaan (CMU), Nico Daheim (RWTH Aachen University)

Dataset Structure

Data Fields

• observation_1 : A string describing an observation / event.
• observation_2 : A string describing an observation / event.
• label : A string that plausibly explains why observation_1 and observation_2 might have happened.

How were labels chosen?

Explanations were authored by crowdworkers on the Amazon Mechanical Turk platform using a custom template designed by the creators of the dataset.

Example Instance

{
'gem_id': 'GEM-ART-validation-0',
'observation_1': 'Stephen was at a party.',
'observation_2': 'He checked it but it was completely broken.',
'label': 'Stephen knocked over a vase while drunk.'
}

Data Splits

• train : Consists of training instances.
• dev : Consists of dev instances.
• test : Consists of test instances.

Dataset in GEM

Rationale for Inclusion in GEM

Why is the Dataset in GEM?

Abductive reasoning is a crucial capability of humans and ART is the first dataset curated to study language-based abductive reasoning.

Similar Datasets

no

Ability that the Dataset measures

Whether models can reason abductively about a given pair of observations.

GEM-Specific Curation

Modificatied for GEM?

no

Additional Splits?

no

Getting Started with the Task

Pointers to Resources

• Paper
• Code

Previous Results

Previous Results

Measured Model Abilities

Whether models can reason abductively about a given pair of observations.

Metrics

BLEU , BERT-Score , ROUGE

Previous results available?

no

Dataset Curation

Original Curation

Sourced from Different Sources

no

Language Data

How was Language Data Obtained?

Crowdsourced

Where was it crowdsourced?

Amazon Mechanical Turk

Language Producers

Language producers were English speakers in U.S., Canada, U.K and Australia.

Topics Covered

No

Data Validation

validated by crowdworker

Was Data Filtered?

algorithmically

Filter Criteria

Adversarial filtering algorithm as described in the paper

Structured Annotations

Additional Annotations?

automatically created

Annotation Service?

no

Annotation Values

Each observation is associated with a list of COMET ( https://arxiv.org/abs/1906.05317 ) inferences.

Any Quality Control?

none

Consent

Any Consent Policy?

no

Private Identifying Information (PII)

Contains PII?

no PII

Justification for no PII

The dataset contains day-to-day events. It does not contain names, emails, addresses etc.

Maintenance

Any Maintenance Plan?

no

Broader Social Context

Previous Work on the Social Impact of the Dataset

Usage of Models based on the Data

no

Impact on Under-Served Communities

Addresses needs of underserved Communities?

no

Discussion of Biases

Any Documented Social Biases?

no

Considerations for Using the Data

PII Risks and Liability

Potential PII Risk

None

Licenses

Copyright Restrictions on the Dataset

public domain

Copyright Restrictions on the Language Data

public domain

Known Technical Limitations