Dataset Card for GEM/wiki_auto_asset_turk

Link to Main Data Card

You can find the main data card on the GEM Website .

Dataset Summary

WikiAuto is an English simplification dataset that we paired with ASSET and TURK, two very high-quality evaluation datasets, as test sets. The input is an English sentence taken from Wikipedia and the target a simplified sentence. ASSET and TURK contain the same test examples but have references that are simplified in different ways (splitting sentences vs. rewriting and splitting).

You can load the dataset via:

import datasets
data = datasets.load_dataset('GEM/wiki_auto_asset_turk')

The data loader can be found here .

website

n/a

paper

WikiAuto , ASSET , TURK

authors

WikiAuto: Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, Wei Xu; ASSET: Fernando Alva-Manchego, Louis Martin, Antoine Bordes, Carolina Scarton, and Benoîıt Sagot, and Lucia Specia; TURK: Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch

Dataset Overview

Where to find the Data and its Documentation

Download

Wiki-Auto repository , ASSET repository , TURKCorpus

Paper

WikiAuto , ASSET , TURK

BibTex

WikiAuto:

@inproceedings{jiang-etal-2020-neural,
    title = "Neural {CRF} Model for Sentence Alignment in Text Simplification",
    author = "Jiang, Chao  and
      Maddela, Mounica  and
      Lan, Wuwei  and
      Zhong, Yang  and
      Xu, Wei",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.acl-main.709",
    doi = "10.18653/v1/2020.acl-main.709",
    pages = "7943--7960",
}

ASSET:

@inproceedings{alva-manchego-etal-2020-asset,
    title = "{ASSET}: {A} Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations",
    author = "Alva-Manchego, Fernando  and
      Martin, Louis  and
      Bordes, Antoine  and
      Scarton, Carolina  and
      Sagot, Beno{\^\i}t  and
      Specia, Lucia",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.acl-main.424",
    pages = "4668--4679",
}

TURK:

@article{Xu-EtAl:2016:TACL,
 author = {Wei Xu and Courtney Napoles and Ellie Pavlick and Quanze Chen and Chris Callison-Burch},
 title = {Optimizing Statistical Machine Translation for Text Simplification},
 journal = {Transactions of the Association for Computational Linguistics},
 volume = {4},
 year = {2016},
 url = {https://cocoxu.github.io/publications/tacl2016-smt-simplification.pdf},
 pages = {401--415}
 }

Contact Name

WikiAuto: Chao Jiang; ASSET: Fernando Alva-Manchego and Louis Martin; TURK: Wei Xu

Contact Email

jiang.1530@osu.edu , f.alva@sheffield.ac.uk , louismartincs@gmail.com , wei.xu@cc.gatech.edu

Has a Leaderboard?

no

Languages and Intended Use

Multilingual?

no

Covered Languages

English

Whose Language?

Wiki-Auto contains English text only (BCP-47: en ). It is presented as a translation task where Wikipedia Simple English is treated as its own idiom. For a statement of what is intended (but not always observed) to constitute Simple English on this platform, see Simple English in Wikipedia . Both ASSET and TURK use crowdsourcing to change references, and their language is thus a combination of the WikiAuto data and the language of the demographic on mechanical Turk

License

other: Other license

Intended Use

WikiAuto provides a set of aligned sentences from English Wikipedia and Simple English Wikipedia as a resource to train sentence simplification systems.

The authors first crowd-sourced a set of manual alignments between sentences in a subset of the Simple English Wikipedia and their corresponding versions in English Wikipedia (this corresponds to the manual config in this version of the dataset), then trained a neural CRF system to predict these alignments.

The trained alignment prediction model was then applied to the other articles in Simple English Wikipedia with an English counterpart to create a larger corpus of aligned sentences (corresponding to the auto and auto_acl configs here).

ASSET (Alva-Manchego et al., 2020) is multi-reference dataset for the evaluation of sentence simplification in English. The dataset uses the same 2,359 sentences from TurkCorpus (Xu et al., 2016) and each sentence is associated with 10 crowdsourced simplifications. Unlike previous simplification datasets, which contain a single transformation (e.g., lexical paraphrasing in TurkCorpus or sentence splitting in HSplit ), the simplifications in ASSET encompass a variety of rewriting transformations.

TURKCorpus is a high quality simplification dataset where each source (not simple) sentence is associated with 8 human-written simplifications that focus on lexical paraphrasing. It is one of the two evaluation datasets for the text simplification task in GEM. It acts as the validation and test set for paraphrasing-based simplification that does not involve sentence splitting and deletion.

Add. License Info

WikiAuto: CC BY-NC 3.0 , ASSET: CC BY-NC 4.0 , TURK: GNU General Public License v3.0

Primary Task

Simplification

Communicative Goal

The goal is to communicate the main ideas of source sentence in a way that is easier to understand by non-native speakers of English.

Credit

Curation Organization Type(s)

academic , industry

Curation Organization(s)

Ohio State University, University of Sheffield, Inria, Facebook AI Research, Imperial College London, University of Pennsylvania, John Hopkins University

Dataset Creators

WikiAuto: Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, Wei Xu; ASSET: Fernando Alva-Manchego, Louis Martin, Antoine Bordes, Carolina Scarton, and Benoîıt Sagot, and Lucia Specia; TURK: Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch

Funding

WikiAuto: NSF, ODNI, IARPA, Figure Eight AI, and Criteo. ASSET: PRAIRIE Institute, ANR. TURK: NSF

Who added the Dataset to GEM?

GEM v1 had separate data cards for WikiAuto, ASSET, and TURK. They were contributed by Dhruv Kumar and Mounica Maddela. The initial data loader was written by Yacine Jernite. Sebastian Gehrmann merged and extended the data cards and migrated the loader to the v2 infrastructure.

Dataset Structure

Data Fields

• source : A source sentence from one of the datasets
• target : A single simplified sentence corresponding to source
• references : In the case of ASSET/TURK, references is a list of strings corresponding to the different references.

Reason for Structure

The underlying datasets have extensive secondary annotations that can be used in conjunction with the GEM version. We omit those annotations to simplify the format into one that can be used by seq2seq models.

Example Instance

{
  'source': 'In early work, Rutherford discovered the concept of radioactive half-life , the radioactive element radon, and differentiated and named alpha and beta radiation .',
 'target': 'Rutherford discovered the radioactive half-life, and the three parts of radiation which he named Alpha, Beta, and Gamma.'
}

Data Splits

In WikiAuto, which is used as training and validation set, the following splits are provided:

Tain	Dev	Test
Total sentence pairs	373801	73249	118074
Aligned sentence pairs	1889	346	677

ASSET does not contain a training set; many models use WikiLarge (Zhang and Lapata, 2017) for training. For GEM, Wiki-Auto will be used for training the model.

Each input sentence has 10 associated reference simplified sentences. The statistics of ASSET are given below.

Dev	Test	Total
Input Sentences	2000	359	2359
Reference Simplifications	20000	3590	23590

The test and validation sets are the same as those of TurkCorpus . The split was random.

There are 19.04 tokens per reference on average (lower than 21.29 and 25.49 for TurkCorpus and HSplit, respectively). Most (17,245) of the referece sentences do not involve sentence splitting.

TURKCorpus does not contain a training set; many models use WikiLarge (Zhang and Lapata, 2017) or Wiki-Auto (Jiang et. al 2020) for training.

Each input sentence has 8 associated reference simplified sentences. 2,359 input sentences are randomly split into 2,000 validation and 359 test sentences.

Dev	Test	Total
Input Sentences	2000	359	2359
Reference Simplifications	16000	2872	18872

There are 21.29 tokens per reference on average.

Splitting Criteria

In our setup, we use WikiAuto as training/validation corpus and ASSET and TURK as test corpora. ASSET and TURK have the same inputs but differ in their reference style. Researchers can thus conduct targeted evaluations based on the strategies that a model should learn.

Dataset in GEM

Rationale for Inclusion in GEM

Why is the Dataset in GEM?

WikiAuto is the largest open text simplification dataset currently available. ASSET and TURK are high quality test sets that are compatible with WikiAuto.

Similar Datasets

yes

Unique Language Coverage

no

Difference from other GEM datasets

It's unique setup with multiple test sets makes the task interesting since it allows for evaluation of multiple generations and systems that simplify in different ways.

Ability that the Dataset measures

simplification

GEM-Specific Curation

Modificatied for GEM?

yes

GEM Modifications

other

Modification Details

We removed secondary annotations and focus on the simple input->output format, but combine the different sub-datasets.

Additional Splits?

yes

Split Information

we split the original test set according to syntactic complexity of the source sentences. To characterize sentence syntactic complexity, we use the 8-level developmental level (d-level) scale proposed by Covington et al. (2006) and the implementation of Lu, Xiaofei (2010) . We thus split the original test set into 8 subsets corresponding to the 8 d-levels assigned to source sentences. We obtain the following number of instances per level and average d-level of the dataset:

Total nb. sentences	L0	L1	L2	L3	L4	L5	L6	L7	Mean Level
359	166	0	58	32	5	28	7	63	2.38

Split Motivation

The goal was to assess performance when simplifying source sentences with different syntactic structure and complexity.

Getting Started with the Task

Pointers to Resources

There are recent supervised ( Martin et al., 2019 , Kriz et al., 2019 , Dong et al., 2019 , Zhang and Lapata, 2017 ) and unsupervised ( Martin et al., 2020 , Kumar et al., 2020 , Surya et al., 2019 ) text simplification models that can be used as baselines.

Technical Terms

The common metric used for automatic evaluation is SARI (Xu et al., 2016) .

Previous Results

Previous Results

Measured Model Abilities

Simplification

Metrics

Other: Other Metrics , BLEU

Other Metrics

SARI: A simplification metric that considers both input and references to measure the "goodness" of words that are added, deleted, and kept.

Proposed Evaluation

The original authors of WikiAuto and ASSET used human evaluation to assess the fluency, adequacy, and simplicity (details provided in the paper). For TURK, the authors measured grammaticality, meaning-preservation, and simplicity gain (details in the paper).

Previous results available?

no

Dataset Curation

Original Curation

Original Curation Rationale

Wiki-Auto provides a new version of the Wikipedia corpus that is larger, contains 75% less defective pairs and has more complex rewrites than the previous WIKILARGE dataset.

ASSET was created in order to improve the evaluation of sentence simplification. It uses the same input sentences as the TurkCorpus dataset from (Xu et al., 2016) . The 2,359 input sentences of TurkCorpus are a sample of "standard" (not simple) sentences from the Parallel Wikipedia Simplification (PWKP) dataset (Zhu et al., 2010) , which come from the August 22, 2009 version of Wikipedia. The sentences of TurkCorpus were chosen to be of similar length (Xu et al., 2016) . No further information is provided on the sampling strategy.

The TurkCorpus dataset was developed in order to overcome some of the problems with sentence pairs from Standard and Simple Wikipedia: a large fraction of sentences were misaligned, or not actually simpler (Xu et al., 2016) . However, TurkCorpus mainly focused on lexical paraphrasing , and so cannot be used to evaluate simplifications involving compression (deletion) or sentence splitting . HSplit (Sulem et al., 2018) , on the other hand, can only be used to evaluate sentence splitting. The reference sentences in ASSET include a wider variety of sentence rewriting strategies, combining splitting, compression and paraphrasing. Annotators were given examples of each kind of transformation individually, as well as all three transformations used at once, but were allowed to decide which transformations to use for any given sentence.

An example illustrating the differences between TurkCorpus, HSplit and ASSET is given below:

Original: He settled in London, devoting himself chiefly to practical teaching.

TurkCorpus: He rooted in London, devoting himself mainly to practical teaching.

HSplit: He settled in London. He devoted himself chiefly to practical teaching.

ASSET: He lived in London. He was a teacher.

Communicative Goal

The goal is to communicate the same information as the source sentence using simpler words and grammar.

Sourced from Different Sources

yes

Source Details

Wikipedia

Language Data

How was Language Data Obtained?

Found

Where was it found?

Single website

Language Producers

The dataset uses language from Wikipedia: some demographic information is provided here .

Data Validation

not validated

Was Data Filtered?

algorithmically

Filter Criteria

The authors mention that they "extracted 138,095 article pairs from the 2019/09 Wikipedia dump using an improved version of the WikiExtractor library". The SpaCy library is used for sentence splitting.

Structured Annotations

Additional Annotations?

crowd-sourced

Number of Raters

11<n<50

Rater Qualifications

WikiAuto (Figure Eight): No information provided.

ASSET (MTurk):

• Having a HIT approval rate over 95%, and over 1000 HITs approved. No other demographic or compensation information is provided.
• Passing a Qualification Test (appropriately simplifying sentences). Out of 100 workers, 42 passed the test.
• Being a resident of the United States, United Kingdom or Canada.

TURK (MTurk):

• Reference sentences were written by workers with HIT approval rate over 95%. No other demographic or compensation information is provided.

Raters per Training Example

1

Raters per Test Example

5

Annotation Service?

yes

Which Annotation Service

Amazon Mechanical Turk , Appen

Annotation Values

WikiAuto: Sentence alignment labels were crowdsourced for 500 randomly sampled document pairs (10,123 sentence pairs total). The authors pre-selected several alignment candidates from English Wikipedia for each Simple Wikipedia sentence based on various similarity metrics, then asked the crowd-workers to annotate these pairs. Finally, they trained their alignment model on this manually annotated dataset to obtain automatically aligned sentences (138,095 document pairs, 488,332 sentence pairs). No demographic annotation is provided for the crowd workers. The Figure Eight platform now part of Appen) was used for the annotation process.

ASSET: The instructions given to the annotators are available here .

TURK: The references are crowdsourced from Amazon Mechanical Turk. The annotators were asked to provide simplifications without losing any information or splitting the input sentence. No other demographic or compensation information is provided in the TURKCorpus paper. The instructions given to the annotators are available in the paper.

Any Quality Control?

none

Consent

Any Consent Policy?

yes

Consent Policy Details

Both Figure Eight and Amazon Mechanical Turk raters forfeit the right to their data as part of their agreements.

Private Identifying Information (PII)

Contains PII?

no PII

Justification for no PII

Since the dataset is created from Wikipedia/Simple Wikipedia, all the information contained in the dataset is already in the public domain.

Maintenance

Any Maintenance Plan?

no

Broader Social Context

Previous Work on the Social Impact of the Dataset

Usage of Models based on the Data

no

Impact on Under-Served Communities

Addresses needs of underserved Communities?

no

Discussion of Biases

Any Documented Social Biases?

yes

Links and Summaries of Analysis Work

The dataset may contain some social biases, as the input sentences are based on Wikipedia. Studies have shown that the English Wikipedia contains both gender biases (Schmahl et al., 2020) and racial biases (Adams et al., 2019) .

Considerations for Using the Data

PII Risks and Liability

Potential PII Risk

All the data is in the public domain.

Licenses

Copyright Restrictions on the Dataset

open license - commercial use allowed

Copyright Restrictions on the Language Data

open license - commercial use allowed

Known Technical Limitations

Technical Limitations

The dataset may contain some social biases, as the input sentences are based on Wikipedia. Studies have shown that the English Wikipedia contains both gender biases (Schmahl et al., 2020) and racial biases (Adams et al., 2019) .

Unsuited Applications

Since the test datasets contains only 2,359 sentences that are derived from Wikipedia, they are limited to a small subset of topics present on Wikipedia.