Dataset Card for GEM/surface_realisation_st_2020

Link to Main Data Card

You can find the main data card on the GEM Website .

Dataset Summary

This dataset was used as part of the multilingual surface realization shared task in which a model gets full or partial universal dependency structures and has to reconstruct the natural language. This dataset support 11 languages.

You can load the dataset via:

import datasets
data = datasets.load_dataset('GEM/surface_realisation_st_2020')

The data loader can be found here .

website

Website

paper

ACL Anthology

authors

Simon Mille (Pompeu Fabra University); Leo Wanner (Pompeu Fabra University); Anya Belz (Brighton University); Bernd Bohnet (Google Inc.); Thiago Castro Ferreira (Federal University of Minas Gerais); Yvette Graham (ADAPT/Trinity College Dublin)

Dataset Overview

Where to find the Data and its Documentation

Webpage

Website

Download

Website

Paper

ACL Anthology

BibTex

@inproceedings{mille-etal-2020-third,
    title = "The Third Multilingual Surface Realisation Shared Task ({SR}{'}20): Overview and Evaluation Results",
    author = "Mille, Simon  and
      Belz, Anya  and
      Bohnet, Bernd  and
      Castro Ferreira, Thiago  and
      Graham, Yvette  and
      Wanner, Leo",
    booktitle = "Proceedings of the Third Workshop on Multilingual Surface Realisation",
    month = dec,
    year = "2020",
    address = "Barcelona, Spain (Online)",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2020.msr-1.1",
    pages = "1--20",
    abstract = "This paper presents results from the Third Shared Task on Multilingual Surface Realisation (SR{'}20) which was organised as part of the COLING{'}20 Workshop on Multilingual Surface Realisation. As in SR{'}18 and SR{'}19, the shared task comprised two tracks: (1) a Shallow Track where the inputs were full UD structures with word order information removed and tokens lemmatised; and (2) a Deep Track where additionally, functional words and morphological information were removed. Moreover, each track had two subtracks: (a) restricted-resource, where only the data provided or approved as part of a track could be used for training models, and (b) open-resource, where any data could be used. The Shallow Track was offered in 11 languages, whereas the Deep Track in 3 ones. Systems were evaluated using both automatic metrics and direct assessment by human evaluators in terms of Readability and Meaning Similarity to reference outputs. We present the evaluation results, along with descriptions of the SR{'}19 tracks, data and evaluation methods, as well as brief summaries of the participating systems. For full descriptions of the participating systems, please see the separate system reports elsewhere in this volume.",
}

Contact Name

Simon Mille

Contact Email

sfmille@gmail.com

Has a Leaderboard?

no

Languages and Intended Use

Multilingual?

yes

Covered Dialects

No multiple dialects.

Covered Languages

Arabic , Chinese , English , French , Hindi , Indonesian , Japanese , Korean , Portuguese , Russian , Spanish, Castilian

Whose Language?

Unknown

License

cc-by-2.5: Creative Commons Attribution 2.5 Generic

Intended Use

The dataset is intended to be used for training models to solve several NLG subtasks, such as function word introduction, morphological agreement resolution, word order determination and inflection generation.

Comment about the license: the dataset has multiple licences, since each original dataset has their own type of licence. All datasets but one are CC-BY and subclasses of it, the other one is GPL (French Sequoia).

Primary Task

Data-to-Text

Communicative Goal

The models are able to introduce surface features (syntax, morphology, topology) from more or less abstract inputs in different, the most abstract being predicate-argument structures. The datasets cover a large variety of domains (news, blogs, forums, wikipedia pages, etc.).

Credit

Curation Organization Type(s)

industry , academic

Curation Organization(s)

Pompeu Fabra University, Google Inc., University of Brighton, Federal University of Minas Gerais, ADAPT/Trinity College Dublin

Dataset Creators

Simon Mille (Pompeu Fabra University); Leo Wanner (Pompeu Fabra University); Anya Belz (Brighton University); Bernd Bohnet (Google Inc.); Thiago Castro Ferreira (Federal University of Minas Gerais); Yvette Graham (ADAPT/Trinity College Dublin)

Funding

Mostly EU funds via H2020 projects

Who added the Dataset to GEM?

Simon Mille (Pompeu Fabra University)

Dataset Structure

Data Fields

input (string): this field contains an input tree in CoNLL-U format; the CoNLL-U format is a one-word-per-line format with the following tab-separated 10 columns (see here ): [1] Position, [2] Lemma, [3] Wordform, [4] Part of Speech, [5] Fine-grained Part of Speech (if available), [6] Features (FEATS), [7] governor, [8] dependency relation, [9] additional dependency information, and [10] metadata. For the surface task, the input is a Universal Dependency tree of a given language in which the word order was scrambled and the surface forms removed (only lemmas are available); for the deep task, the input is a tree derived from the surface input, with predicate-argument relations between content words only (function words were removed) and without any morphological agreement information.

target_tokenized (string): this field contains the target sentence to generate, in which every non-initial and non-final token is surrounded by two spaces. This output is usually used for automatic evaluations.

target (string): this field contains the detokenised target sentence to generate. This output is usually used for human evaluations.

gem_id (string): a unique ID.

sentence_id (string): the original ID of a sentence in the UD dataset.

Reason for Structure

The structure of the input (CoNLL-U) was chosen according to the standards in parsing, and because the original UD datasets were provided in this format.

How were labels chosen?

The input labels for the surface track are the original labels in the UD treebanks; see here for the dependencies, here for the features, and here for the PoS tags.

The input labels for the deep track are a subset of the PoS tags and features of the surface track, and for the relations, universal predicate-argument relations augmented with a few specific relations to capture coordinations and named entity relations for instance.

Example Instance

{"input": "1\tGoogle\t_\tPROPN\tNNP\tNumber=Sing\t5\tnsubj\t_\t_\n2\t\t_\tPUNCT\t.\tlin=+1\t5\tpunct\t_\t_\n3\tinto\t_\tADP\tIN\t_\t6\tcase\t_\t_\n4\tif\t_\tSCONJ\tIN\t_\t5\tmark\t_\t_\n5\tmorph\t_\tVERB\tVBD\tMood=Ind|Tense=Past|VerbForm=Fin\t7\tadvcl\t_\t_\n6\tGoogleOS\t_\tPROPN\tNNP\tNumber=Sing\t5\tobl\t_\t_\n7\twhat\t_\tPRON\tWP\tPronType=Int\t0\troot\t_\t_", "target_tokenized": "What if Google Morphed Into GoogleOS ?", "target": "What if Google Morphed Into GoogleOS?", "gem_id": "GEM-surface_realisation_st_2020-T1-test-en_ewt-ud-test-0", "sentence_id": ""}

Data Splits

There are 119 splits in the dataset:

• 29 training sets, which correspond to 20 UD datasets (11 languages), 9 of which have both surface and deep inputs (3 languages);
• 29 development set which correspond to the 29 training sets above;
• 29 test sets for the data described above;
• 4 out-of-domain test sets, 3 surface inputs and 1 deep one (3 languages for which PUD out-of-domain datasets were available);
• 9 automatically parsed in-domain test sets, 6 surface inputs and 3 deep inputs (6 languages for which good UD parsers were available);
• 9 automatically parsed out-of-domain test sets, 6 surface inputs and 3 deep inputs (6 languages for which we were able to create clean Wikipedia text and that had a good UD parser).

Splitting Criteria

Described above for more clarity.

An outlier would usually be an input that corresponds to a very long sentence (e.g. 159 words in English, when the average number of words per sentence is around 25).

Dataset in GEM

Rationale for Inclusion in GEM

Why is the Dataset in GEM?

The datset includes languages from different families and some languages not often used in NLG (e.g. Arabic, Indonesian, Korean, Hindi). It proposes two tasks, which can be tackled both separately and in one shot, with different levels of difficulty: the most superficial task (T1) consits in ordering and inflecting some trees, and the deeper task (T2) includes extra tasks such as defining the syntactic structure and introducing function words and morphological agreement information. Both tasks can allow for developing modules for pipeline NLG architectures. T1 is rather straightforward to evaluate: BLEU works quite well for some languages since all the words are present in the input and few word orders only can be possible for a syntactic tree. But T2 is more challenging to evaluate, since more outputs are correct given one particular input.

There is a large variety of sizes in the datasets, both clean and noisy data, parallel data in different languages, and many already available system outputs to use as baselines.

Similar Datasets

yes

Unique Language Coverage

yes

Difference from other GEM datasets

This is possibly the only dataset that starts the generation process from predicate-argument structures and from syntactic structures. It also has parallel datasets in a few languages (coming from the PUD parallel annotations).

Ability that the Dataset measures

Syntacticisation, functional word introduction, word order resolution, agreement resolution, morphological inflection

GEM-Specific Curation

Modificatied for GEM?

no

Additional Splits?

no

Getting Started with the Task

Pointers to Resources

Website

Technical Terms

Syntacticisation: prediction of the syntactic

Previous Results

Previous Results

Measured Model Abilities

Syntacticisation, functional word introduction, word order resolution, morphological agreement resolution, morphological inflection

Metrics

BLEU , BERT-Score , Other: Other Metrics

Other Metrics

NIST: n-gram similarity metric weighted in favour of less frequent n-grams which are taken to be more informative.

Normalised edit distance (DIST): inverse, normalised, character-based string-edit distance that starts by computing the minimum number of character inserts, deletes and substitutions (all at cost 1) required to turn the system output into the (single) reference text.

Proposed Evaluation

BLEU, NIST, BERTScore and DIST simply aim at calculating in different ways the similarity between a predicted and a reference sentence.

Two additional criteria have been used for human evaluation, Readability and Meaning SImilarity. The statement to be assessed in the Readability evaluation was: "The text reads well and is free from grammatical errors and awkward constructions.". The corresponding statement in the Meaning Similarity evaluation, in which system outputs (‘the black text’) were compared to reference sentences (‘the gray text’), was: "The meaning of the gray text is adequately expressed by the black text."

Previous results available?

yes

Other Evaluation Approaches

Same as above.

Relevant Previous Results

• Fast and Accurate Non-Projective Dependency Tree Linearization
• Shape of Synth to Come: Why We Should Use Synthetic Data for English Surface Realization

Dataset Curation

Original Curation

Original Curation Rationale

The datasets were created in the context of the Surface Realisation Shared Task series.

Communicative Goal

The dataset's objective was to allow for training systems to perform tasks related to surface realisation (introduction of function words, syntacticisation, resolution of morphological agreements, word order resolution, inflection generation.

Sourced from Different Sources

yes

Source Details

Each of the 20 used UD datasets comes from various sources, all listed on the individual page of each UD treeebank ( https://universaldependencies.org/ ).

Additional test sets were created for the task, and were obtained from Wikipedia pages for 6 languages.

Language Data

How was Language Data Obtained?

Found

Where was it found?

Multiple websites

Language Producers

There are numerous sources of language in the multiple datasets.

Topics Covered

There is a large variety of topics in the multiple datasets.

Data Validation

not validated

Data Preprocessing

The text data was detokenised so as to create references for automatic evaluations (several languages don't use spaces to separate words, and running metrics like BLEU would not make sense without separating all the tokens in a sentence).

Was Data Filtered?

hybrid

Filter Criteria

For the Wikipedia test created for the shared task, extensive filtering was applied to achieve reasonably good text quality. Sentences that include special characters, contain unusual tokens (e.g. ISBN), or have unbalanced quotation marks or brackets were skipped. Furthermore, only sentences with more than 5 tokens and shorter than 50 tokens were selected. After the initial filtering, quite a few malformed sentences remained. In order to remove those, the sentences were scored with BERT and only the top half scored sentences were kept. Finally, via manual inspection, patterns and expressions were identified to further reduce the number of malformed sentences.

Structured Annotations

Additional Annotations?

none

Annotation Service?

no

Consent

Any Consent Policy?

no

Justification for Using the Data

The Universal Dependency data had been previously used for shared tasks on parsing, so it made sense to reuse it for generation.

Private Identifying Information (PII)

Contains PII?

unlikely

Any PII Identification?

no identification

Maintenance

Any Maintenance Plan?

no

Broader Social Context

Previous Work on the Social Impact of the Dataset

Usage of Models based on the Data

no

Impact on Under-Served Communities

Addresses needs of underserved Communities?

yes

Details on how Dataset Addresses the Needs

Thanks to the original work of the UD dataset creators, the surface realisation dataset addresses a few languages which are possibly under-served in NLG: e.g. Arabic, Hindi, Indonesian, Korean.

Discussion of Biases

Any Documented Social Biases?

no

Are the Language Producers Representative of the Language?

It is very likely that the distribution of language producers is not fully represented in the datasets of each language.

Considerations for Using the Data

PII Risks and Liability

Potential PII Risk

No risks foreseen.

Licenses

Copyright Restrictions on the Dataset

multiple licenses , open license - commercial use allowed

Copyright Restrictions on the Language Data

multiple licenses , open license - commercial use allowed

Known Technical Limitations

Technical Limitations

The deep track inputs (predicate-argument structures) are not of perfect quality, they were derived automatically from gold or predicted syntactic parses using handcrafted grammars.

Unsuited Applications

The datasets are probably not fitted to train tools to produce "unusual" languages (e.g. poetry, kid writing etc.).

Discouraged Use Cases

To be thought of :)