Dataset Card for GEM/turku_paraphrase_corpus

Link to Main Data Card

You can find the main data card on the GEM Website .

Dataset Summary

This is a Finnish paraphrase corpus which consists of pairs of text passages, where a typical passage is about a sentence long. It can be used to either identify or generate paraphrases.

You can load the dataset via:

import datasets
data = datasets.load_dataset('GEM/turku_paraphrase_corpus')

The data loader can be found here .

website

Website

paper

ACL Anthology

authors

Jenna Kanerva, Filip Ginter, Li-Hsin Chang, Iiro Rastas, Valtteri Skantsi, Jemina Kilpeläinen, Hanna-Mari Kupari, Aurora Piirto, Jenna Saarni, Maija Sevón, Otto Tarkka (TurkuNLP / University of Turku)

Dataset Overview

Where to find the Data and its Documentation

Webpage

Website

Download

Github

Paper

ACL Anthology

BibTex

@inproceedings{kanerva-etal-2021-finnish,
    title = {Finnish Paraphrase Corpus},
    author = {Kanerva, Jenna and Ginter, Filip and Chang, Li-Hsin and Rastas, Iiro and Skantsi, Valtteri and Kilpel{\"a}inen, Jemina and Kupari, Hanna-Mari and Saarni, Jenna and Sev{\'o}n, Maija and Tarkka, Otto},
    booktitle = {Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa'21)},
    year = {2021},
    publisher = {Link{\"o}ping University Electronic Press, Sweden},
    url = {https://aclanthology.org/2021.nodalida-main.29},
    pages = {288--298}
}

Contact Name

Jenna Kanerva, Filip Ginter

Contact Email

jmnybl@utu.fi , figint@utu.fi

Has a Leaderboard?

no

Languages and Intended Use

Multilingual?

no

Covered Dialects

written standard language, spoken language

Covered Languages

Finnish

License

cc-by-sa-4.0: Creative Commons Attribution Share Alike 4.0 International

Intended Use

Paraphrase classification, paraphrase generation

Primary Task

Paraphrasing

Communicative Goal

The corpus provides naturally occurring Finnish paraphrases striving for low lexical overlap, thus supporting many different downstream applications requiring language understanding.

Credit

Curation Organization Type(s)

academic

Curation Organization(s)

University of Turku

Dataset Creators

Jenna Kanerva, Filip Ginter, Li-Hsin Chang, Iiro Rastas, Valtteri Skantsi, Jemina Kilpeläinen, Hanna-Mari Kupari, Aurora Piirto, Jenna Saarni, Maija Sevón, Otto Tarkka (TurkuNLP / University of Turku)

Funding

The Turku paraphrase corpus project was funded by the Academy of Finland, as well as the European Language Grid project through its open call for pilot projects. The European Language Grid project has received funding from the European Union’s Horizon 2020 Research and Innovation programme under Grant Agreement no. 825627 (ELG).

Who added the Dataset to GEM?

Jenna Kanerva, Filip Ginter (TurkuNLP / University of Turku)

Dataset Structure

Data Fields

The dataset consist of pairs of text passages, where a typical passage is about a sentence long, however, a passage may also be longer or shorter than a sentence. Thus, each example include two text passages (string), a manually annotated label to indicate the paraphrase type (string), and additional metadata.

The dataset include three different modes , plain, classification, and generation. The plain mode loads the original data without any additional preprocessing or transformations, while the classification mode directly builds the data in a form suitable for training a paraphrase classifier, where each example is doubled in the data with different directions (text1, text2, label) --> (text2, text1, label) taking care of the label flipping as well if needed (paraphrases with directionality flag < or >). In the generation mode, the examples are preprocessed to be directly suitable for paraphrase generation task. In here, paraphrases not suitable for generation are discarded (negative, and highly context-dependent paraphrases), and directional paraphrases are provided so that the generation goes from more detailed passage to the more general one in order to prevent model hallucination (i.e. model learning to introduce new information). The rest of the paraphrases are provided in both directions (text1, text2, label) --> (text2, text1, label).

Each pair in plain and classification mode will include fields:

gem_id : Identifier of the paraphrase pair (string) goeswith : Identifier of the document from which the paraphrase was extracted, can be not available in case the source of the paraphrase is not from document-structured data (string) fold : 0-99, data split into 100 parts respecting document boundaries, you can use this e.g. to implement crossvalidation safely as all paraphrases from one document are in one fold (int) text1 : First paraphrase passage (string) text2 : Second paraphrase passage (string) label : Manually annotated labels (string) binary_label : Label turned into binary with values positive (paraphrase) and negative (not-paraphrase) (string) is_rewrite : Indicator whether the example is human produced rewrite or naturally occurring paraphrase (bool)

Each pair in generation mode will include the same fields expect text1 and text2 are renamed to input and output in order to indicate the generation direction. Thus the fields are:

gem_id : Identifier of the paraphrase pair (string) goeswith : Identifier of the document from which the paraphrase was extracted, can be not available in case the source of the paraphrase is not from document-structured data (string) fold : 0-99, data split into 100 parts respecting document boundaries, you can use this e.g. to implement crossvalidation safely as all paraphrases from one document are in one fold (int) input : The input paraphrase passage for generation (string) output : The output paraphrase passage for generation (string) label : Manually annotated labels (string) binary_label : Label turned into binary with values positive (paraphrase) and negative (not-paraphrase) (string) is_rewrite : Indicator whether the example is human produced rewrite or naturally occurring paraphrase (bool)

Example Instance

{
  'gem_id':  'gem-turku_paraphrase_corpus-train-15',
  'goeswith': 'episode-02243',
  'fold': 0,
  'text1': 'Mitä merkitystä sillä on?',
  'text2': 'Mitä väliä sillä edes on?',
  'label': '4',
  'binary_label': 'positive',
  'is_rewrite': False
}

Data Splits

The corpus include 3 splits: train, validation, and test.

Splitting Criteria

The data is split randomly into the three section with a restriction of all paraphrases from the same document (movie, TV episode, news article, student translation, or exam question) being in the same section. All splits are manually annotated.

Dataset in GEM

Rationale for Inclusion in GEM

Why is the Dataset in GEM?

This dataset provides a large amount of high quality (manually collected and verified) paraphrases for Finnish.

Similar Datasets

yes

Unique Language Coverage

no

Ability that the Dataset measures

natural language understanding, language variation

GEM-Specific Curation

Modificatied for GEM?

yes

GEM Modifications

data points modified

Modification Details

Data structure is slightly simplified, and the release provides ready made transformations into two tasks (paraphrase classification and generation), where some data instances are doubled with different direction, and some are discarded as not being suitable for generation (e.g. negatives).

Additional Splits?

no

Getting Started with the Task

Previous Results

Previous Results

Measured Model Abilities

natural language understanding, language variation

Previous results available?

yes

Other Evaluation Approaches

F-score in paraphrase classification

Dataset Curation

Original Curation

Original Curation Rationale

The dataset is fully manually annotated. The dataset strives for interesting paraphrases with low lexical overlap, thus the annotation is two fold. First the paraphrases are manually extracted from two related documents, where the annotators are instructed to extract only interesting paraphrases. In the second phrase, all extracted paraphrases are manually labeled given the annotation scheme.

The annotation scheme is: 4 : paraphrase in all reasonably possible contexts 3 : paraphrase in the given document contexts, but not in general 2 : related but not paraphrase During annotation also labels 1 (unrelated) and x (skip, e.g. wrong language) were used, however, the insignificant amount of examples annotated with these labels were discarded from the released corpus.

The following flags are annotated to label 4 paraphrases: < : txt1 is more general than txt2; txt2 is more specific than txt1 (directional paraphrase where txt2 can be replaced with txt1 in all contexts but not to the other direction)

: txt2 is more general than txt1; txt1 is more specific than txt2 (directional paraphrase where txt1 can be replaced with txt2 in all contexts but not to the other direction) i : minor traceable difference (differing in terms of grammatical number or case, 'this' vs 'that', etc.) s : style or strength difference (e.g. equivalent meaning, but one of the statements substantially more colloquial than the other)

For paraphrases where the annotated label was something else than label 4 without any flags, the annotators had an option to rewrite the text passages so that the rewritten paraphrase pair formed label 4 (universal) paraphrase. This was used for cases where simple edit would turn e.g. contextual or directional paraphrase into universal one. For the rewritten examples, both the original and the rewritten pairs are available with corresponding labels annotated.

Communicative Goal

Representing text passages with identical meaning but different surface realization.

Sourced from Different Sources

yes

Source Details

movie and TV series subtitles (82%) news articles (9%) discussion forum messages (8%) university translation exercises (1%) university course essays and exams (<1%)

Language Data

How was Language Data Obtained?

Found , Other

Where was it found?

Multiple websites , Offline media collection , Other

Language Producers

The movie and TV series subtitles are extracted from OPUS OpenSubtitles2018 collection, which is based on data from OpenSubtitles . The news articles are collected from two Finnish news sites, YLE and HS, during years 2017-2020. Discussion forum messages are obtained from the Finnish Suomi24 discussion forum released for academic use ( http://urn.fi/urn:nbn:fi:lb-2020021801 ). University translation exercises, essays and exams are collected during the project.

Data Validation

validated by data curator

Was Data Filtered?

not filtered

Structured Annotations

Additional Annotations?

expert created

Number of Raters

2<n<10

Rater Qualifications

Members of the TurkuNLP research group, native speakers of Finnish, each annotator has a strong background in language studies by having an academic degree or ongoing studies in a field related to languages or linguistics.

Raters per Training Example

1

Raters per Test Example

1

Annotation Service?

no

Annotation Values

Manual extraction of interesting paraphrases from two related documents.

Manual labeling of each extracted paraphrase based on the given annotation scheme, e.g. distinguishing contextual and universal paraphrases, marking style or strength differences, etc.

Any Quality Control?

validated by another rater

Quality Control Details

Partial double annotation, double annotation batches are assigned regularly in order to monitor annotation consistency. In double annotation, one annotator first extracts the candidate paraphrases, and these candidates are assigned to two different annotators, who does the label annotation independently from each other. Afterwards, the label annotations are merged, and conflicting labels are resolved together with the whole annotation team.

Consent

Any Consent Policy?

yes

Consent Policy Details

The corpus is mostly based on public/open data. For other data sources (student material), the licensing was agreed with the data providers during the collection.

Private Identifying Information (PII)

Contains PII?

likely

Categories of PII

generic PII

Any PII Identification?

no identification

Maintenance

Any Maintenance Plan?

no

Broader Social Context

Previous Work on the Social Impact of the Dataset

Usage of Models based on the Data

no

Impact on Under-Served Communities

Addresses needs of underserved Communities?

no

Discussion of Biases

Any Documented Social Biases?

no

Considerations for Using the Data

PII Risks and Liability

Potential PII Risk

None

Licenses

Copyright Restrictions on the Dataset

open license - commercial use allowed

Copyright Restrictions on the Language Data

open license - commercial use allowed

Known Technical Limitations