Dataset Card for GEM/cochrane-simplification

Link to Main Data Card

You can find the main data card on the GEM Website .

Dataset Summary

Cochrane is an English dataset for paragraph-level simplification of medical texts. Cochrane is a database of systematic reviews of clinical questions, many of which have summaries in plain English targeting readers without a university education. The dataset comprises about 4,500 of such pairs.

You can load the dataset via:

import datasets
data = datasets.load_dataset('GEM/cochrane-simplification')

The data loader can be found here .

website

Link

paper

Link

authors

Ashwin Devaraj (The University of Texas at Austin), Iain J. Marshall (King's College London), Byron C. Wallace (Northeastern University), Junyi Jessy Li (The University of Texas at Austin)

Dataset Overview

Where to find the Data and its Documentation

Webpage

Link

Download

Link

Paper

Link

BibTex

@inproceedings{devaraj-etal-2021-paragraph,
    title = "Paragraph-level Simplification of Medical Texts",
    author = "Devaraj, Ashwin  and
      Marshall, Iain  and
      Wallace, Byron  and
      Li, Junyi Jessy",
    booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    month = jun,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.naacl-main.395",
    doi = "10.18653/v1/2021.naacl-main.395",
    pages = "4972--4984",
}

Contact Name

Ashwin Devaraj

Contact Email

ashwin.devaraj@utexas.edu

Has a Leaderboard?

no

Languages and Intended Use

Multilingual?

no

Covered Languages

English

License

cc-by-4.0: Creative Commons Attribution 4.0 International

Intended Use

The intended use of this dataset is to train models that simplify medical text at the paragraph level so that it may be more accessible to the lay reader.

Primary Task

Simplification

Communicative Goal

A model trained on this dataset can be used to simplify medical texts to make them more accessible to readers without medical expertise.

Credit

Curation Organization Type(s)

academic

Curation Organization(s)

The University of Texas at Austin, King's College London, Northeastern University

Dataset Creators

Ashwin Devaraj (The University of Texas at Austin), Iain J. Marshall (King's College London), Byron C. Wallace (Northeastern University), Junyi Jessy Li (The University of Texas at Austin)

Funding

National Institutes of Health (NIH) grant R01-LM012086, National Science Foundation (NSF) grant IIS-1850153, Texas Advanced Computing Center (TACC) computational resources

Who added the Dataset to GEM?

Ashwin Devaraj (The University of Texas at Austin)

Dataset Structure

Data Fields

• gem_id : string, a unique identifier for the example
• doi : string, DOI identifier for the Cochrane review from which the example was generated
• source : string, an excerpt from an abstract of a Cochrane review
• target : string, an excerpt from the plain-language summary of a Cochrane review that roughly aligns with the source text

Example Instance

{
    "gem_id": "gem-cochrane-simplification-train-766",
    "doi": "10.1002/14651858.CD002173.pub2",
    "source": "Of 3500 titles retrieved from the literature, 24 papers reporting on 23 studies could be included in the review. The studies were published between 1970 and 1997 and together included 1026 participants. Most were cross-over studies. Few studies provided sufficient information to judge the concealment of allocation. Four studies provided results for the percentage of symptom-free days. Pooling the results did not reveal a statistically significant difference between sodium cromoglycate and placebo. For the other pooled outcomes, most of the symptom-related outcomes and bronchodilator use showed statistically significant results, but treatment effects were small. Considering the confidence intervals of the outcome measures, a clinically relevant effect of sodium cromoglycate cannot be excluded. The funnel plot showed an under-representation of small studies with negative results, suggesting publication bias. There is insufficient evidence to be sure about the efficacy of sodium cromoglycate over placebo. Publication bias is likely to have overestimated the beneficial effects of sodium cromoglycate as maintenance therapy in childhood asthma.",
    "target": "In this review we aimed to determine whether there is evidence for the effectiveness of inhaled sodium cromoglycate as maintenance treatment in children with chronic asthma. Most of the studies were carried out in small groups of patients. Furthermore, we suspect that not all studies undertaken have been published. The results show that there is insufficient evidence to be sure about the beneficial effect of sodium cromoglycate compared to placebo. However, for several outcome measures the results favoured sodium cromoglycate."
}

Data Splits

• train : 3568 examples
• validation : 411 examples
• test : 480 examples

Dataset in GEM

Rationale for Inclusion in GEM

Why is the Dataset in GEM?

This dataset is the first paragraph-level simplification dataset published (as prior work had primarily focused on simplifying individual sentences). Furthermore, this dataset is in the medical domain, which is an especially useful domain for text simplification.

Similar Datasets

no

Ability that the Dataset measures

This dataset measures the ability for a model to simplify paragraphs of medical text through the omission non-salient information and simplification of medical jargon.

GEM-Specific Curation

Modificatied for GEM?

no

Additional Splits?

no

Getting Started with the Task

Previous Results

Previous Results

Measured Model Abilities

This dataset measures the ability for a model to simplify paragraphs of medical text through the omission non-salient information and simplification of medical jargon.

Metrics

Other: Other Metrics , BLEU

Other Metrics

SARI measures the quality of text simplification

Previous results available?

yes

Relevant Previous Results

The paper which introduced this dataset trained BART models (pretrained on XSum) with unlikelihood training to produce simplification models achieving maximum SARI and BLEU scores of 40 and 43 respectively.

Dataset Curation

Original Curation

Sourced from Different Sources

no

Language Data

Data Validation

not validated

Was Data Filtered?

not filtered

Structured Annotations

Additional Annotations?

none

Annotation Service?

no

Consent

Any Consent Policy?

no

Private Identifying Information (PII)

Contains PII?

yes/very likely

Any PII Identification?

no identification

Maintenance

Any Maintenance Plan?

no

Broader Social Context

Previous Work on the Social Impact of the Dataset

Usage of Models based on the Data

no

Impact on Under-Served Communities

Addresses needs of underserved Communities?

yes

Details on how Dataset Addresses the Needs

This dataset can be used to simplify medical texts that may otherwise be inaccessible to those without medical training.

Discussion of Biases

Any Documented Social Biases?

unsure

Are the Language Producers Representative of the Language?

The dataset was generated from abstracts and plain-language summaries of medical literature reviews that were written by medical professionals and thus does was not generated by people representative of the entire English-speaking population.

Considerations for Using the Data

PII Risks and Liability

Licenses

Known Technical Limitations

Technical Limitations

The main limitation of this dataset is that the information alignment between the abstract and plain-language summary is often rough, so the plain-language summary may contain information that isn't found in the abstract. Furthermore, the plain-language targets often contain formulaic statements like "this evidence is current to [month][year]" not found in the abstracts. Another limitation is that some plain-language summaries do not simplify the technical abstracts very much and still contain medical jargon.

Unsuited Applications

The main pitfall to look out for is errors in factuality. Simplification work so far has not placed a strong emphasis on the logical fidelity of model generations with the input text, and the paper introducing this dataset does not explore modeling techniques to combat this. These kinds of errors are especially pernicious in the medical domain, and the models introduced in the paper do occasionally alter entities like disease and medication names.