Dataset Card for GEM/SciDuet

Link to Main Data Card

You can find the main data card on the GEM Website .

Dataset Summary

This dataset supports the document-to-slide generation task where a model has to generate presentation slide content from the text of a document.

You can load the dataset via:

import datasets
data = datasets.load_dataset('GEM/SciDuet')

The data loader can be found here .

website

Huggingface

paper

ACL Anthology

authors

Edward Sun, Yufang Hou, Dakuo Wang, Yunfeng Zhang, Nancy Wang

Dataset Overview

Where to find the Data and its Documentation

Webpage

Huggingface

Download

Github

Paper

ACL Anthology

BibTex

@inproceedings{sun-etal-2021-d2s,
    title = "{D}2{S}: Document-to-Slide Generation Via Query-Based Text Summarization",
    author = "Sun, Edward  and
      Hou, Yufang  and
      Wang, Dakuo  and
      Zhang, Yunfeng  and
      Wang, Nancy X. R.",
    booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    month = jun,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.naacl-main.111",
    doi = "10.18653/v1/2021.naacl-main.111",
    pages = "1405--1418",
    abstract = "Presentations are critical for communication in all areas of our lives, yet the creation of slide decks is often tedious and time-consuming. There has been limited research aiming to automate the document-to-slides generation process and all face a critical challenge: no publicly available dataset for training and benchmarking. In this work, we first contribute a new dataset, SciDuet, consisting of pairs of papers and their corresponding slides decks from recent years{'} NLP and ML conferences (e.g., ACL). Secondly, we present D2S, a novel system that tackles the document-to-slides task with a two-step approach: 1) Use slide titles to retrieve relevant and engaging text, figures, and tables; 2) Summarize the retrieved context into bullet points with long-form question answering. Our evaluation suggests that long-form QA outperforms state-of-the-art summarization baselines on both automated ROUGE metrics and qualitative human evaluation.",
}

Has a Leaderboard?

no

Languages and Intended Use

Multilingual?

no

Covered Languages

English

License

apache-2.0: Apache License 2.0

Intended Use

Promote research on the task of document-to-slides generation

Primary Task

Text-to-Slide

Credit

Curation Organization Type(s)

industry

Curation Organization(s)

IBM Research

Dataset Creators

Edward Sun, Yufang Hou, Dakuo Wang, Yunfeng Zhang, Nancy Wang

Funding

IBM Research

Who added the Dataset to GEM?

Yufang Hou (IBM Research), Dakuo Wang (IBM Research)

Dataset Structure

How were labels chosen?

The original papers and slides (both are in PDF format) are carefully processed by a combination of PDF/Image processing tookits. The text contents from multiple slides that correspond to the same slide title are mreged.

Data Splits

Training, validation and testing data contain 136, 55, and 81 papers from ACL Anthology and their corresponding slides, respectively.

Splitting Criteria

The dataset integrated into GEM is the ACL portion of the whole dataset described in the paper , It contains the full Dev and Test sets, and a portion of the Train dataset. Note that although we cannot release the whole training dataset due to copyright issues, researchers can still use our released data procurement code to generate the training dataset from the online ICML/NeurIPS anthologies.

Dataset in GEM

Rationale for Inclusion in GEM

Why is the Dataset in GEM?

SciDuet is the first publicaly available dataset for the challenging task of document2slides generation, which requires a model has a good ability to "understand" long-form text, choose appropriate content and generate key points.

Similar Datasets

no

Ability that the Dataset measures

content selection, long-form text undersanding and generation

GEM-Specific Curation

Modificatied for GEM?

no

Additional Splits?

no

Getting Started with the Task

Previous Results

Previous Results

Measured Model Abilities

content selection, long-form text undersanding and key points generation

Metrics

ROUGE

Proposed Evaluation

Automatical Evaluation Metric: ROUGE Human Evaluation: (Readability, Informativeness, Consistency)

Readability: The generated slide content is coherent, concise, and grammatically correct;

Informativeness: The generated slide provides sufficient and necessary information that corresponds to the given slide title, regardless of its similarity to the original slide;

Consistency: The generated slide content is similar to the original author’s reference slide.

Previous results available?

yes

Other Evaluation Approaches

ROUGE + Human Evaluation

Relevant Previous Results

Paper "D2S: Document-to-Slide Generation Via Query-Based Text Summarization" reports 20.47, 5.26 and 19.08 for ROUGE-1, ROUGE-2 and ROUGE-L (f-score).

Dataset Curation

Original Curation

Original Curation Rationale

Provide a benchmark dataset for the document-to-slides task.

Sourced from Different Sources

no

Language Data

How was Language Data Obtained?

Other

Data Validation

not validated

Data Preprocessing

Text on papers was extracted through Grobid. Figures andcaptions were extracted through pdffigures. Text on slides was extracted through IBM Watson Discovery package and OCR by pytesseract. Figures and tables that appear on slides and papers were linked through multiscale template matching by OpenCV. Further dataset cleaning was performed with standard string-based heuristics on sentence building, equation and floating caption removal, and duplicate line deletion.

Was Data Filtered?

algorithmically

Filter Criteria

the slide context text shouldn't contain additional format information such as "*** University"

Structured Annotations

Additional Annotations?

none

Annotation Service?

no

Consent

Any Consent Policy?

yes

Consent Policy Details

The original dataset was open-sourced under Apache-2.0. Some of the original dataset creators are part of the GEM v2 dataset infrastructure team and take care of integrating this dataset into GEM.

Private Identifying Information (PII)

Contains PII?

yes/very likely

Categories of PII

generic PII

Any PII Identification?

no identification

Maintenance

Any Maintenance Plan?

no

Broader Social Context

Previous Work on the Social Impact of the Dataset

Usage of Models based on the Data

no

Impact on Under-Served Communities

Addresses needs of underserved Communities?

no

Discussion of Biases

Any Documented Social Biases?

unsure

Considerations for Using the Data

PII Risks and Liability

Licenses

Copyright Restrictions on the Dataset

non-commercial use only

Copyright Restrictions on the Language Data

research use only

Known Technical Limitations