Dataset Card for "openwebtext"

Dataset Summary

An open-source replication of the WebText dataset from OpenAI, that was used to train GPT-2.

This distribution was created by Aaron Gokaslan and Vanya Cohen of Brown University.

Supported Tasks and Leaderboards

More Information Needed

Languages

More Information Needed

Dataset Structure

Data Instances

plain_text

• Size of downloaded dataset files: 13.51 GB
• Size of the generated dataset: 41.70 GB
• Total amount of disk used: 55.21 GB

An example of 'train' looks as follows.

This example was too long and was cropped:

{
    "text": "\"A magazine supplement with an image of Adolf Hitler and the title 'The Unreadable Book' is pictured in Berlin. No law bans “Mei..."
}

Data Fields

The data fields are the same among all splits.

plain_text

• text : a string feature.

Data Splits

name	train
plain_text	8013769

Dataset Creation

Curation Rationale

More Information Needed

Source Data

Initial Data Collection and Normalization

The authors started by extracting all Reddit post urls from the Reddit submissions dataset. These links were deduplicated, filtered to exclude non-html content, and then shuffled randomly. The links were then distributed to several machines in parallel for download, and all web pages were extracted using the newspaper python package. Using Facebook FastText, non-English web pages were filtered out.

Subsequently, near-duplicate documents were identified using local-sensitivity hashing (LSH). Documents were hashed into sets of 5-grams and all documents that had a similarity threshold of greater than 0.5 were removed. The the remaining documents were tokenized, and documents with fewer than 128 tokens were removed. This left 38GB of text data (40GB using SI units) from 8,013,769 documents.

Who are the source language producers?

More Information Needed

Annotations

The dataset doesn't contain annotations.

Personal and Sensitive Information

More Information Needed

Considerations for Using the Data

Social Impact of Dataset

More Information Needed

Discussion of Biases

More Information Needed

Other Known Limitations

More Information Needed

Additional Information

Dataset Curators

More Information Needed

Licensing Information

These data are released under this licensing scheme from the original authors ( source ):

We do not own any of the text from which these data has been extracted.

We license the actual packaging of these parallel data under the [Creative Commons CC0 license (“no rights reserved”)](https://creativecommons.org/share-your-work/public-domain/cc0/)

Notice policy

Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please:

Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted.

Clearly identify the copyrighted work claimed to be infringed.

Clearly identify the material that is claimed to be infringing and information reasonably sufficient to allow us to locate the material.

And contact us at the following email address: openwebtext at gmail.com and datasets at huggingface.co

Take down policy

The original authors will comply to legitimate requests by removing the affected sources from the next release of the corpus. Hugging Face will also update this repository accordingly.

Citation Information

@misc{Gokaslan2019OpenWeb,
    title={OpenWebText Corpus},
    author={Aaron Gokaslan*, Vanya Cohen*, Ellie Pavlick, Stefanie Tellex},
    howpublished{\url{http://Skylion007.github.io/OpenWebTextCorpus}},
    year={2019}
}

Contributions

Thanks to @richarddwang for adding this dataset.