Dataset Card for ScandiReddit

Dataset Summary

ScandiReddit is a filtered and post-processed corpus consisting of comments from Reddit .

All Reddit comments from December 2005 up until October 2022 were downloaded through PushShift , after which these were filtered based on the FastText language detection model. Any comment which was classified as Danish ( da ), Norwegian ( no ), Swedish ( sv ) or Icelandic ( is ) with a confidence score above 70% was kept.

The resulting comments were then deduplicated, removing roughly 438,000 comments. 5,000 comments written by Reddit bots were removed, and roughly 189,000 comments belonging to inappropriate subreddits (explicit and drug-related) were also removed.

Lastly, we remove roughly 40,000 near-duplicate comments from the resulting corpus, where near-duplicate here means that the comments have more than 80% of their word 5-grams in common.

Supported Tasks and Leaderboards

Training language models is the intended task for this dataset. No leaderboard is active at this point.

Languages

The dataset is available in Danish ( da ), Swedish ( sv ), Norwegian ( no ) and Icelandic ( is ).

Dataset Structure

Data Instances

• Size of downloaded dataset files: 2341 MB
• Size of the generated dataset: 3594 MB
• Total amount of disk used: 5935 MB

An example from the dataset looks as follows.

{
    'doc': 'Bergen er ødelagt. Det er ikke moro mer.',
    'subreddit': 'Norway',
    'language': 'da',
    'language_confidence': 0.7472341656684875
}

Data Fields

The data fields are the same among all splits.

• doc : a string feature.
• subreddit : a string feature.
• language : a string feature.
• language_confidence : a float64 feature.

Language Distribution

Top-50 Subreddit Distribution

Dataset Creation

Curation Rationale

The Scandinavian languages do not have many open source social media datasets.

Source Data

The raw Reddit data was collected through PushShift .

Additional Information

Dataset Curators

Dan Saattrup Nielsen from the The Alexandra Institute curated this dataset.

Licensing Information

The dataset is licensed under the CC BY 4.0 license .