Dataset card for Measuring Hate Speech

This is a public release of the dataset described in Kennedy et al. (2020) and Sachdeva et al. (2022), consisting of 39,565 comments annotated by 7,912 annotators, for 135,556 combined rows. The primary outcome variable is the "hate speech score" but the 10 constituent ordinal labels (sentiment, (dis)respect, insult, humiliation, inferior status, violence, dehumanization, genocide, attack/defense, hate speech benchmark) can also be treated as outcomes. Includes 8 target identity groups (race/ethnicity, religion, national origin/citizenship, gender, sexual orientation, age, disability, political ideology) and 42 target identity subgroups, as well as 6 annotator demographics and 40 subgroups. The hate speech score incorporates an IRT adjustment by estimating variation in annotator interpretation of the labeling guidelines.

This dataset card is a work in progress and will be improved over time.

Key dataset columns

• hate_speech_score - continuous hate speech measure, where higher = more hateful and lower = less hateful. > 0.5 is approximately hate speech, < -1 is counter or supportive speech, and -1 to +0.5 is neutral or ambiguous.
• text - lightly processed text of a social media post
• comment_id - unique ID for each comment
• annotator_id - unique ID for each annotator
• sentiment - ordinal label that is combined into the continuous score
• respect - ordinal label that is combined into the continuous score
• insult - ordinal label that is combined into the continuous score
• humiliate - ordinal label that is combined into the continuous score
• status - ordinal label that is combined into the continuous score
• dehumanize - ordinal label that is combined into the continuous score
• violence - ordinal label that is combined into the continuous score
• genocide - ordinal label that is combined into the continuous score
• attack_defend - ordinal label that is combined into the continuous score
• hatespeech - ordinal label that is combined into the continuous score
• annotator_severity - annotator's estimated survey interpretation bias

Code to download

The dataset can be downloaded using the following python code:

import datasets 
dataset = datasets.load_dataset('ucberkeley-dlab/measuring-hate-speech', 'binary')   
df = dataset['train'].to_pandas()
df.describe()

Citation

@article{kennedy2020constructing,
  title={Constructing interval variables via faceted Rasch measurement and multitask deep learning: a hate speech application},
  author={Kennedy, Chris J and Bacon, Geoff and Sahn, Alexander and von Vacano, Claudia},
  journal={arXiv preprint arXiv:2009.10277},
  year={2020}
}

Contributions

Dataset curated by @ck37 , @pssachdeva , et al.

References

Kennedy, C. J., Bacon, G., Sahn, A., & von Vacano, C. (2020). Constructing interval variables via faceted Rasch measurement and multitask deep learning: a hate speech application . arXiv preprint arXiv:2009.10277.

Pratik Sachdeva, Renata Barreto, Geoff Bacon, Alexander Sahn, Claudia von Vacano, and Chris Kennedy. 2022. The Measuring Hate Speech Corpus: Leveraging Rasch Measurement Theory for Data Perspectivism . In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022 , pages 83–94, Marseille, France. European Language Resources Association.