Dataset Card for PKU-SafeRLHF

Warning: this dataset contains data that may be offensive or harmful. The data are intended for research purposes, especially research that can make models less harmful. The views expressed in the data do not reflect the views of PKU-Alignment Team or any of its members.

Dataset Summary

The preference dataset consists of 30k+ expert comparison data. Each entry in this dataset includes two responses to a question, along with safety meta-labels and preferences for both responses, taking into consideration their helpfulness and harmlessness.

The annotation pipeline for this dataset is depicted in the following image:

Human-Preference on Harmlessness and Helpfulness

Harmlessness of a QA Pair

The harmlessness of a QA pair is evaluated based on ethical considerations and safety implications inherent in the interaction. A QA pair is deemed harmless when it is classified as risk-neutral across all 14 harm categories outlined in our paper. By risk-neutral, we mean that the QA pair does not engender or promote any harmful consequences or risks as per the definitions in these categories. Thus, a risk-neutral QA pair neither incite harm nor leads to unsafe outcomes, effectively aligning with our safety and ethical guidelines.

Helpfulness of a Response

The helpfulness of a response pertains to how effectively it addresses a given prompt. This measure is independent of the harmlessness of the response, as it focuses solely on the quality, clarity, and relevance of the provided information. Consequently, the helpfulness judgment can be distinctly different from the harmlessness judgment. For instance, consider a situation where a user asks about the procedure to synthesize methamphetamine. In such a case, a detailed, step-by-step response would be considered helpful due to its accuracy and thoroughness. However, due to the harmful implications of manufacturing illicit substances, this QA pair would be classified as extremely harmful.

Ranking of Responses

Once the helpfulness and harmlessness of responses are evaluated, they are ranked accordingly. It is important to note that this is a two-dimensional ranking: responses are ranked separately for helpfulness and harmlessness. This is due to the distinctive and independent nature of these two attributes. The resulting rankings provide a nuanced perspective on the responses, allowing us to balance information quality with safety and ethical considerations. These separate rankings of helpfulness and harmlessness contribute to a more comprehensive understanding of LLM outputs, particularly in the context of safety alignment. We have enforced a logical order to ensure the correctness of the harmlessness ranking: harmless responses (i.e. all 14 harm categories risk-neutral) are always ranked higher than harmful ones (i.e., at least 1 category risky).

Usage

To load our dataset, use the load_dataset() function as follows:

from datasets import load_dataset

dataset = load_dataset("PKU-Alignment/PKU-SafeRLHF")

Paper

You can find more information in our paper

• Dataset Paper: https://arxiv.org/abs/2307.04657

Contact

The original authors host this dataset on GitHub here: https://github.com/PKU-Alignment/beavertails .