Dataset Card for comparison-data-falcon-with-feedback

This dataset has been created with Argilla .

As shown in the sections below, this dataset can be loaded into Argilla as explained in Load with Argilla , or used directly with the datasets library in Load with datasets .

Dataset Summary

This dataset contains:

• A dataset configuration file conforming to the Argilla dataset format named argilla.cfg . This configuration file will be used to configure the dataset when using the FeedbackDataset.from_huggingface method in Argilla.

• Dataset records in a format compatible with HuggingFace datasets . These records will be loaded automatically when using FeedbackDataset.from_huggingface and can be loaded independently using the datasets library via load_dataset .

• The annotation guidelines that have been used for building and curating the dataset, if they've been defined in Argilla.

Load with Argilla

To load with Argilla, you'll just need to install Argilla as pip install argilla --upgrade and then use the following code:

import argilla as rg

ds = rg.FeedbackDataset.from_huggingface("argilla/comparison-data-falcon-with-feedback")

Load with datasets

To load this dataset with datasets , you'll just need to install datasets as pip install datasets --upgrade and then use the following code:

from datasets import load_dataset

ds = load_dataset("argilla/comparison-data-falcon-with-feedback")

Supported Tasks and Leaderboards

This dataset can contain multiple fields, questions and responses so it can be used for different NLP tasks, depending on the configuration. The dataset structure is described in the Dataset Structure section .

There are no leaderboards associated with this dataset.

Languages

[More Information Needed]

Dataset Structure

Data in Argilla

The dataset is created in Argilla with: fields , questions , and guidelines .

The fields are the dataset records themselves, for the moment just text fields are suppported. These are the ones that will be used to provide responses to the questions.

Field Name	Title	Type	Required	Markdown
instruction	User instruction	TextField	True	False
response-1	Response-1	TextField	True	False
response-2	Response-2	TextField	True	False

The questions are the questions that will be asked to the annotators. They can be of different types, such as rating, text, single choice, or multiple choice.

Question Name	Title	Type	Required	Description	Values/Labels
choose-best	Choose the best response:	RatingQuestion	True	Choose the most helpful, harmless, and truthful response. Select 1 for response-1, 2 for response-2, or discard if both are equally good/bad.	[1, 2]

Finally, the guidelines are just a plain string that can be used to provide instructions to the annotators. Find those in the annotation guidelines section.

Data Instances

An example of a dataset instance in Argilla looks as follows:

{
    "external_id": "518",
    "fields": {
        "instruction": "What is Depreciation",
        "response-1": "Depreciation is the drop in value of an asset due to wear and tear, age and obsolescence (going out of date) as recorded in an organization\u0027s financial records.",
        "response-2": "What is Depreciation \u2013 10 Important Facts to Know?\nWhen a business buys a new asset, the purchase price of that asset is depreciated over time to reflect its usage and eventual obsolescence. Depreciation expense can be a tax deductible expense and is usually a non-cash expense reported on a company\u2019s income statement and balance sheet. The amount of depreciation expense a company reports each year is the difference between the original purchase price of the asset and what the current value of that asset might be. Here are 10 important facts to know about depreciation:\n1. Depreciation is a non-cash expense. It is an expense that is reported in a business\u2019s income statement and balance sheet and not a cash flow expense.\n2. Depreciation is an accounting standard and it is required to be disclosed in a business\u2019s financial statements.\n3. The amount of depreciation is usually a tax expense and not a cash expense reported on a company\u2019s income statement"
    },
    "responses": [
        {
            "status": "submitted",
            "user_id": null,
            "values": {
                "choose-best": {
                    "value": 1
                }
            }
        }
    ]
}

While the same record in HuggingFace datasets looks as follows:

{
    "choose-best": {
        "status": [
            "submitted"
        ],
        "user_id": [
            null
        ],
        "value": [
            1
        ]
    },
    "external_id": "518",
    "instruction": "What is Depreciation",
    "response-1": "Depreciation is the drop in value of an asset due to wear and tear, age and obsolescence (going out of date) as recorded in an organization\u0027s financial records.",
    "response-2": "What is Depreciation \u2013 10 Important Facts to Know?\nWhen a business buys a new asset, the purchase price of that asset is depreciated over time to reflect its usage and eventual obsolescence. Depreciation expense can be a tax deductible expense and is usually a non-cash expense reported on a company\u2019s income statement and balance sheet. The amount of depreciation expense a company reports each year is the difference between the original purchase price of the asset and what the current value of that asset might be. Here are 10 important facts to know about depreciation:\n1. Depreciation is a non-cash expense. It is an expense that is reported in a business\u2019s income statement and balance sheet and not a cash flow expense.\n2. Depreciation is an accounting standard and it is required to be disclosed in a business\u2019s financial statements.\n3. The amount of depreciation is usually a tax expense and not a cash expense reported on a company\u2019s income statement"
}

Data Fields

Among the dataset fields, we differentiate between the following:

• Fields: These are the dataset records themselves, for the moment just text fields are suppported. These are the ones that will be used to provide responses to the questions.

  • instruction is of type TextField .
  • response-1 is of type TextField .
  • response-2 is of type TextField .

• Questions: These are the questions that will be asked to the annotators. They can be of different types, such as rating, text, single choice, or multiple choice.

  • choose-best is of type RatingQuestion with the following allowed values [1, 2], and description "Choose the most helpful, harmless, and truthful response. Select 1 for response-1, 2 for response-2, or discard if both are equally good/bad.".

Additionally, we also have one more field which is optional and is the following:

• external_id: This is an optional field that can be used to provide an external ID for the dataset record. This can be useful if you want to link the dataset record to an external resource, such as a database or a file.

Data Splits

The dataset contains a single split, which is train .

Dataset Creation

Curation Rationale

[More Information Needed]

Source Data

Initial Data Collection and Normalization

[More Information Needed]

Who are the source language producers?

[More Information Needed]

Annotations

Annotation guidelines

These guidelines are based on the paper Training Language Models to Follow Instructions with Human Feedback

You are given a text-based description of a task, submitted by a user.

This task description may be in the form of an explicit instruction (e.g. "Write a story about a wise frog."). The task may also be specified indirectly, for example by using several examples of the desired behavior (e.g. given a sequence of movie reviews followed by their sentiment, followed by one movie review without a sentiment, you can assume the task is to predict the sentiment of the final review), or by generating the start of a desired output (e.g. given "There once was a wise frog named Julius," you can assume the task is to continue the story).

You will also be given several text outputs, intended to help the user with their task. Your job is to evaluate these outputs to ensure that they are helpful, truthful, and harmless, and select the best according to these criteria.

In case none of the outputs is correct, conforming to the criteria, or they're exactly equal, use the discard option to not rank them .

For most tasks, being truthful and harmless is more important than being helpful.

By helpful , we mean that the output should follow the user's intention and help the user solve their task. Some examples of helpful behavior:

• Writing in clear language.
• Answering the question they meant to ask, even if they mis-asked it.
• Being sensitive to internationality (e.g., "football" shouldn't mean American football, and "the president" doesn't necessarily mean the US president).
• If an instruction is too confusing, asking for clarification and explaining why the instruction is confusing.
• Not giving overly long or rambling answers, or repeating information from the question.
• Not assuming extraneous extra context outside of what's given (besides things like facts about the world), unless that's an implied part of the task. For example, if asked to "write a polite email response to this email: email body," the output shouldn't assume "I can't make it this time, but am free next weekend." But if asked to "Write an email to Socrates," then you can safely assume the output can make up a context.

By truthful , we mean that the output contains accurate information and doesn't mislead the user. Some examples of truthful behavior:

• On tasks like summarization, where the output should only use information from input, not making up details that are not part of the input description.
• Not producing clearly false information about the world (e.g., making up facts or promoting conspiracies). For example, the output should not state that Hillary Clinton has served time in prison.
• Avoid generating misleading information or information with questionable authenticity. For example, if the input asks "Why did Hillary Clinton go to jail?", the output should not say "It's not totally clear," but rather should refute the premise of the question.

By harmless , we mean that the output should not cause physical, psychological, or social harm to people; damage to or loss of equipment or property; damage to the environment; or harm to institutions or resources necessary to human well-being. Some examples of harmless behavior:

• Treating other humans with kindness, respect, and consideration; not denigrating members of certain groups or using biased language against a particular group.
• Not generating abusive, threatening, or offensive language or promoting violence.
• Not writing sexual or violent content if it's not asked for.
• Not giving bad real-world advice or promoting illegal activity.

Evaluating model outputs may involve making trade-offs between these criteria. These trade-offs will depend on the task.

Annotation process

[More Information Needed]

Who are the annotators?

[More Information Needed]

Personal and Sensitive Information

[More Information Needed]

Considerations for Using the Data

Social Impact of Dataset

[More Information Needed]

Discussion of Biases

[More Information Needed]

Other Known Limitations

[More Information Needed]

Additional Information

Dataset Curators

[More Information Needed]

Licensing Information

[More Information Needed]

Citation Information

[More Information Needed]

Contributions

[More Information Needed]