Dataset Card for AttrScore

• Repository: https://github.com/OSU-NLP-Group/AttrScore
• Paper: [Automatic Evaluation of Attribution by Large Language Models] ( https://arxiv.org/pdf/2305.06311.pdf )
• Point of Contact: Xiang Yue

Citation Information

@article{yue2023automatic,
  title={Automatic Evaluation of Attribution by Large Language Models},
  author={Yue, Xiang and Wang, Boshi and Zhang, Kai and Chen, Ziru and Su, Yu and Sun, Huan},
  journal={arXiv preprint arXiv:2305.06311},
  year={2023}
}

What's New?

In the current version 0.2, we fixed some wrong annotated labels in the AttrEval-GenSearch dataset. (Commit: 4da294f )

Dataset Summary

A recent focus of large language model (LLM) development, as exemplified by generative search engines, is to incorporate external references to generate and support its claims. However, evaluating the attribution, i.e., verifying whether the generated statement is indeed fully supported by the cited reference, remains an open problem.

We construct this dataset, which contains both training and test data for the evaluation of attribution. The training data are repurposed from related tasks, such as question answering, fact-checking, natural language inference, and summarization. The test data, cotains a set simulated from QA datasets and a set manually curated from a generative search engine, New Bing.

Dataset Structure

Data Instances

{ "query": "", "answer": "Bastedo cared for all the animals that inhabit the earth.", "reference": "Alexandra Lendon Bastedo (9 March 1946 - 12 January 2014) was a British actress, best known for her role as secret agent Sharron Macready in the 1968 British espionage/science fiction adventure series "The Champions". She has been cited as a sex symbol of the 1960s and 1970s. Bastedo was a vegetarian and animal welfare advocate.", "label": "Extrapolatory", "dataset": "anli" }

{ "query": The persian gulf war began when iraq invaded what country? "answer": The Persian Gulf War began when Iraq invaded Kuwait. "reference": First Iraq War or Iraq War, before the term "Iraq War" became identified instead with the 2003 Iraq War. The Iraqi Army's occupation of Kuwait that began 2 August 1990 was met with international condemnation and brought immediate economic sanctions against Iraq by members of the UN Security Council. Together with the UK's prime minister Margaret Thatcher - who had resisted the invasion by Argentina of the Falkland Islands a decade earlier - George H. W. Bush deployed US forces into Saudi Arabia, and urged other countries to send their own forces to the scene. An array of nations joined the coalition, forming the", "label": "Attributable", "dataset": "NaturalQuestions" }

Data Fields

• "query": query (may be empty)
• "answer": answer to the query
• "reference": a document or a paragraph
• "label": whether the reference can support the answer to the query ("attributable", "extrapolatory", "contradictory")
• "dataset": the original dataset of the data instance