GEM References

What is it?

This repository contains all the reference datasets that are used for running evaluation on the GEM benchmark. Some of these datasets were originally hosted as a GitHub release on the GEM-metrics repository, but have been migrated to the Hugging Face Hub.

Converting datasets to JSON

We provide a convert_dataset_to_json.py conversion script that converts the datasets in the GEM organisation to the JSON format expected by the GEM-metrics library. To run the script, first install jq and then install the script's Python dependencies:

python -m pip install -r requirements.txt

You can then run the script as follows:

python generate_evaluation_datasets.py

This script will:

• Download and convert the datasets under the GEM organisation to JSON format
• Validate that the each dataset has the expected columns of gem_id , target , and references