Dataset Card for Variance-Aware MT Test Sets

Dataset Summary

This dataset comprises 70 small and discriminative test sets for machine translation (MT) evaluation called variance-aware test sets (VAT), covering 35 translation directions from WMT16 to WMT20 competitions. VAT is automatically created by a novel variance-aware filtering method that filters the indiscriminative test instances of the current MT benchmark without any human labor. Experimental results show that VAT outperforms the original WMT benchmark in terms of the correlation with human judgment across mainstream language pairs and test sets. Further analysis on the properties of VAT reveals the challenging linguistic features (e.g., translation of low-frequency words and proper nouns) for the competitive MT systems, providing guidance for constructing future MT test sets.

Disclaimer : *The VAT test sets are hosted through Github by the Natural Language Processing & Portuguese-Chinese Machine Translation Laboratory (NLP2CT Lab) of the University of Macau. They were introduced by the paper Variance-Aware Machine Translation Test Sets by Runzhe Zhan , Xuebo Liu , Derek F. Wong , Lidia S. Chao and follow the original licensing for WMT test sets.

Supported Tasks and Leaderboards

Refer to the original paper for additional details on model evaluation on VAT.

Languages

The following table taken from the original paper lists the languages supported by the VAT test sets, for a total of 70 language pairs:

To use any one of the test set, pass wmtXX_src_tgt as configuration name to the load_dataset command. E.g. to load the English-Russian test set from wmt16 , use load_dataset('gsarti/wmt_vat', 'wmt16_en_ru') .

Dataset Structure

Data Instances

A sample from the test split (the only available split) for the WMT16 English-Russian language ( wmt16_en_ru config) is provided below. All configurations have the same structure.

{
    'orig_id': 0,
    'source': 'The social card of residents of Ivanovo region is to be recognised as an electronic payment instrument.',
    'reference': 'Социальная карта жителя Ивановской области признается электронным средством платежа.'
}

The text is provided as-in the original dataset, without further preprocessing or tokenization.

Data Fields

• orig_id : Id corresponding to the row id in the original dataset, before variance-aware filtering.
• source : The source sentence.
• reference : The reference sentence in the target language.

Data Splits

Taken from the original repository:

Dataset Creation

The dataset was created by retaining a subset of the top 40% instances from various WMT test sets for which the variance between automatic scores (BLEU, BLEURT, COMET, BERTScore) was the highest. Please refer to the original article Variance-Aware Machine Translation Test Sets for additional information on dataset creation.

Additional Information

Dataset Curators

The original authors of VAT are the curators of the original dataset. For problems or updates on this ? Datasets version, please contact gabriele.sarti996@gmail.com .

Licensing Information

The variance-aware test set were created based on the original WMT test set. Thus, the the original data licensing plan already stated by WMT organizers is still applicable:

The data released for the WMT news translation task can be freely used for research purposes, we just ask that you cite the WMT shared task overview paper, and respect any additional citation requirements on the individual data sets. For other uses of the data, you should consult with original owners of the data sets.

Citation Information

Please cite the authors if you use these corpora in your work. It is also advised to cite the original WMT shared task paper for the specific test sets that were used.

@inproceedings{
    zhan2021varianceaware,
    title={Variance-Aware Machine Translation Test Sets},
    author={Runzhe Zhan and Xuebo Liu and Derek F. Wong and Lidia S. Chao},
    booktitle={Thirty-fifth Conference on Neural Information Processing Systems, Datasets and Benchmarks Track},
    year={2021},
    url={https://openreview.net/forum?id=hhKA5k0oVy5}
}