X-GENRE classifier - multilingual text genre classifier

Text classification model based on xlm-roberta-base and fine-tuned on a combination of three genre datasets: Slovene GINCO 1 dataset, the English CORE 2 dataset and the English FTD 3 dataset. The model can be used for automatic genre identification, applied to any text in a language, supported by the xlm-roberta-base .

Model description

The model was fine-tuned on the "X-GENRE" dataset which consists of three genre datasets: CORE, FTD and GINCO dataset. Each of the datasets has their own genre schema, so they were combined into a joint schema ("X-GENRE" schema) based on the comparison of labels and cross-dataset experiments (described in details here ).

Fine-tuning hyperparameters

Fine-tuning was performed with simpletransformers . Beforehand, a brief hyperparameter optimization was performed and the presumed optimal hyperparameters are:

model_args= {
            "num_train_epochs": 15,
            "learning_rate": 1e-5,
            "max_seq_length": 512,
            }        
      

Intended use and limitations

Usage

An example of preparing data for genre identification and post-processing of the results can be found here where we applied X-GENRE classifier to the English part of MaCoCu parallel corpora.

For reliable results, genre classifier should be applied to documents of sufficient length (the rule of thumbs is at least 75 words). It is advised that the predictions, predicted with confidence lower than 0.9, are not used. Furthermore, the label "Other" can be used as another indicator of low confidence of the predictions, as it often indicates that the text does not have enough features of any genre, and these predictions can be discarded as well.

After proposed post-processing (removal of low-confidence predictions, labels "Other" and in this specific case also label "Forum"), the performance on the MaCoCu data based on manual inspection reached macro and micro F1 of 0.92.

Use examples

from simpletransformers.classification import ClassificationModel
model_args= {
            "num_train_epochs": 15,
            "learning_rate": 1e-5,
            "max_seq_length": 512,
            "silent": True
            }
model = ClassificationModel(
    "xlmroberta", "classla/xlm-roberta-base-multilingual-text-genre-classifier", use_cuda=True,
    args=model_args
    
)
predictions, logit_output = model.predict(["How to create a good text classification model? First step is to prepare good data. Make sure not to skip the exploratory data analysis. Pre-process the text if necessary for the task. The next step is to perform hyperparameter search to find the optimum hyperparameters. After fine-tuning the model, you should look into the predictions and analyze the model's performance. You might want to perform the post-processing of data as well and keep only reliable predictions.", 
                                        "On our site, you can find a great genre identification model which you can use for thousands of different tasks. With our model, you can fastly and reliably obtain high-quality genre predictions and explore which genres exist in your corpora. Available for free!"]
                                        )
predictions
# Output: array([3, 8])

[model.config.id2label[i] for i in predictions]
# Output: ['Instruction', 'Promotion']

Use example for prediction on a dataset, using batch processing, is available via Google Collab .

X-GENRE categories

List of labels:

labels_list=['Other', 'Information/Explanation', 'News', 'Instruction', 'Opinion/Argumentation', 'Forum', 'Prose/Lyrical', 'Legal', 'Promotion'],

labels_map={'Other': 0, 'Information/Explanation': 1, 'News': 2, 'Instruction': 3, 'Opinion/Argumentation': 4, 'Forum': 5, 'Prose/Lyrical': 6, 'Legal': 7, 'Promotion': 8}

Description of labels:

Performance

Comparison with other models at in-dataset and cross-dataset experiments

The X-GENRE model was compared with xlm-roberta-base classifiers, fine-tuned on each of genre datasets separately, using the X-GENRE schema (see experiments in https://github.com/TajaKuzman/Genre-Datasets-Comparison ).

At the in-dataset experiments (trained and tested on splits of the same dataset), it outperforms all datasets, except the FTD dataset which has a smaller number of X-GENRE labels.

When applied on test splits of each of the datasets, the classifier performs well:

The classifier was compared with other classifiers on 2 additional genre datasets (to which the X-GENRE schema was mapped):

• EN-GINCO: a sample of the English enTenTen20 corpus
• FinCORE : Finnish CORE corpus

At cross-dataset and cross-lingual experiments, it was shown that the X-GENRE classifier, trained on all three datasets, outperforms classifiers that were trained on just one of the datasets.

Citation

If you use the model, please cite the GitHub repository where the fine-tuning experiments are explained:

 @misc{Kuzman2022,
  author = {Kuzman, Taja},
  title = {{Comparison of genre datasets: CORE, GINCO and FTD}},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/TajaKuzman/Genre-Datasets-Comparison}}
}

and the following paper on which the original model is based:

@article{DBLP:journals/corr/abs-1911-02116,
  author    = {Alexis Conneau and
               Kartikay Khandelwal and
               Naman Goyal and
               Vishrav Chaudhary and
               Guillaume Wenzek and
               Francisco Guzm{\'{a}}n and
               Edouard Grave and
               Myle Ott and
               Luke Zettlemoyer and
               Veselin Stoyanov},
  title     = {Unsupervised Cross-lingual Representation Learning at Scale},
  journal   = {CoRR},
  volume    = {abs/1911.02116},
  year      = {2019},
  url       = {http://arxiv.org/abs/1911.02116},
  eprinttype = {arXiv},
  eprint    = {1911.02116},
  timestamp = {Mon, 11 Nov 2019 18:38:09 +0100},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1911-02116.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

To cite the datasets that were used for fine-tuning:

CORE dataset:

@article{egbert2015developing,
  title={Developing a bottom-up, user-based method of web register classification},
  author={Egbert, Jesse and Biber, Douglas and Davies, Mark},
  journal={Journal of the Association for Information Science and Technology},
  volume={66},
  number={9},
  pages={1817--1831},
  year={2015},
  publisher={Wiley Online Library}
}

GINCO dataset:

@InProceedings{kuzman-rupnik-ljubei:2022:LREC,
  author    = {Kuzman, Taja  and  Rupnik, Peter  and  Ljube{\v{s}}i{\'c}, Nikola},
  title     = {{The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild}},
  booktitle      = {Proceedings of the Language Resources and Evaluation Conference},
  month          = {},
  year           = {2022},
  address        = {Marseille, France},
  publisher      = {European Language Resources Association},
  pages     = {1584--1594},
  url       = {https://aclanthology.org/2022.lrec-1.170}
}

FTD dataset:

@article{sharoff2018functional,
  title={Functional text dimensions for the annotation of web corpora},
  author={Sharoff, Serge},
  journal={Corpora},
  volume={13},
  number={1},
  pages={65--95},
  year={2018},
  publisher={Edinburgh University Press The Tun-Holyrood Road, 12 (2f) Jackson's Entry~…}
}

The datasets are available at: