KPWR-NER

Description

KPWR-NER is a part the Polish Corpus of Wrocław University of Technology ( Korpus Języka Polskiego Politechniki Wrocławskiej ). Its objective is named entity recognition for fine-grained categories of entities. It is the ‘n82’ version of the KPWr, which means that number of classes is restricted to 82 (originally 120). During corpus creation, texts were annotated by humans from various sources, covering many domains and genres.

Tasks (input, output and metrics)

Named entity recognition (NER) - tagging entities in text with their corresponding type.

Input (' tokens' column): sequence of tokens

Output (' ner' column): sequence of predicted tokens’ classes in BIO notation (82 possible classes, described in detail in the annotation guidelines)

Measurements : F1-score (seqeval)

Example :

Input: [‘Roboty’, ‘mają’, ‘kilkanaście’, ‘lat’, ‘i’, ‘pochodzą’, ‘z’, ‘USA’, ‘,’, ‘Wysokie’, ‘napięcie’, ‘jest’, ‘dużo’, ‘młodsze’, ‘,’, ‘powstało’, ‘w’, ‘Niemczech’, ‘.’]

Input (translated by DeepL): Robots are more than a dozen years old and come from the US, High Voltage is much younger, having been developed in Germany.

Output: [‘B-nam_pro_title’, ‘O’, ‘O’, ‘O’, ‘O’, ‘O’, ‘O’, ‘B-nam_loc_gpe_country’, ‘O’, ‘B-nam_pro_title’, ‘I-nam_pro_title’, ‘O’, ‘O’, ‘O’, ‘O’, ‘O’, ‘O’, ‘B-nam_loc_gpe_country’, ‘O’]

Data splits

Class distribution (without "O" and "I-*")

Citation

@inproceedings{broda-etal-2012-kpwr,
    title = "{KPW}r: Towards a Free Corpus of {P}olish",
    author = "Broda, Bartosz  and
      Marci{\'n}czuk, Micha{\l}  and
      Maziarz, Marek  and
      Radziszewski, Adam  and
      Wardy{\'n}ski, Adam",
    booktitle = "Proceedings of the Eighth International Conference on Language Resources and Evaluation ({LREC}'12)",
    month = may,
    year = "2012",
    address = "Istanbul, Turkey",
    publisher = "European Language Resources Association (ELRA)",
    url = "http://www.lrec-conf.org/proceedings/lrec2012/pdf/965_Paper.pdf",
    pages = "3218--3222",
    abstract = "This paper presents our efforts aimed at collecting and annotating a free Polish corpus. The corpus will serve for us as training and testing material for experiments with Machine Learning algorithms. As others may also benefit from the resource, we are going to release it under a Creative Commons licence, which is hoped to remove unnecessary usage restrictions, but also to facilitate reproduction of our experimental results. The corpus is being annotated with various types of linguistic entities: chunks and named entities, selected syntactic and semantic relations, word senses and anaphora. We report on the current state of the project as well as our ultimate goals.",
}

License

Creative Commons Attribution 3.0 Unported Licence

Links

HuggingFace

Source

Paper

KPWr annotation guidelines

KPWr annotation guidelines - named entities

Examples

Loading

from pprint import pprint

from datasets import load_dataset

dataset = load_dataset("clarin-pl/kpwr-ner")
pprint(dataset['train'][0])

# {'lemmas': ['roborally', 'czy', 'wysoki', 'napięcie', '?'],
#  'ner': [73, 160, 73, 151, 160],
#  'orth': ['subst:sg:nom:n',
#           'qub',
#           'adj:sg:nom:n:pos',
#           'subst:sg:nom:n',
#           'interp'],
#  'tokens': ['RoboRally', 'czy', 'Wysokie', 'napięcie', '?']}

Evaluation

import random
from pprint import pprint

from datasets import load_dataset, load_metric

dataset = load_dataset("clarin-pl/kpwr-ner")
references = dataset["test"]["ner"]

# generate random predictions
predictions = [
    [
        random.randrange(dataset["train"].features["ner"].feature.num_classes)
        for _ in range(len(labels))
    ]
    for labels in references
]

# transform to original names of labels
references_named = [
    [dataset["train"].features["ner"].feature.names[label] for label in labels]
    for labels in references
]
predictions_named = [
    [dataset["train"].features["ner"].feature.names[label] for label in labels]
    for labels in predictions
]

# utilise seqeval to evaluate
seqeval = load_metric("seqeval")
seqeval_score = seqeval.compute(
    predictions=predictions_named, references=references_named, scheme="IOB2"
)

pprint(seqeval_score, depth=1)

# {'nam_adj': {...},
#  'nam_adj_city': {...},
#  'nam_adj_country': {...},
#  'nam_adj_person': {...},
#  'nam_eve': {...},
#  'nam_eve_human': {...},
#  'nam_eve_human_cultural': {...},
#  'nam_eve_human_holiday': {...},
#  'nam_eve_human_sport': {...},
#  'nam_fac_bridge': {...},
#  'nam_fac_goe': {...},
#  'nam_fac_goe_stop': {...},
#  'nam_fac_park': {...},
#  'nam_fac_road': {...},
#  'nam_fac_square': {...},
#  'nam_fac_system': {...},
#  'nam_liv_animal': {...},
#  'nam_liv_character': {...},
#  'nam_liv_god': {...},
#  'nam_liv_habitant': {...},
#  'nam_liv_person': {...},
#  'nam_loc': {...},
#  'nam_loc_astronomical': {...},
#  'nam_loc_country_region': {...},
#  'nam_loc_gpe_admin1': {...},
#  'nam_loc_gpe_admin2': {...},
#  'nam_loc_gpe_admin3': {...},
#  'nam_loc_gpe_city': {...},
#  'nam_loc_gpe_conurbation': {...},
#  'nam_loc_gpe_country': {...},
#  'nam_loc_gpe_district': {...},
#  'nam_loc_gpe_subdivision': {...},
#  'nam_loc_historical_region': {...},
#  'nam_loc_hydronym': {...},
#  'nam_loc_hydronym_lake': {...},
#  'nam_loc_hydronym_ocean': {...},
#  'nam_loc_hydronym_river': {...},
#  'nam_loc_hydronym_sea': {...},
#  'nam_loc_land': {...},
#  'nam_loc_land_continent': {...},
#  'nam_loc_land_island': {...},
#  'nam_loc_land_mountain': {...},
#  'nam_loc_land_peak': {...},
#  'nam_loc_land_region': {...},
#  'nam_num_house': {...},
#  'nam_num_phone': {...},
#  'nam_org_company': {...},
#  'nam_org_group': {...},
#  'nam_org_group_band': {...},
#  'nam_org_group_team': {...},
#  'nam_org_institution': {...},
#  'nam_org_nation': {...},
#  'nam_org_organization': {...},
#  'nam_org_organization_sub': {...},
#  'nam_org_political_party': {...},
#  'nam_oth': {...},
#  'nam_oth_currency': {...},
#  'nam_oth_data_format': {...},
#  'nam_oth_license': {...},
#  'nam_oth_position': {...},
#  'nam_oth_tech': {...},
#  'nam_oth_www': {...},
#  'nam_pro': {...},
#  'nam_pro_award': {...},
#  'nam_pro_brand': {...},
#  'nam_pro_media': {...},
#  'nam_pro_media_periodic': {...},
#  'nam_pro_media_radio': {...},
#  'nam_pro_media_tv': {...},
#  'nam_pro_media_web': {...},
#  'nam_pro_model_car': {...},
#  'nam_pro_software': {...},
#  'nam_pro_software_game': {...},
#  'nam_pro_title': {...},
#  'nam_pro_title_album': {...},
#  'nam_pro_title_article': {...},
#  'nam_pro_title_book': {...},
#  'nam_pro_title_document': {...},
#  'nam_pro_title_song': {...},
#  'nam_pro_title_treaty': {...},
#  'nam_pro_title_tv': {...},
#  'nam_pro_vehicle': {...},
#  'overall_accuracy': 0.006156203762418094,
#  'overall_f1': 0.0009844258777797407,
#  'overall_precision': 0.0005213624939842789,
#  'overall_recall': 0.008803611738148984}