Dataset Card for Aksharantar

Dataset Summary

Bhasha-Abhijnaanam is a language identification test set for native-script as well as Romanized text which spans 22 Indic languages.

Supported Tasks and Leaderboards

[More Information Needed]

Languages

Dataset Structure

Data Instances

A random sample from Hindi (hin) Test dataset.
{
    "unique_identifier": "hin1", 
    "native sentence": "",
    "romanized sentence": "",
    "language": "Hindi", 
    "script": "Devanagari", 
    "source": "Dakshina",
}

Data Fields

• unique_identifier (string): 3-letter language code followed by a unique number in Test set.

• native sentence (string): A sentence in Indic language.

• romanized sentence (string): Transliteration of native sentence in English (Romanized sentence).

• language (string): Language of native sentence.

• script (string): Script in which native sentence is written.

• source (string): Source of the data.

  For created data sources, depending on the destination/sampling method of a pair in a language, it will be one of:

  • Dakshina Dataset
  • Flores-200
  • Manually Romanized
  • Manually generated

Data Splits

Dataset Creation

Information in the paper. Bhasha-Abhijnaanam: Native-script and romanized Language Identification for 22 Indic languages

Curation Rationale

[More Information Needed]

Source Data

Information in the paper. Bhasha-Abhijnaanam: Native-script and romanized Language Identification for 22 Indic languages

[More Information Needed]

Annotations

Information in the paper. Bhasha-Abhijnaanam: Native-script and romanized Language Identification for 22 Indic languages

Information in the paper. Bhasha-Abhijnaanam: Native-script and romanized Language Identification for 22 Indic languages

Considerations for Using the Data

Social Impact of Dataset

[More Information Needed]

Discussion of Biases

[More Information Needed]

Other Known Limitations

[More Information Needed]

Additional Information

Dataset Curators

[More Information Needed]

Licensing Information

This data is released under the following licensing scheme:

• Manually collected data: Released under CC0 license.

CC0 License Statement

• We do not own any of the text from which this data has been extracted.
• We license the actual packaging of manually collected data under the Creative Commons CC0 license (“no rights reserved”) .
• To the extent possible under law, AI4Bharat has waived all copyright and related or neighboring rights to Aksharantar manually collected data and existing sources.
• This work is published from: India.

Citation Information

@misc{madhani2023bhashaabhijnaanam,
      title={Bhasha-Abhijnaanam: Native-script and romanized Language Identification for 22 Indic languages}, 
      author={Yash Madhani and Mitesh M. Khapra and Anoop Kunchukuttan},
      year={2023},
      eprint={2305.15814},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Contributions