Dataset Card for "LexGLUE"

Dataset Summary

Inspired by the recent widespread use of the GLUE multi-task benchmark NLP dataset (Wang et al., 2018), the subsequent more difficult SuperGLUE (Wang et al., 2019), other previous multi-task NLP benchmarks (Conneau and Kiela, 2018; McCann et al., 2018), and similar initiatives in other domains (Peng et al., 2019), we introduce the Legal General Language Understanding Evaluation (LexGLUE) benchmark , a benchmark dataset to evaluate the performance of NLP methods in legal tasks. LexGLUE is based on seven existing legal NLP datasets, selected using criteria largely from SuperGLUE.

As in GLUE and SuperGLUE (Wang et al., 2019b,a), one of our goals is to push towards generic (or ‘foundation’) models that can cope with multiple NLP tasks, in our case legal NLP tasks possibly with limited task-specific fine-tuning. Another goal is to provide a convenient and informative entry point for NLP researchers and practitioners wishing to explore or develop methods for legalNLP. Having these goals in mind, the datasets we include in LexGLUE and the tasks they address have been simplified in several ways to make it easier for newcomers and generic models to address all tasks.

LexGLUE benchmark is accompanied by experimental infrastructure that relies on Hugging Face Transformers library and resides at: https://github.com/coastalcph/lex-glue .

Supported Tasks and Leaderboards

The supported tasks are the following:

Dataset	Source	Sub-domain	Task Type	Classes
ECtHR (Task A)	Chalkidis et al. (2019)	ECHR	Multi-label classification	10+1
ECtHR (Task B)	Chalkidis et al. (2021a)	ECHR	Multi-label classification	10+1
SCOTUS	Spaeth et al. (2020)	US Law	Multi-class classification	14
EUR-LEX	Chalkidis et al. (2021b)	EU Law	Multi-label classification	100
LEDGAR	Tuggener et al. (2020)	Contracts	Multi-class classification	100
UNFAIR-ToS	Lippi et al. (2019)	Contracts	Multi-label classification	8+1
CaseHOLD	Zheng et al. (2021)	US Law	Multiple choice QA	n/a

ecthr_a

The European Court of Human Rights (ECtHR) hears allegations that a state has breached human rights provisions of the European Convention of Human Rights (ECHR). For each case, the dataset provides a list of factual paragraphs (facts) from the case description. Each case is mapped to articles of the ECHR that were violated (if any).

ecthr_b

The European Court of Human Rights (ECtHR) hears allegations that a state has breached human rights provisions of the European Convention of Human Rights (ECHR). For each case, the dataset provides a list of factual paragraphs (facts) from the case description. Each case is mapped to articles of ECHR that were allegedly violated (considered by the court).

scotus

The US Supreme Court (SCOTUS) is the highest federal court in the United States of America and generally hears only the most controversial or otherwise complex cases which have not been sufficiently well solved by lower courts. This is a single-label multi-class classification task, where given a document (court opinion), the task is to predict the relevant issue areas. The 14 issue areas cluster 278 issues whose focus is on the subject matter of the controversy (dispute).

eurlex

European Union (EU) legislation is published in EUR-Lex portal. All EU laws are annotated by EU's Publications Office with multiple concepts from the EuroVoc thesaurus, a multilingual thesaurus maintained by the Publications Office. The current version of EuroVoc contains more than 7k concepts referring to various activities of the EU and its Member States (e.g., economics, health-care, trade). Given a document, the task is to predict its EuroVoc labels (concepts).

ledgar

LEDGAR dataset aims contract provision (paragraph) classification. The contract provisions come from contracts obtained from the US Securities and Exchange Commission (SEC) filings, which are publicly available from EDGAR. Each label represents the single main topic (theme) of the corresponding contract provision.

unfair_tos

The UNFAIR-ToS dataset contains 50 Terms of Service (ToS) from on-line platforms (e.g., YouTube, Ebay, Facebook, etc.). The dataset has been annotated on the sentence-level with 8 types of unfair contractual terms (sentences), meaning terms that potentially violate user rights according to the European consumer law.

case_hold

The CaseHOLD (Case Holdings on Legal Decisions) dataset includes multiple choice questions about holdings of US court cases from the Harvard Law Library case law corpus. Holdings are short summaries of legal rulings accompany referenced decisions relevant for the present case. The input consists of an excerpt (or prompt) from a court decision, containing a reference to a particular case, while the holding statement is masked out. The model must identify the correct (masked) holding statement from a selection of five choices.

The current leaderboard includes several Transformer-based (Vaswaniet al., 2017) pre-trained language models, which achieve state-of-the-art performance in most NLP tasks (Bommasani et al., 2021) and NLU benchmarks (Wang et al., 2019a). Results reported by Chalkidis et al. (2021) :

Task-wise Test Results

Dataset	ECtHR A	ECtHR B	SCOTUS	EUR-LEX	LEDGAR	UNFAIR-ToS	CaseHOLD
Model	μ-F1 / m-F1	μ-F1 / m-F1	μ-F1 / m-F1	μ-F1 / m-F1	μ-F1 / m-F1	μ-F1 / m-F1	μ-F1 / m-F1
TFIDF+SVM	64.7 / 51.7	74.6 / 65.1	78.2 / 69.5	71.3 / 51.4	87.2 / 82.4	95.4 / 78.8	n/a
Medium-sized Models (L=12, H=768, A=12)
BERT	71.2 / 63.6	79.7 / 73.4	68.3 / 58.3	71.4 / 57.2	87.6 / 81.8	95.6 / 81.3	70.8
RoBERTa	69.2 / 59.0	77.3 / 68.9	71.6 / 62.0	71.9 / 57.9	87.9 / 82.3	95.2 / 79.2	71.4
DeBERTa	70.0 / 60.8	78.8 / 71.0	71.1 / 62.7	72.1 / 57.4	88.2 / 83.1	95.5 / 80.3	72.6
Longformer	69.9 / 64.7	79.4 / 71.7	72.9 / 64.0	71.6 / 57.7	88.2 / 83.0	95.5 / 80.9	71.9
BigBird	70.0 / 62.9	78.8 / 70.9	72.8 / 62.0	71.5 / 56.8	87.8 / 82.6	95.7 / 81.3	70.8
Legal-BERT	70.0 / 64.0	80.4 / 74.7	76.4 / 66.5	72.1 / 57.4	88.2 / 83.0	96.0 / 83.0	75.3
CaseLaw-BERT	69.8 / 62.9	78.8 / 70.3	76.6 / 65.9	70.7 / 56.6	88.3 / 83.0	96.0 / 82.3	75.4
Large-sized Models (L=24, H=1024, A=18)
RoBERTa	73.8 / 67.6	79.8 / 71.6	75.5 / 66.3	67.9 / 50.3	88.6 / 83.6	95.8 / 81.6	74.4

Averaged (Mean over Tasks) Test Results

Averaging	Arithmetic	Harmonic	Geometric
Model	μ-F1 / m-F1	μ-F1 / m-F1	μ-F1 / m-F1
Medium-sized Models (L=12, H=768, A=12)
BERT	77.8 / 69.5	76.7 / 68.2	77.2 / 68.8
RoBERTa	77.8 / 68.7	76.8 / 67.5	77.3 / 68.1
DeBERTa	78.3 / 69.7	77.4 / 68.5	77.8 / 69.1
Longformer	78.5 / 70.5	77.5 / 69.5	78.0 / 70.0
BigBird	78.2 / 69.6	77.2 / 68.5	77.7 / 69.0
Legal-BERT	79.8 / 72.0	78.9 / 70.8	79.3 / 71.4
CaseLaw-BERT	79.4 / 70.9	78.5 / 69.7	78.9 / 70.3
Large-sized Models (L=24, H=1024, A=18)
RoBERTa	79.4 / 70.8	78.4 / 69.1	78.9 / 70.0

Languages

We only consider English datasets, to make experimentation easier for researchers across the globe.

Dataset Structure

Data Instances

ecthr_a

An example of 'train' looks as follows.

{
  "text": ["8. The applicant was arrested in the early morning of 21 October 1990 ...", ...],
  "labels": [6]
}

ecthr_b

An example of 'train' looks as follows.

{
  "text": ["8. The applicant was arrested in the early morning of 21 October 1990 ...", ...],
  "label": [5, 6]
}

scotus

An example of 'train' looks as follows.

{
  "text": "Per Curiam\nSUPREME COURT OF THE UNITED STATES\nRANDY WHITE, WARDEN v. ROGER L. WHEELER\n Decided December 14, 2015\nPER CURIAM.\nA death sentence imposed by a Kentucky trial court and\naffirmed by the ...",
  "label": 8
}

eurlex

An example of 'train' looks as follows.

{
  "text": "COMMISSION REGULATION (EC) No 1629/96 of 13 August 1996 on an invitation to tender for the refund on export of wholly milled round grain rice to certain third countries ...",
  "labels": [4, 20, 21, 35, 68]
}

ledgar

An example of 'train' looks as follows.

{
  "text": "All Taxes shall be the financial responsibility of the party obligated to pay such Taxes as determined by applicable law and neither party is or shall be liable at any time for any of the other party ...",
  "label": 32
}

unfair_tos

An example of 'train' looks as follows.

{
  "text": "tinder may terminate your account at any time without notice if it believes that you have violated this agreement.",
  "label": 2
}

casehold

An example of 'test' looks as follows.

{
  "context": "In Granato v. City and County of Denver, No. CIV 11-0304 MSK/BNB, 2011 WL 3820730 (D.Colo. Aug. 20, 2011), the Honorable Marcia S. Krieger, now-Chief United States District Judge for the District of Colorado, ruled similarly: At a minimum, a party asserting a Mo-nell claim must plead sufficient facts to identify ... to act pursuant to City or State policy, custom, decision, ordinance, re d 503, 506-07 (3d Cir.l985)(<HOLDING>).",
  "endings": ["holding that courts are to accept allegations in the complaint as being true including monell policies and writing that a federal court reviewing the sufficiency of a complaint has a limited task",
    "holding that for purposes of a class certification motion the court must accept as true all factual allegations in the complaint and may draw reasonable inferences therefrom", 
    "recognizing that the allegations of the complaint must be accepted as true on a threshold motion to dismiss", 
    "holding that a court need not accept as true conclusory allegations which are contradicted by documents referred to in the complaint", 
    "holding that where the defendant was in default the district court correctly accepted the fact allegations of the complaint as true"
  ],
  "label": 0
}

Data Fields

ecthr_a

• text : a list of string features (list of factual paragraphs (facts) from the case description).
• labels : a list of classification labels (a list of violated ECHR articles, if any) . List of ECHR articles "Article 2", "Article 3", "Article 5", "Article 6", "Article 8", "Article 9", "Article 10", "Article 11", "Article 14", "Article 1 of Protocol 1"

ecthr_b

• text : a list of string features (list of factual paragraphs (facts) from the case description)
• labels : a list of classification labels (a list of articles considered). List of ECHR articles "Article 2", "Article 3", "Article 5", "Article 6", "Article 8", "Article 9", "Article 10", "Article 11", "Article 14", "Article 1 of Protocol 1"

scotus

• text : a string feature (the court opinion).
• label : a classification label (the relevant issue area). List of issue areas (1, Criminal Procedure), (2, Civil Rights), (3, First Amendment), (4, Due Process), (5, Privacy), (6, Attorneys), (7, Unions), (8, Economic Activity), (9, Judicial Power), (10, Federalism), (11, Interstate Relations), (12, Federal Taxation), (13, Miscellaneous), (14, Private Action)

eurlex

• text : a string feature (an EU law).
• labels : a list of classification labels (a list of relevant EUROVOC concepts). List of EUROVOC concepts The list is very long including 100 EUROVOC concepts. You can find the EUROVOC concepts descriptors here .

ledgar

• text : a string feature (a contract provision/paragraph).
• label : a classification label (the type of contract provision). List of contract provision types "Adjustments", "Agreements", "Amendments", "Anti-Corruption Laws", "Applicable Laws", "Approvals", "Arbitration", "Assignments", "Assigns", "Authority", "Authorizations", "Base Salary", "Benefits", "Binding Effects", "Books", "Brokers", "Capitalization", "Change In Control", "Closings", "Compliance With Laws", "Confidentiality", "Consent To Jurisdiction", "Consents", "Construction", "Cooperation", "Costs", "Counterparts", "Death", "Defined Terms", "Definitions", "Disability", "Disclosures", "Duties", "Effective Dates", "Effectiveness", "Employment", "Enforceability", "Enforcements", "Entire Agreements", "Erisa", "Existence", "Expenses", "Fees", "Financial Statements", "Forfeitures", "Further Assurances", "General", "Governing Laws", "Headings", "Indemnifications", "Indemnity", "Insurances", "Integration", "Intellectual Property", "Interests", "Interpretations", "Jurisdictions", "Liens", "Litigations", "Miscellaneous", "Modifications", "No Conflicts", "No Defaults", "No Waivers", "Non-Disparagement", "Notices", "Organizations", "Participations", "Payments", "Positions", "Powers", "Publicity", "Qualifications", "Records", "Releases", "Remedies", "Representations", "Sales", "Sanctions", "Severability", "Solvency", "Specific Performance", "Submission To Jurisdiction", "Subsidiaries", "Successors", "Survival", "Tax Withholdings", "Taxes", "Terminations", "Terms", "Titles", "Transactions With Affiliates", "Use Of Proceeds", "Vacations", "Venues", "Vesting", "Waiver Of Jury Trials", "Waivers", "Warranties", "Withholdings",

unfair_tos

• text : a string feature (a ToS sentence)
• labels : a list of classification labels (a list of unfair types, if any). List of unfair types "Limitation of liability", "Unilateral termination", "Unilateral change", "Content removal", "Contract by using", "Choice of law", "Jurisdiction", "Arbitration"

casehold

• context : a string feature (a context sentence incl. a masked holding statement).
• holdings : a list of string features (a list of candidate holding statements).
• label : a classification label (the id of the original/correct holding).

Data Splits

Dataset	Training	Development	Test	Total
ECtHR (Task A)	9,000	1,000	1,000	11,000
ECtHR (Task B)	9,000	1,000	1,000	11,000
SCOTUS	5,000	1,400	1,400	7,800
EUR-LEX	55,000	5,000	5,000	65,000
LEDGAR	60,000	10,000	10,000	80,000
UNFAIR-ToS	5,532	2,275	1,607	9,414
CaseHOLD	45,000	3,900	3,900	52,800

Dataset Creation

Curation Rationale

More Information Needed

Source Data

Dataset	Source	Sub-domain	Task Type
ECtHR (Task A)	Chalkidis et al. (2019)	ECHR	Multi-label classification
ECtHR (Task B)	Chalkidis et al. (2021a)	ECHR	Multi-label classification
SCOTUS	Spaeth et al. (2020)	US Law	Multi-class classification
EUR-LEX	Chalkidis et al. (2021b)	EU Law	Multi-label classification
LEDGAR	Tuggener et al. (2020)	Contracts	Multi-class classification
UNFAIR-ToS	Lippi et al. (2019)	Contracts	Multi-label classification
CaseHOLD	Zheng et al. (2021)	US Law	Multiple choice QA

Initial Data Collection and Normalization

More Information Needed

Who are the source language producers?

More Information Needed

Annotations

Annotation process

More Information Needed

Who are the annotators?

More Information Needed

Personal and Sensitive Information

More Information Needed

Considerations for Using the Data

Social Impact of Dataset

More Information Needed

Discussion of Biases

More Information Needed

Other Known Limitations

More Information Needed

Additional Information

More Information Needed

Dataset Curators

Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. 2022. In the Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Dublin, Ireland.

Licensing Information

More Information Needed

Citation Information

Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. 2022. In the Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Dublin, Ireland.

@inproceedings{chalkidis-etal-2021-lexglue,
        title={LexGLUE: A Benchmark Dataset for Legal Language Understanding in English}, 
        author={Chalkidis, Ilias and Jana, Abhik and Hartung, Dirk and
        Bommarito, Michael and Androutsopoulos, Ion and Katz, Daniel Martin and
        Aletras, Nikolaos},
        year={2022},
        booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics},
        address={Dubln, Ireland},
}

Contributions

Thanks to @iliaschalkidis for adding this dataset.