Model Card for COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

Model Details

Model Description

This model card contains the COCO-LM model ( base++ version) pretrained models on GLUE and SQuAD 2.0 benchmarks.

• Developed by: Microsoft
• Shared by [Optional]: HuggingFace
• Model type: Language model
• Language(s) (NLP): en
• License: MIT
• Related Models: More information needed
  • Parent Model: More information needed
• Resources for more information:
• GitHub Repo
• Associated Paper

Uses

Direct Use

Correcting and Contrasting Text Sequences for Language Model Pretraining

Downstream Use [Optional]

More information needed.

Out-of-Scope Use

The model should not be used to intentionally create hostile or alienating environments for people.

Bias, Risks, and Limitations

Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021) ). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recomendations.

Training Details

Training Data

See the associated dataset card for further details

Training Procedure

Preprocessing

The model devloeprs note in the associated paper :

We employ three standard settings, base, base++, and large++. Base is the BERTBase training configuration: Pretraining on Wikipedia and BookCorpus (16 GB of texts) for 256 million samples on 512 token sequences (125K batches with 2048 batch size). We use the same corpus and 32, 768 uncased BPE vocabulary as with TUPE. Base++ trains the base size model with larger corpora and/or more training steps. Following recent research, we add in OpenWebText, CC-News , and STORIES, to a total of 160 GB texts, and train for 4 billion (with 2048 batch size) samples. We follow the prepossessing of UniLMV2 and use 64, 000 cased BPE vocabulary. Large++ uses the same training corpora as base++ and pretrains for 4 billion samples (2048 batch size). Its Transformer configuration is the same with BERTLarge.

Speeds, Sizes, Times

More information needed

Evaluation

Testing Data, Factors & Metrics

Testing Data

GLUE SQuAD 2.0

Factors

All results are single-task, single-model fine-tuning.

Metrics

More information needed

Results

GLUE Fine-Tuning Results

The General Language Understanding Evaluation (GLUE) benchmark is a collection of sentence- or sentence-pair language understanding tasks for evaluating and analyzing natural language understanding systems.

GLUE dev set results of COCO-LM base++ and large++ models are as follows (median of 5 different random seeds):

GLUE test set results of COCO-LM base++ and large++ models are as follows (no ensemble, task-specific tricks, etc.):

SQuAD 2.0 Fine-Tuning Results

Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.

SQuAD 2.0 dev set results of COCO-LM base++ and large++ models are as follows (median of 5 different random seeds):

Model Examination

The model delevopers note in the associated paper :

Architecture. Removing relative position encoding (Rel-Pos) leads to better numbers on some tasks but significantly hurts MNLI. Using a shallow auxiliary network and keeping the same hidden dimension (768) is more effective than ELECTRA’s 12-layer but 256-hidden dimension generator.

One limitation of this work is that the contrastive pairs are constructed by simple cropping and MLM replacements. Recent studies have shown the effectiveness of advanced data augmentation techniques in fine-tuning language models [16, 38, 51]. A future research direction is to explore better ways to construct contrastive pairs in language model pretraining.

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019) .

• Hardware Type: More information needed
• Hours used: More information needed
• Cloud Provider: More information needed
• Compute Region: More information needed
• Carbon Emitted: More information needed}

Technical Specifications [optional]

Model Architecture and Objective

The model delevopers note in the associated paper :

Model Architecture. Our base/base++ model uses the BERTBase architecture: 12 layer Transformer, 768 hidden size, plus T5 relative position encoding. Our large++ model is the same with BERTLarge, 24 layer and 1024 hidden size, plus T5 relative position encoding. Our auxiliary network uses the same hidden size but a shallow 4-layer Transformer in base/base++ and a 6-layer one in large++. When generating XMLM we disable dropout in the auxiliary model

Compute Infrastructure

More information needed

Hardware

More information needed

Software

More information needed

Citation

BibTeX:

If you find the code and models useful for your research, please cite the following paper:

@inproceedings{meng2021cocolm,
  title={{COCO-LM}: Correcting and contrasting text sequences for language model pretraining},
  author={Meng, Yu and Xiong, Chenyan and Bajaj, Payal and Tiwary, Saurabh and Bennett, Paul and Han, Jiawei and Song, Xia},
  booktitle={Conference on Neural Information Processing Systems},
  year={2021}
}

Glossary [optional]

More information needed

More Information [optional]

More information needed

Model Card Authors [optional]

Microsoft in collaboration with Ezi Ozoani and the HuggingFace team

Model Card Contact

More information needed

How to Get Started with the Model

Use the code below to get started with the model.

 
from transformers import AutoModel
 
model = AutoModel.from_pretrained("microsoft/cocolm-base")