This is a Hugging Face transformers-compatible conversion of the original dense 125M-parameter model from the paper " Efficient Large Scale Language Modeling with Mixtures of Experts " from Artetxe et al. Please refer to the original model card, which can be found at https://github.com/facebookresearch/fairseq/blob/main/examples/moe_lm/model_card.md .