模型:
ethzanalytics/dolly-v2-12b-sharded
This is a sharded checkpoint (with ~4GB shards) of the databricks/dolly-v2-12b model. Refer to the original model for all details.
install transformers , accelerate , and bitsandbytes .
pip install -U -q transformers bitsandbytes accelerate
Load the model in 8bit, then run inference :
from transformers import AutoTokenizer, AutoModelForCausalLM
model_name = "ethzanalytics/dolly-v2-12b-sharded"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name, load_in_8bit=True, device_map="auto",
)