SCORE Claim Identification

This is a model card for detecting claims from an abstract of social science publications. The model takes an abstract, performs sentence tokenization, and predict a claim probability of each sentence. This model card is released by training on a SCORE dataset. It achieves the following results on the test set:

• Accuracy: 0.931597
• Precision: 0.764563
• Recall: 0.722477
• F1: 0.742925

Model Usage

You can access the model with huggingface's transformers as follows:

import spacy
from transformers import AutoTokenizer
from transformers import AutoModelForSequenceClassification

nlp = spacy.load("en_core_web_lg")
model_name = "biodatlab/score-claim-identification"
tokenizer_name = "allenai/scibert_scivocab_uncased"

tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

def inference(abstract: str):
    """
    Split an abstract into sentences and perform claim identification.
    """
    if abstract.strip() == "":
        return "Please provide an abstract as an input."
    claims = []
    sents = [sent.text for sent in nlp(abstract).sents]  # a list of sentences
    inputs = tokenizer(
        sents,
        return_tensors="pt",
        truncation=True,
        padding="longest"
    )
    logits = model(**inputs).logits
    preds = logits.argmax(dim=1)  # convert logits to predictions
    claims = [sent for sent, pred in zip(sents, preds) if pred == 1]
    if len(claims) > 0:
        return ".\n".join(claims)
    else:
        return "No claims found from a given abstract."

claims = inference(abstract)  # string of claim joining with \n

Intended usage

Takes in a statement and classifies as Claim (1) or Null (0). Here are some examples -

Training procedure

Training Hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 32
• eval_batch_size: 32
• n_epochs: 6

Training results

Framework versions

• transformers 4.28.0
• sentence-transformers 2.2.2
• accelerate 0.19.0
• datasets 2.12.0
• spacy 3.5.3

See more on gradio application in biodatlab space.