Document Understanding model (finetuned LayoutXLM base at paragraph level on DocLayNet base)

This model is a fine-tuned version of microsoft/layoutxlm-base with the DocLayNet base dataset. It achieves the following results on the evaluation set:

• Loss: 0.1796
• Precision: 0.8062
• Recall: 0.7441
• F1: 0.7739
• Token Accuracy: 0.9693
• Paragraph Accuracy: 0.8655

Accuracy at paragraph level

• Paragraph Accuracy: 86.55%
• Accuracy by label
  • Caption: 63.76%
  • Footnote: 31.91%
  • Formula: 95.33%
  • List-item: 79.31%
  • Page-footer: 99.51%
  • Page-header: 88.75%
  • Picture: 90.91%
  • Section-header: 83.16%
  • Table: 68.25%
  • Text: 91.37%
  • Title: 50.0%

References

Blog posts

• Layout XLM base
  • (03/31/2023) Document AI | Inference APP and fine-tuning notebook for Document Understanding at paragraph level with LayoutXLM base
  • (03/25/2023) Document AI | APP to compare the Document Understanding LiLT and LayoutXLM (base) models at line level
  • (03/05/2023) Document AI | Inference APP and fine-tuning notebook for Document Understanding at line level with LayoutXLM base
• LiLT base - (02/16/2023) Document AI | Inference APP and fine-tuning notebook for Document Understanding at paragraph level - (02/14/2023) Document AI | Inference APP for Document Understanding at line level - (02/10/2023) Document AI | Document Understanding model at line level with LiLT, Tesseract and DocLayNet dataset - (01/31/2023) Document AI | DocLayNet image viewer APP - (01/27/2023) Document AI | Processing of DocLayNet dataset to be used by layout models of the Hugging Face hub (finetuning, inference)

Notebooks (paragraph level)

• Layout XLM base
  • Document AI | Inference at paragraph level with a Document Understanding model (LayoutXLM base fine-tuned on DocLayNet dataset)
  • Document AI | Inference APP at paragraph level with a Document Understanding model (LayoutXLM base fine-tuned on DocLayNet base dataset)
  • Document AI | Fine-tune LayoutXLM base on DocLayNet base in any language at paragraph level (chunk of 512 tokens with overlap)
• LiLT base
  • Document AI | Inference APP at paragraph level with a Document Understanding model (LiLT fine-tuned on DocLayNet dataset)
  • Document AI | Inference at paragraph level with a Document Understanding model (LiLT fine-tuned on DocLayNet dataset)
  • Document AI | Fine-tune LiLT on DocLayNet base in any language at paragraph level (chunk of 512 tokens with overlap)

Notebooks (line level)

• Layout XLM base
  • Document AI | Inference APP at line level with 2 Document Understanding models (LiLT and LayoutXLM base fine-tuned on DocLayNet base dataset)
  • Document AI | Inference at line level with a Document Understanding model (LayoutXLM base fine-tuned on DocLayNet dataset)
  • Document AI | Inference APP at line level with a Document Understanding model (LayoutXLM base fine-tuned on DocLayNet base dataset)
  • Document AI | Fine-tune LayoutXLM base on DocLayNet base in any language at line level (chunk of 384 tokens with overlap)
• LiLT base
  • Document AI | Inference at line level with a Document Understanding model (LiLT fine-tuned on DocLayNet dataset)
  • Document AI | Inference APP at line level with a Document Understanding model (LiLT fine-tuned on DocLayNet dataset)
  • Document AI | Fine-tune LiLT on DocLayNet base in any language at line level (chunk of 384 tokens with overlap)
  • DocLayNet image viewer APP
  • Processing of DocLayNet dataset to be used by layout models of the Hugging Face hub (finetuning, inference)

APP

You can test this model with this APP in Hugging Face Spaces: Inference APP for Document Understanding at paragraph level (v2) .

You can run as well the corresponding notebook: Document AI | Inference APP at paragraph level with a Document Understanding model (LayoutXLM base fine-tuned on DocLayNet dataset)

DocLayNet dataset

DocLayNet dataset (IBM) provides page-by-page layout segmentation ground-truth using bounding-boxes for 11 distinct class labels on 80863 unique pages from 6 document categories.

Until today, the dataset can be downloaded through direct links or as a dataset from Hugging Face datasets:

• direct links: doclaynet_core.zip (28 GiB), doclaynet_extra.zip (7.5 GiB)
• Hugging Face dataset library: dataset DocLayNet

Paper: DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis (06/02/2022)

Model description

The model was finetuned at paragraph level on chunk of 512 tokens with overlap of 128 tokens . Thus, the model was trained with all layout and text data of all pages of the dataset.

At inference time, a calculation of best probabilities give the label to each paragraph bounding boxes.

Inference

See notebook: Document AI | Inference at paragraph level with a Document Understanding model (LayoutXLM base fine-tuned on DocLayNet dataset)

Training and evaluation data

See notebook: Document AI | Fine-tune LayoutXLM base on DocLayNet base in any language at paragraph level (chunk of 512 tokens with overlap)

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 4
• mixed_precision_training: Native AMP

Training results

Framework versions

• Transformers 4.27.3
• Pytorch 1.10.0+cu111
• Datasets 2.10.1
• Tokenizers 0.13.2

Other models

• Line level
  • Document Understanding model (finetuned LiLT base at line level on DocLayNet base) (accuracy | tokens: 85.84% - lines: 91.97%)
  • Document Understanding model (finetuned LayoutXLM base at line level on DocLayNet base) (accuracy | tokens: 93.73% - lines: ...)
• Paragraph level
  • Document Understanding model (finetuned LiLT base at paragraph level on DocLayNet base) (accuracy | tokens: 86.34% - paragraphs: 68.15%)
  • Document Understanding model (finetuned LayoutXLM base at paragraph level on DocLayNet base) (accuracy | tokens: 96.93% - paragraphs: 86.55%)