Document Understanding model (finetuned LiLT base at line level on DocLayNet base)

This model is a fine-tuned version of nielsr/lilt-xlm-roberta-base with the DocLayNet base dataset. It achieves the following results on the evaluation set:

• Loss: 1.0003
• Precision: 0.8584
• Recall: 0.8584
• F1: 0.8584
• Tokens Accuracy: 0.8584
• Line Accuracy: 0.9197

Accuracy at line level

• Line Accuracy: 91.97%
• Accuracy by label
  • Caption: 79.42%
  • Footnote: 68.21%
  • Formula: 98.02%
  • List-item: 82.72%
  • Page-footer: 99.17%
  • Page-header: 84.18%
  • Picture: 83.2%
  • Section-header: 76.92%
  • Table: 97.65%
  • Text: 91.17%
  • Title: 77.46%

References

Blog posts

• Layout XLM base
  • (03/05/2023) Document AI | Inference APP and fine-tuning notebook for Document Understanding at line level with LayoutXLM base
• LiLT base
  • (02/16/2023) Document AI | Inference APP and fine-tuning notebook for Document Understanding at paragraph level
  • (02/14/2023) Document AI | Inference APP for Document Understanding at line level
  • (02/10/2023) Document AI | Document Understanding model at line level with LiLT, Tesseract and DocLayNet dataset
  • (01/31/2023) Document AI | DocLayNet image viewer APP
  • (01/27/2023) Document AI | Processing of DocLayNet dataset to be used by layout models of the Hugging Face hub (finetuning, inference)

Notebooks (paragraph level)

• LiLT base
  • Document AI | Inference APP at paragraph level with a Document Understanding model (LiLT fine-tuned on DocLayNet dataset)
  • Document AI | Inference at paragraph level with a Document Understanding model (LiLT fine-tuned on DocLayNet dataset)
  • Document AI | Fine-tune LiLT on DocLayNet base in any language at paragraph level (chunk of 512 tokens with overlap)

Notebooks (line level)

• Layout XLM base
  • Document AI | Inference at line level with a Document Understanding model (LayoutXLM base fine-tuned on DocLayNet dataset)
  • Document AI | Inference APP at line level with a Document Understanding model (LayoutXLM base fine-tuned on DocLayNet base dataset)
  • Document AI | Fine-tune LayoutXLM base on DocLayNet base in any language at line level (chunk of 384 tokens with overlap)
• LiLT base
  • Document AI | Inference at line level with a Document Understanding model (LiLT fine-tuned on DocLayNet dataset)
  • Document AI | Inference APP at line level with a Document Understanding model (LiLT fine-tuned on DocLayNet dataset)
  • Document AI | Fine-tune LiLT on DocLayNet base in any language at line level (chunk of 384 tokens with overlap)
  • DocLayNet image viewer APP
  • Processing of DocLayNet dataset to be used by layout models of the Hugging Face hub (finetuning, inference)

APP

You can test this model with this APP in Hugging Face Spaces: Inference APP for Document Understanding at line level (v1) .

DocLayNet dataset

DocLayNet dataset (IBM) provides page-by-page layout segmentation ground-truth using bounding-boxes for 11 distinct class labels on 80863 unique pages from 6 document categories.

Until today, the dataset can be downloaded through direct links or as a dataset from Hugging Face datasets:

• direct links: doclaynet_core.zip (28 GiB), doclaynet_extra.zip (7.5 GiB)
• Hugging Face dataset library: dataset DocLayNet

Paper: DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis (06/02/2022)

Model description

The model was finetuned at line level on chunk of 384 tokens with overlap of 128 tokens . Thus, the model was trained with all layout and text data of all pages of the dataset.

At inference time, a calculation of best probabilities give the label to each line bounding boxes.

Inference

See notebook: Document AI | Inference at line level with a Document Understanding model (LiLT fine-tuned on DocLayNet dataset)

Training and evaluation data

See notebook: Document AI | Fine-tune LiLT on DocLayNet base in any language at line level (chunk of 384 tokens with overlap)

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 5
• mixed_precision_training: Native AMP

Training results

Framework versions

• Transformers 4.26.0
• Pytorch 1.13.1+cu116
• Datasets 2.9.0
• Tokenizers 0.13.2

Other models

• Line level
  • Document Understanding model (finetuned LiLT base at line level on DocLayNet base) (accuracy | tokens: 85.84% - lines: 91.97%)
  • Document Understanding model (finetuned LayoutXLM base at line level on DocLayNet base) (accuracy | tokens: 93.73% - lines: ...)
• Paragraph level
  • Document Understanding model (finetuned LiLT base at paragraph level on DocLayNet base) (accuracy | tokens: 86.34% - paragraphs: 68.15%)
  • Document Understanding model (finetuned LayoutXLM base at paragraph level on DocLayNet base) (accuracy | tokens: 96.93% - paragraphs: 86.55%)