Dataset Card for CLEVR-Math

Dataset Summary

Dataset for compositional multimodal mathematical reasoning based on CLEVR.

from transformers import CLIPPreprocessor
from datasets import load_dataset, DownloadConfig


dl_config = DownloadConfig(resume_download=True,
        num_proc=8,
        force_download=True)

# Load 'general' instance of dataset
dataset = load_dataset('dali-does/clevr-math', download_config=dl_config)

# Load version with only multihop in test data
dataset_multihop = load_dataset('dali-does/clevr-math', 'multihop',
                                download_config=dl_config)


model_path = "openai/clip-vit-base-patch32"
extractor = CLIPProcessor.from_pretrained(model_path)
def transform_tokenize(e):
  e['image'] = [image.convert('RGB') for image in e['image']]
  return extractor(text=e['question'],
                           images=e['image'],
                           padding=True)

dataset = dataset.map(transform_tokenize,
                      batched=True,
                      num_proc=8,
                      padding='max_length')

dataset_subtraction = dataset.filter(lambda e:
        e['template'].startswith('subtraction'), num_proc=4)

Supported Tasks and Leaderboards

Leaderboard will be announced at a later date.

Languages

The dataset is currently only available in English. To extend the dataset to other languages, the CLEVR templates must be rewritten in the target language.

Dataset Structure

Data Instances

• general containing the default version with multihop questions in train and test
• multihop containing multihop questions only in test data to test generalisation of reasoning

Data Fields

features = datasets.Features(
  {
    "template": datasets.Value("string"),
    "id": datasets.Value("string"),
    "question": datasets.Value("string"),
    "image": datasets.Image(),
    "label": datasets.Value("int64")
  }
)

Data Splits

train/val/test

Dataset Creation

Data is generated using code provided with the CLEVR-dataset, using blender and templates constructed by the dataset curators.

Considerations for Using the Data

Other Known Limitations

[More Information Needed]

Additional Information

Dataset Curators

Adam Dahlgren Lindström - dali@cs.umu.se

Licensing Information

Licensed under Creative Commons Attribution Share Alike 4.0 International (CC-by 4.0).

Citation Information

[More Information Needed]

@misc{https://doi.org/10.48550/arxiv.2208.05358,
  doi = {10.48550/ARXIV.2208.05358},
  url = {https://arxiv.org/abs/2208.05358},
  author = {Lindström, Adam Dahlgren and Abraham, Savitha Sam},
  keywords = {Machine Learning (cs.LG), Computation and Language (cs.CL), Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences, I.2.7; I.2.10; I.2.6; I.4.8; I.1.4},
  title = {CLEVR-Math: A Dataset for Compositional Language, Visual, and Mathematical Reasoning},
  publisher = {arXiv},
  year = {2022},
  copyright = {Creative Commons Attribution Share Alike 4.0 International}
}

Contributions

Thanks to @dali-does for adding this dataset.