SciVQA: Scientific Visual Question Answering Shared Task

Hosted as part of the SDP 2025 Workshop at ACL 2025 31 July 2025 Vienna, Austria

SciVQA Shared Task: https://sdproc.org/2025/scivqa.html Competition on Codabench: https://www.codabench.org/competitions/5904

Shared Task Overview

Scholarly articles convey valuable information not only through unstructured text but also via (semi-)structured figures such as charts and diagrams. Automatically interpreting the semantics of knowledge encoded in these figures can be beneficial for downstream tasks such as question answering (QA). In SciVQA, challenge participants will develop multimodal QA systems using a dataset of 3000 images of scientific figures from ACL Anthology and arXiv papers. Each figure is annotated with seven QA pairs (21000 in total) and includes metadata such as caption, ID, type (e.g., compound, line graph, bar chart, scatter plot), and QA pair type. This shared task specifically focuses on closed-ended visual (i.e., addressing visual attributes such as colour, shape, size, height, etc.) and non-visual (not addressing figure visual attributes) questions.

Evaluation

Systems will be evaluated using BERTscore, ROUGE-L, and ROUGE-1 metrics. Automated evaluations of submitted systems will be performed through Codabench.

Important Dates

Release of training data: April 1, 2025 Release of testing data: April 15, 2025 Deadline for system submissions: May 16, 2025 Paper submission deadline: May 23, 2025 Notification of acceptance: June 13, 2025 Camera-ready paper due: June 20, 2025 Workshop: July 31, 2025

Participants are invited to submit papers on their systems. Successful submissions will be published in the proceedings of the SDP 2025 workshop.

Organizers

Ekaterina Borisova (DFKI, Berlin, Germany) Georg Rehm (DFKI, Berlin, Germany)