SciVQA: Scientific Visual Question Answering Shared Task
Hosted as part of the SDP 2025 Workshop July 31 or August 1st, 2025 (tbc) Vienna, Austria (co-located with ACL 2025)
SciVQA Shared Task: https://sdproc.org/2025/scivqa.html SDP 2025 Workshop: https://sdproc.org/2025/index.html
Task Overview
Scholarly articles convey valuable information not only through unstructured text but also via (semi-)structured figures such as charts and diagrams. Automatically interpreting the semantics of knowledge encoded in these figures can be beneficial for downstream tasks such as question answering (QA).
In the SciVQA challenge, participants will develop multimodal QA systems using a dataset of scientific figures from ACL Anthology and arXiv papers. Each figure image is annotated with seven QA pairs and includes metadata such as caption, figure ID, figure type (e.g., compound, line graph, bar chart, scatter plot, etc.), QA pair type. This shared task specifically focuses on closed-ended visual (i.e., addressing visual attributes of a figure such as colour, shape, size, height, etc.) and non-visual (not addressing figure visual attributes) questions.
Evaluation
Systems will be evaluated using metrics such as BLEU, METEOR, and ROUGE. Automated evaluations of submitted systems will be done through the Codabench platform (link will be provided soon on the webpage).
Important Dates
Release of training data: April 1, 2025 Release of testing data: April 15, 2025 Deadline for system submissions: May 16, 2025 Paper submission deadline: May 23, 2025 Notification of acceptance: June 13, 2025 Camera-ready paper due: June 20, 2025 Workshop: July 31, 2025 or August 1, 2025 (TBA) Participants are also invited to submit papers on their systems. Successful submissions will be published in the proceedings of the SDP 2025 workshop.
Organizers
Ekaterina Borisova (DFKI, Berlin, Germany)
Georg Rehm (DFKI, Berlin, Germany)