ClimateCheck: Shared Task on Scientific Fact-Checking of Social Media Claims on Climate Change
Hosted as part of the SDP 2025 Workshop at ACL 2025 31 July 2025 Vienna, Austria
ClimateCheck Shared Task: https://sdproc.org/2025/climatecheck.html Competition on Codabench: https://www.codabench.org/competitions/6639/
Shared Task Overview
Social media facilitates discussions on critical issues such as climate change, but it also contributes to the rapid dissemination of misinformation, which complicates efforts to maintain an informed public and create evidence-based policies. In this shared task, we emphasise the need to link public discourse to peer-reviewed scholarly articles by gathering claims from social media about climate change (both real-life and automatically generated ones) as well as a corpus of about 400,000 abstracts of publications from climate science research. The participants will be asked to retrieve relevant abstracts for each claim (subtask I) and classify the relation between the claim and abstract as ‘supports’, ‘refutes’, or ‘not enough information’ (subtask II). Participants are allowed to take part either in subtask I only, or in both subtasks.
Subtask I: Abstracts Retrieval Task: given a claim from social media about climate change and a corpus of abstracts, retrieve the top 10 most relevant abstracts. Evaluation: Recall@K (K=2, 5, 10) and B-Pref based on annotated gold data; additional unjudged documents will be reviewed by annotators. Subtask II: Claim Verification Task: given the claim-abstract pairs received from the previous subtask, classify their relation as ‘supports’, ‘refutes’, or ‘not enough information’. Evaluation: Precision, Recall, and F1 weighted scores based on annotated documents.
Important dates
Release of training data: April 1, 2025 Release of testing data: April 15, 2025 Deadline for system submissions: May 16, 2025 Paper submission deadline: May 23, 2025 Notification of acceptance: June 13, 2025 Camera-ready paper due: June 20, 2025 Workshop: July 31, 2025
We encourage and invite participation from junior researchers and students from diverse backgrounds. Participants are also encouraged to submit a paper describing their systems to the SDP 2025 workshop.
Organisers
Raia Abu Ahmad (DFKI, Berlin, Germany) Aida Usmanova (Leuphana University, Lüneburg, Germany) Georg Rehm (DFKI, Berlin, Germany)