ClimateCheck: Shared Task on Scientific Fact-Checking of Social Media Claims on Climate Change

Hosted as part of the SDP 2025 Workshop July 31 or August 1st, 2025 (tbc) Vienna, Austria (co-located with ACL 2025)

ClimateCheck Shared Task: https://sdproc.org/2025/climatecheck.html SDP 2025 Workshop: https://sdproc.org/2025/index.html

Task Overview Social media facilitates discussions on critical issues such as climate change, but it also contributes to the rapid dissemination of misinformation, which complicates efforts to maintain an informed public and create evidence-based policies. In this shared task, we emphasise the need to link public discourse to peer-reviewed scholarly articles by gathering claims from social media about climate change (both real-life and automatically generated ones) as well as a corpus of about 400K abstracts of publications from the climate sciences domains. The participants will be asked to retrieve relevant abstracts for each claim (subtask I) and classify the relation between the claim and abstract as ‘supports’, ‘refutes’, or ‘not enough information’ (subtask II). The task will be hosted on Codabench (link will be provided soon on the webpage). Participants are allowed to take part either in subtask I only, or in both subtasks.

Subtask I: Abstracts Retrieval Task: given a claim from social media about climate change and a corpus of abstracts, retrieve the top K most relevant abstracts. Evaluation: MAP and B-Pref accounting for retrieving relevant abstracts and not penalising unjudged documents. Subtask II: Claim Verification Task: given the claim-abstract pairs received from the previous subtask, classify their relation as ‘support’, ‘refutes’, or ‘not enough information’. Evaluation: F1 score based on judged documents from gold data; unjudged documents will not be included in computing the score. Important dates Release of training data: April 1, 2025 Release of testing data: April 15, 2025 Deadline for system submissions: May 16, 2025 Paper submission deadline: May 23, 2025 Notification of acceptance: June 13, 2025 Camera-ready paper due: June 20, 2025 Workshop: July 31, 2025 or August 1, 2025 (TBA) We encourage and invite participation from junior researchers and students from diverse backgrounds. Participants are also encouraged to submit a paper describing their systems to the SDP 2025 workshop.

Organisers

Raia Abu Ahmad (DFKI, Berlin, Germany) Aida Usmanova (Leuphana University, Lüneburg, Germany) Georg Rehm (DFKI, Berlin, Germany)