[apologies if you receive multiple copies of this call]

Dear colleagues and friends,

*We are pleased to release the 1st Call for Participation - CLEF 2024 SimpleText Task4: SOTA?*

*Overview:* SOTA? is introduced as Task 4 in the SimpleText track of CLEF 2024. The goal of the SOTA? shared task is to develop systems which given the full text of an AI paper, are capable of recognizing whether an incoming AI paper indeed reports model scores on benchmark datasets, and if so, to extract all pertinent (Task, Dataset, Metric, Score) quadruples presented within the paper.

More info on the task website: https://sites.google.com/view/simpletext-sota/home

SOTA? will be divided into two evaluation phases:

- Evaluation Phase 1: Few-shot Testing; - Evaluation Phase 2: Zero-shot Testing

*To participate in SOTA? i.e. SimpleText Task 4 @ CLEF 2024, please register your team*:

1. CLEF 2024 official registration page https://clef2024.imag.fr/index.php?page=Pages/registration.html 2. Codalab competition site: https://codalab.lisn.upsaclay.fr/competitions/16616

Note, SOTA? is organized as a new task this year under the "SimpleText - Improving Access to Scientific Texts for Everyone" initiative https://simpletext-project.com/. Please take a look at the other 3 tasks, i.e. Task 1, 2, and 3, offered by SimpleText and select one or more of those task options too if you are interested. Note that there is no interdependence of the dataset between "Task 4 - SOTA?" and the other three tasks of SimpleText.

*Dates*

Training and validation datasets available: Feb 1, 2024 Test data available/Evaluation starts: April 23, 2024 Evaluation ends: May 3, 2024 Participant paper submissions due: May 31, 2024 Notification to authors: June 24, 2024 Camera ready due: July 8, 2024 CLEF 2024 Workshop, Grenoble, France: 9-12 September 2024

*Task Organizers*

Jennifer D’Souza (TIB Leibniz Information Centre for Science and Technology - Germany) Salomon Kabongo (L3S Research Center, Germany) Hamed Babaei Giglou (TIB Leibniz Information Centre for Science and Technology - Germany) Yue Zhang (Berlin Technical University, Germany) Sören Auer (TIB Leibniz Information Centre for Science and Technology - Germany)

*We look forward to having you on board!*

*Contact:* sota.task [at] gmail.com