SMAFIRA Shared Task Website: https://smafira-bf3r.github.io/smafira-st/

Dear all,

We'd like to invite you to participate in the SMAFIRA Shared Task that will take place in the BioNLP Workshop ( https://aclweb.org/aclwiki/BioNLP_Workshop) at ACL'25. The task aims at collecting annotations about the similarity of the research goals between two biomedical articles (title and abstract only). The collected annotations will later be used for training and evaluating our SMAFIRA tool (https://smafira.bf3r.de/), a tool for searching alternative methods to animal experiments.

*Important dates* (subject to changes)

- Release of the test data (batch1): Feb/2025 *(already available!)* - Release of the test data (batch2): Mar/2025 - Submission of annotations (batch1): April 30th, 2025 - Submission of participant paper: May 5th, 2025 - Final submissions of participant paper: May 20th, 2025 - Submission of annotations (batch2): July/2025

*Tasks and test data*

The tasks address two communities: biomedical experts for the *manual* annotation; and the NLP community for the *automatic* annotation.

The test data consists of a list of reference articles, i.e., Pubmed ( https://pubmed.ncbi.nlm.nih.gov) identifiers and their respective list of top 20 similar articles. A valid submission should contain annotations for all articles in the top 20 list. The annotation consists of *one* of the following degrees of similarity: *similar*, *uncertain *or *not similar*.

*Evaluation*

The evaluation will consist of inter-annotator agreement between the various submissions, as well as ranking the participants in terms of their contributions, i.e., number of annotations.

*Publication*

Contributions to the task can be published as a participant paper to the BioNLP Workshop at ACL'25 and/or as a co-authorship in a joint overview paper in a journal (after the end of the shared task, later this year).

Please visit our website (https://smafira-bf3r.github.io/smafira-st/) for mode details!

Best regards,

Mariana Neves German Federal Institute for Risk Assessment (BfR)