Call for Participation

We're excited to announce "SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data" (NLI4CT). This task aims to enhance the application of Large Language Models (LLMs) in clinical settings, addressing challenges like factual inconsistency, shortcut learning, and performance degradation. The SemEval workshop will be co-located with NAACL 2024 in Mexico City, Mexico on June 16–21, 2024.

Background: LLMs are crucial in Natural Language Processing but face limitations in real-world scenarios, especially in medical applications. There's a growing need to analyze over 400,000 Clinical Trial Reports (CTRs) efficiently. NLI offers a solution for large-scale interpretation and retrieval of medical evidence. Task Overview:

Task: Textual Entailment in the context of breast cancer CTRs. Determine the inference relation (entailment vs contradiction) between CTR-statement pairs. We provide a training set and a specially designed test set to evaluate the effectiveness of models in clinical NLI settings.

Example: Consider this scenario from a Clinical Trial Report (CTR):

Eligibility Criteria: - Gender: Female - ERBB2 positive BC - Treatment history: No previous chemotherapy

Exclusion Criteria: - BMI > 40 - Smoker

Given Statement: "Morbidly obese female patients with ERBB2+ Breast Cancer are not eligible for Trial X."

In this case, the statement contradicts the eligibility criteria of the CTR. While the patient's gender and medical condition align with the inclusion criteria, the term "morbidly obese" generally indicates a BMI > 40, which falls under the exclusion criteria. Therefore, the relationship between the CTR and the statement is a contradiction.

Research Aims: Investigate the consistency and faithfulness of NLI models in clinical settings. Develop and apply novel evaluation methodologies through interventional and causal analyses.

Important Dates: Practice Task: Already underway! Main Task: Starts January 10, 2024. Post-Competition Task: Begins February 1, 2024.

Get Involved: Register Here: https://codalab.lisn.upsaclay.fr/competitions/16190 Join Our Slack: https://join.slack.com/t/semeval2024task2/shared_invite/zt-22m7o4u7o-wa_Pnhi...