Please consider contributing and/or forwarding to appropriate colleagues and groups.
*******We apologize for the multiple copies of this e-mail******
--------------------------------------------------------------------------------------------------------------------
Call for Participation --------------------------------------------------------------------------------------------------------------------
DETESTS-Dis IberLEF 2024
Task: DETESTS-Dis (DETEction and classification of racial Stereotypes in Spanish – Learning with Disagreement)
This task will take part of IberLEF 2024, the 6th Workshop on Iberian Languages Evaluation Forum at the SEPLN 2024 Conference, which will be held in Valladolid, Spain, on September 24th.
-------------------------------------------------------------------------------------------------------------------
Here, we introduce the second edition of the DETESTS task (Ariza-Casabona, 2022), which was first presented at IberLEF 2022. The aim of the new edition, DETESTS-Dis, is to detect and classify explicit and implicit stereotypes in texts from social media and comments on news articles, incorporating learning with disagreement techniques. Next, a description of both subtasks is provided:
-
Subtask 1, Stereotype Identification: This is a binary classification task the aim of which is to determine whether a comment or sentence contains at least one stereotype or none, considering the full distribution of labels provided by the annotators. This subtask follows the SemEval 2021 Task 12 (Uma et al., 2021) proposal about learning with disagreement, in which the authors state that there does not necessarily exist a single gold label for every sample in the dataset. This fact is particularly evident when multiple contradictory annotations arise at the data labeling stage due to “debatable, subjective, or linguistic ambiguity”. The actual gold label of this subtask is left as a proxy to determine the subset of comments that will be evaluated in the posterior subtask.
-
Subtask 2 (Optional), Implicitness Identification: This subtask introduces a novel binary classification problem to determine whether the stereotype is manifested or latent within the text, that is, whether the stereotype is implicit or explicit. The added difficulty in this case is that implicit stereotypes are not directly expressed in the text, and a process of inference must be applied by the annotators. Moreover, there are different strategies in which an implicit stereotype can be coded, such as metaphors, irony and other figures of speech, evaluations of the in-group, and the overgeneralization of a social group from features of some of its members. This subtask will be presented as a hierarchical binary classification problem.
Although we recommend participating in both subtasks, participants are allowed to participate just in one of them (e.g., subtask 1).
Teams will be allowed (and encouraged) to submit multiple runs (max. 5).
To avoid any conflict with the sources of the comments regarding their intellectual property rights (IPR), the data will be sent privately to each participant who is interested in the task. The corpus will only be made available for research purposes.
Important dates (All deadlines are 11:59 PM UTC-12:00):
Training dataset release: March 04, 2024
Test dataset release: April 15, 2024
Systems results: April 29, 2024
Results notification: May 13, 2024
Working papers submission: June 3, 2024
Working papers (peer-)reviewed: June 17, 2024
Camera-ready versions: July 4, 2024
Workshop: September 24, 2024
Task organizers:
-
Mariona Taulé (Universitat de Barcelona, UB) -
Wolfgang Schmeisser (Universitat de Barcelona, UB) -
Alejandro Ariza (Universitat de Barcelona, UB) -
Pol Pastells (Universitat de Barcelona, UB) -
Mireia Farrús (Universitat de Barcelona, UB) -
Simona Frenda (Università degli Studi di Torino, UniTo) -
Paolo Rosso (Universitat Politècnica de València, UPV)
Contact:
Contact the organizers by writing to: detests.iberlef@gmail.com
Web page: https://detests-dis.github.io/
We invite participants to join our Google Groups to be kept up to date with the latest news related to the task.