Call for participation: DETESTS-Dis IberLEF 2024 - Corpora

27 Feb 2024


      Please consider contributing and/or forwarding to appropriate colleagues
and groups.
*******We apologize for the multiple copies of this e-mail******
--------------------------------------------------------------------------------------------------------------------
Call for Participation
--------------------------------------------------------------------------------------------------------------------
DETESTS-Dis IberLEF 2024
Task: DETESTS-Dis (DETEction and classification of racial Stereotypes in
Spanish – Learning with Disagreement)
This task will take part of IberLEF 2024, the 6th Workshop on Iberian
Languages Evaluation Forum at the SEPLN 2024 Conference, which will be held
in Valladolid, Spain, on September 24th.
-------------------------------------------------------------------------------------------------------------------
Here, we introduce the second edition of the DETESTS task (Ariza-Casabona,
2022), which was first presented at IberLEF 2022. The aim of the new
edition, DETESTS-Dis, is to detect and classify explicit and implicit
stereotypes in texts from social media and comments on news articles,
incorporating learning with disagreement techniques. Next, a description of
both subtasks is provided:
-
Subtask 1, Stereotype Identification: This is a binary classification
   task the aim of which is to determine whether a comment or sentence
   contains at least one stereotype or none, considering the full distribution
   of labels provided by the annotators. This subtask follows the SemEval 2021
   Task 12 (Uma et al., 2021) proposal about learning with disagreement, in
   which the authors state that there does not necessarily exist a single gold
   label for every sample in the dataset. This fact is particularly evident
   when multiple contradictory annotations arise at the data labeling stage
   due to “debatable, subjective, or linguistic ambiguity”. The actual gold
   label of this subtask is left as a proxy to determine the subset of
   comments that will be evaluated in the posterior subtask.
-
Subtask 2 (Optional), Implicitness Identification: This subtask
   introduces a novel binary classification problem to determine whether the
   stereotype is manifested or latent within the text, that is, whether the
   stereotype is implicit or explicit. The added difficulty in this case is
   that implicit stereotypes are not directly expressed in the text, and a
   process of inference must be applied by the annotators. Moreover, there are
   different strategies in which an implicit stereotype can be coded, such as
   metaphors, irony and other figures of speech, evaluations of the in-group,
   and the overgeneralization of a social group from features of some of its
   members. This subtask will be presented as a hierarchical binary
   classification problem.
Although we recommend participating in both subtasks, participants are
allowed to participate just in one of them (e.g., subtask 1).
Teams will be allowed (and encouraged) to submit multiple runs (max. 5).
To avoid any conflict with the sources of the comments regarding their
intellectual property rights (IPR), the data will be sent privately to each
participant who is interested in the task. The corpus will only be made
available for research purposes.
Important dates (All deadlines are 11:59 PM UTC-12:00):
Training dataset release: March 04, 2024
Test dataset release: April 15, 2024
Systems results: April 29, 2024
Results notification: May 13, 2024
Working papers submission: June 3, 2024
Working papers (peer-)reviewed: June 17, 2024
Camera-ready versions: July 4, 2024
Workshop: September 24, 2024
Task organizers:
-
Mariona Taulé (Universitat de Barcelona, UB)
   -
Wolfgang Schmeisser (Universitat de Barcelona, UB)
   -
Alejandro Ariza (Universitat de Barcelona, UB)
   -
Pol Pastells (Universitat de Barcelona, UB)
   -
Mireia Farrús (Universitat de Barcelona, UB)
   -
Simona Frenda (Università degli Studi di Torino, UniTo)
   -
Paolo Rosso (Universitat Politècnica de València, UPV)
Contact:
Contact the organizers by writing to: detests.iberlef@gmail.com
Web page: https://detests-dis.github.io/
We invite participants to join our Google Groups to be kept up to date with
the latest news related to the task.