[Apologies for cross-posting]
ODESIA CHALLENGE @ SEPLN 2024 – Evaluation of NLP Systems in Spanish https://leaderboard.odesia.uned.es/en/leaderboard/challenge
Call for Participation
The members of the ODESIAhttps://odesia.es/ project (Space for the Observation of the Development of Spanish in Artificial Intelligence) and the organizers of SEPLN 2024 are pleased to invite you to participate in the ODESIA Challenge @ SEPLN 2024. This competition aims to promote the development and evaluation of language technologies in Spanish using the evaluation platform and datasets provided by ODESIA (see full rules herehttps://unedo365-my.sharepoint.com/:b:/g/personal/al_benito_lsi_uned_es/EWmKb2Y861JPh3ovoEZiA34BSmYsommwul5ZCTais0urMw?e=XVzioS).
Participants must create a system capable of solving 10 discriminative Natural Language Processing (NLP) tasks in Spanish belonging to the ODESIA Leaderboardhttps://leaderboard.odesia.uned.es/en/node/1. The winning team will receive a cash prize of 3,000 euros, donated by the company Llorente y Cuenca Madrid, SL. (conditions apply, see below).
Tasks The ODESIA-CORE benchmark consists of 10 discriminative tasks with public training datasets and private test datasets (not previously distributed by any means) created within the ODESIA initiative. The private nature of the test data guarantees the absence of contamination in the leaderboard results: no LLM should have seen the test set annotations in its pre-training phase.
Accepted Systems All types of Natural Language Processing (NLP) systems that are applied uniformly to all tasks will be accepted. That is, each participation must be a single system that applies to all tasks, instead of different approaches for each task. A submission in which the solution for each task is constructed independently will not be acceptable. For illustrative purposes, systems with the following characteristics (the list is non-exhaustive) are acceptable:
1. The system is an encoder-type LLM (or an ensemble of LLMs), to which a fine-tuning process is applied for each of the challenge tasks, using the training data provided in the participants’ package or from other sources as deemed appropriate by the participating team. 2. The system uses one or more generative LLMs, combined with a uniform zero-shot, one-shot or few-shot prompting strategy. 3. The system uses one or more generative LLMs combined with a retrieval-augmented generation strategy on the training dataset or other external sources. 4. Any combination of the above methods, as long as it is applied uniformly to all datasets.
Registration and Participation Teams will have to pre-register before they can participate. Each team will register a single account on the ODESIA Leaderboard evaluation platform using the form provided for this purpose (linkhttps://forms.office.com/e/Tg0Yv6AtHw). The organizers will provide a username and password on the ODESIA Leaderboard platform upon validation of the registration data.
Prize A single prize of 3,000 euros -donated by Llorente y Cuenca Madrid, SL- will be awarded to the team that submits the system with the best global average performance in the ODESIA-CORE tasks for Spanish, and that outperforms the best current model in the Leaderboard (XLM-Roberta-Large - 0.5873). Please note that for the prize to be awarded, there must be a minimum of five teams submitting results; if this number is not met, the organization reserves the right to defer the challenge's deadline until this number is reached. Also, The winning team commits to present its solution (in-person or online) at the Award ceremony at SEPLN 2024 (25th September 2024, Valladolid - Spain).
Important dates
* Registration opens: 1st July 2024 * Registration closes: 30th July 2024 * * Submission deadline: 14th September 2024 * * Official results announced: 16-20th September 2024 * Award ceremony and presentation of results: 25th September 2024 - 5:30pm, at SEPLN 2024
*23:59 AoE (Anywhere on Earth)
Organizing Committee
* Alejandro Benito-Santos (co-chair, UNED) * Roser Morante (co-chair, UNED) * Julio Gonzalo (UNED) * Jorge Carrillo-de-Albornoz (UNED) * Laura Plaza (UNED) * Enrique Amigó (UNED) * Víctor Fresno (UNED) * Andrés Fernández (UNED) * Adrián Ghajari (UNED) * Guillermo Marco (UNED) * Eva Sánchez (UNED) * Miguel Lucas (LLYC)
Advisory Board
* TBA
Contact and More Information:
* The full contest rules, along with instructions to participate, can be found at the ODESIA Leaderboard websitehttps://leaderboard.odesia.uned.es/leaderboard/challenge and here in PDFhttps://unedo365-my.sharepoint.com/:b:/g/personal/al_benito_lsi_uned_es/EWmKb2Y861JPh3ovoEZiA34BSmYsommwul5ZCTais0urMw?e=XVzioS. * For questions related to the challenge, please join our Discord server: #odesia-challenge-2024.https://discord.gg/psw7ayZzf6 You can also contact the challenge co-chairs, Alejandro Benito-Santos (al.benito@lsi.uned.esmailto:al.benito@lsi.uned.es) and Roser Morante (r.morant@lsi.uned.esmailto:rmorante@lsi.uned.es).
AVISO LEGAL. Este mensaje puede contener información reservada y confidencial. Si usted no es el destinatario no está autorizado a copiar, reproducir o distribuir este mensaje ni su contenido. Si ha recibido este mensaje por error, le rogamos que lo notifique al remitente. Le informamos de que sus datos personales, que puedan constar en este mensaje, serán tratados en calidad de responsable de tratamiento por la UNIVERSIDAD NACIONAL DE EDUCACIÓN A DISTANCIA (UNED) c/ Bravo Murillo, 38, 28015-MADRID-, con la finalidad de mantener el contacto con usted. La base jurídica que legitima este tratamiento, será su consentimiento, el interés legítimo o la necesidad para gestionar una relación contractual o similar. En cualquier momento podrá ejercer sus derechos de acceso, rectificación, supresión, oposición, limitación al tratamiento o portabilidad de los datos, ante la UNED, Departamento de Política Jurídica de Seguridad de la Informaciónhttps://www.uned.es/dpj, o a través de la Sede electrónicahttps://sede.uned.es/ de la Universidad. Para más información visite nuestra Política de Privacidadhttps://descargas.uned.es/publico/pdf/Politica_privacidad_UNED.pdf.