Call for Participation: The 2nd Shared Task on Multi-lingual Multi-task Information Retrieval

Organized by the 4th Workshop on Multilingual Representation Learning

In collocation with EMNLP 2024

Website: https://sigtyp.github.io/st2024-mrl.html

With the advancement of language models accessing and processing tons of information in different formats and languages, it has become of great importance to be able to assess the capabilities to access and provide the right information useful to different audiences. Large language models (LLMs) continue to demonstrate outstanding performance in many applications that require competence in language understanding and generation, but this performance is especially prominent in English, where large amounts of public evaluation benchmarks for various downstream tasks are available and the extent to which language models can be reliably deployed in terms of different languages and domains are not still well established.

In this new shared task we provide new high-quality annotations in a selected set of data-scarce and typologically-diverse languages that can be used in evaluation of multilingual LLMs in information retrieval tasks; including Named Entity Recognition (NER) and Reading Comprehension (RC), on test sets curated from articles on Wikipedia. The main objective of the shared task is to assess and understand the multilingual characteristics of the inference capability of multilingual LLMs in understanding and generating language based on logical, factual or causal relationships between knowledge contained over long contexts of text, especially under low-resource settings.

Task and Evaluation

Our task provides a multi-task evaluation format that assesses reading comprehension capabilities of language models in terms of two subtasks: named entity recognition and question answering.

Named Entity Recognition (NER) is a classification task that identifies phrases in a text that refer to entities or predefined categories (such as dates, person, organization and location names) and it is an important capability for information access systems that perform entity look-ups for knowledge verification, spell-checking or localization applications.

The objective of the system is to tag the named entities in a given text as person (PER), organization (ORG), location (LOC), and date (DATE) (Our tag set uses $$ as delimiter).

Question answering (QA) is an important capability that enables responding to natural language questions with answers found in text. Here we focus on the information-seeking scenario where questions can be asked without knowing the answer—it is the system’s job to locate a suitable answer passage (if any). The information-seeking question-answer pairs tend to exhibit less lexical and morphosyntactic overlap between the question and answer since they are written separately, which is a more suitable setting to evaluate typologically-diverse languages.

Here, the system is given a question, title, and a passage and the system must pick the right answer among a list of 4 different potential options.

We evaluate model performance in the generative task terms of the accuracy in the multi-choice answering task, and the F1 accuracy in the NER task. We obtain a final score by averaging the scores of QA and NER.

Data and Languages

Teams can use any resources relevant to the task.

The test sets for official evaluation will be released one week before the submission date, and will be in the following languages:

-

Turkish -

Uzbek -

Indonesian -

Alemannic -

Yoruba -

Igbo

Important dates

-

June 15, 2024: Release of validation data -

August 1, 2024: Release of test data -

August 20, 2024: Deadline to release external data and resources used in systems -

September 1, 2024: Deadline for submission of systems , and release of external data and resources used in systems. -

September 20, 2024: Release of rankings and results -

September 30, 2024: Deadline for submitting system description papers (All deadlines are 11.59 pm UTC -12h (“anywhere on Earth”))

Organizers

Duygu Ataman

David Ifeoluwa Adelani

Mammad Hajili

Francesco Tinner

Inder Khatri

Shared Task Prize

The winning team will receive an award of 500 USD and will be given a presentation during the workshop.

-- Duygu Ataman, Ph.D. Assistant Professor/Faculty Fellow Courant Institute of Mathematical Sciences New York University www.duyguataman.com