Shared Task on Lexical Simplification for English, Portuguese and Spanish
In conjunction with the TSAR-2022 Workshop @EMNLP2022
*** CALL FOR PARTICIPATION ***
Lexical Simplification is the process of reducing the lexical complexity of a text by replacing difficult words with easier to read (or understand) expressions while preserving the original information and meaning. Lexical Simplification (LS) aims at facilitating reading comprehension to different target readerships such as foreign language learners, native speakers with low literacy levels, second language learners or people with different kinds of reading impairments. This new Lexical Simplification Shared Task features three similar datasets in three different languages: English, Brazilian Portuguese, and Spanish.
Definition of the task
Given a sentence containing a complex word, systems should return an ordered list of “simpler” valid substitutes for the complex word in its original context. The list of simpler words (up to a maximum of 10) returned by the system should be ordered by the confidence the system has in its prediction (best predictions first). The ordered list must not contain ties. An instance of the task for the English language is: 1. “That prompted the military to deploy its largest warship, the BRP Gregorio del Pilar, which was recently acquired from the United States.”
Complex word: deploy
For this instance a system may suggest the following ranked substitutes: send, move, position, redeploy, employ, situate…
Systems should only produce simplifications that are good contextual fits (semantically and syntactically).
Participating teams can register (details below) for three different tracks, one per language.
It is possible to participate in one, two or all three tracks. Participating teams will be allowed to submit up to 3 runs per track.
Data
The three datasets (trial data with gold annotations and test data without gold annotations)and the evaluation script will be available through a GitHub repository. There is no training dataset. However, a sample of 10 or 12 instances with gold standard annotations will be provided. Note that participating teams are allowed to use any other lexical simplification datasets or resources for developing their systems. Test data with gold annotations will also be released via the same GitHub repository at the end of the evaluation period.
Evaluation Metrics
The evaluation metrics to be applied in the TSAR-2022 Shared Task are the following:
MAP@K (Mean Average Precision @ K): K={1,3,5,10}. The MAP@K metric is used commonly for evaluating Information Retrieval models and Recommender Systems. For this Lexical Simplification task, instead of using a ranked list of relevant and irrelevant documents to evaluate our ranking output, we use a ranked list of predicted substitutes, which can be matched (relevant) and not matched (irrelevant) terms against the set of the gold-standard annotations for evaluation. The traditional Precision metric, in the context of Lexical Simplification, can be used to see how many of the predicted substitutes are relevant. But precision fails to capture the order in which correctly predicted substitutes are. Mean Average Precision is designed to work for binary relevance: candidates that match or not in the list of gold annotations. So MAP@K for Lexical Simplification evaluates the following aspects: 1) are the predicted substitutes relevant?, and 2) are the predicted substitutes at the top positions?
Potential@K: K={1,3,5,10}. The percentage of instances for which at least one of the substitutions predicted is present in the set of gold annotations.
Accuracy@K@top1: K={1,2,3}. The ratio of instances where at least one of the K top predicted candidates matches the most frequently suggested synonym/s in the gold list of annotated candidates.
Note 1: Potential@1/MAP@1/Precision@1 will give the same value. Note 2: The exact computation of the metrics will be provided in the official evaluation script.
Publication
Participating teams will be invited to submit system description papers (four pages with an unlimited number of pages for references) which will be peer-reviewed by at least 2 reviewers (at least one member of each participating team will be required to help with the review process) and papers will be published in the TSAR-2022 Workshop proceedings. The submissions will be via SoftConf and details for submission will be communicated to registered teams in due time.
Important dates
* Registration opens: July 19th, 2022 * Release of sample/trial instances with gold annotations: July 20th, 2022 * Release of evaluation metrics and code: July 22th, 2022 * Registration deadline: September 7, 2022 * Test set release (without gold annotations): September 8, 2022 * Submissions of systems' output due: September 15, 2022 * Official results announced: September 30, 2022 * Test set release (wit gold annotations): September 30, 2022 * Submission of Shared Tasks papers deadline: October 15, 2022 * Shared Task Papers Reviews due: November 1, 2022 * Camera-ready deadline for Shared-task papers: November 10, 2022 * TSAR Workshop and Shared Task: December 8, 2022
Please visit the TSAR-2022 Shared Task website to obtain further information about the Guidelines, Datasets, and team registration. https://taln.upf.edu/pages/tsar2022-st
Organisers
* Horacio Saggion, Chair in Computer Science and Artificial Intelligence and Head of the LaSTUS Lab in the TALN-DTIC, Universitat Pompeu Fabra * Sanja Štajner, Senior Research Scientist and R&D Application Manager at Symanto Research * Matthew Shardlow, Senior Lecturer at Manchester Metropolitan University * Marcos Zampieri, Assistant Professor at the Rochester Institute of Technology * Daniel Ferrés, Post-Doctoral Research Assistant at LaSTUS Lab. at TALN-DTIC, Universitat Pompeu Fabra * Kai North, PhD student at the Rochester Institute of Technology * Kim Cheng Sheang, PhD student at LaSTUS Lab. at TALN-DTIC, Universitat Pompeu Fabra
--
Professor Horacio Saggion
Head of the Large Scale Text Understanding Systems Lab
Full Professor / Chair in Computer Science and Artificial Intelligence
