Apologies for the multiple postings. ----
*Indian Language Summarization (ILSUM 2023)* Website: https://ilsum.github.io/
To be organized in conjunction with FIRE 2023 (fire.irsi.res.in) 15th-18th December 2023, Goa, India -------------------------------------------------------
The second shared task on Indian Language Summarization (ILSUM) aims at creating an evaluation benchmark dataset for Indian Languages. This year ILSUM consists of two subtasks
Subtask 1: This task builds upon the task from ILSUM 2022. In the first edition, we covered two major Indian languages Hindi and Gujarati alongside Indian English, a widely recognized dialect of the English Language. This year's edition adds the Bengali language and an expanded dataset for the languages from last year. Further, we will provide abstractive summaries for a subset of each language (~1000 per language) apart from the headlines which are semi-extractive summaries in nature. Like the previous edition, this will be a classic summarization task, where we will provide ~15,000 article-summary pairs for each language and the participants are expected to generate a fixed-length summary.
Subtask 2: The task is centred around identifying factual errors in machine-generated summaries. With the recent implosion of Large Language models, . While these LLMs are very good at summarization, among other NLP tasks, they are often prone to hallucinations. This means the model generates information that is not accurate, not based on its training data, or is completely made up but looks accurate and reliable. Further, such tools can be misused to generate misleading or outright incorrect information. Identifying such inaccuracies can be a challenging task. Through this subtask, we aim to address the problem of identifying factually incorrect information in LLM-generated summaries. Participants will be provided with an article and its corresponding machine-generated summary. The objective is to identify the presence of factual incorrectness in the summaries if any, and classify them in one of the predefined categories.
*Tentative Timeline* ------------- 7st August - Training Data Released and Registrations open 10th September - Test Data Release 20th September - Run Submission Deadline 25th September - Results Declared 10th October - Working notes due 25th October - Reviews Due 30th October - Camera Ready Submissions due
15th-18th December - FIRE 2023 at Goa, India
*Organisers* ---------------- Jagrat Patel, LDRP-ITR, Gandhinagar, India Jaivin Barot, LDRP-ITR, Gandhinagar, India Tanishka Gaur, LDRP-ITR, Gandhinagar, India Shrey Satapara, Indian Institute of Technology, Hyderabad, India Sandip Modha, LDRP-ITR, Gandhinagar, India Parth Mehta, Parmonic, USA Debasis Ganguly, University of Glasgow, Scotland
*For regular updates subscribe to our mailing list: **ilsum@googlegroups.com**