We are pleased to announce the first shared task on Critical Questions Generation, hosted at The 12th Workshop on Argument Mining, co-located in ACL 2025 in Vienna, Austria. All details can be found in: https://hitz-zentroa.github.io/shared-task-critical-questions-generation/
**Introduction**
In recent years, a growing concern within the educational community has been whether the widespread use of LLM-based chats could foster superficial learning habits and weaken students' critical thinking skills. To counter this trend, in this task, we propose using LLMs to guide users towards asking critical questions. That is, questions that can uncover fallacious or poorly constructed arguments. In short: **we want to foster critical thinking by developing a system that generates insightful critical questions when given argumentative texts**.
In the same line, Natural Language Processing applications to deal with misinformation are a popular line of research. However, most applications face challenges regarding three issues: LLMs often lack the required up-to-date knowledge for these tasks, there is not always an agreement on what is the truth, and LLMs themselves can produce hallucinations or rely on unfaithful data, generating misinformation of their own making. **Yet, instead of requiring the LLMs to output factual knowledge, could we use them to point at the missing or potentially uninformed claims?**
**What is the task of Critical Questions Generation?**
The task of Critical Questions Generation consists of generating useful critical questions when given an argumentative text. For this purpose, a dataset of real debate interventions with associated critical questions will be released.
Critical Questions are the set of inquiries that should be asked in order to judge if an argument is acceptable or fallacious. Therefore, these questions are designed to unmask the assumptions held by the premises of the argument and attack its inference.
In the dataset, the argumentative texts are interventions of real debates, which have been annotated with Argumentation Schemes and later associated with a set of critical questions. For every intervention, the speaker, the set of Argumentation Schemes, and the critical questions are provided. These questions have been annotated according to their usefulness for challenging the arguments in each text. The labels are either Useful, Unhelpful, or Invalid. The goal of the task is to generate critical questions that are Useful.
The participant will be asked to develop a system that gets one of the interventions as input, and outputs exactly 3 critical questions. The 3 critical questions should all be useful for challenging the arguments in the intervention. Each of these 3 critical questions will be evaluated separately and then the punctuation will be aggregated. You can either create a system that generates only Useful CQs, or you can decide to generate many CQs and choose the top 3.
For more information on Critical Questions Generation, please read Critical Questions Generation: Motivation and Challenges https://aclanthology.org/2024.conll-1.9/
**Important Dates**
- Sample data available: 17th February - Validation data ready: 21st February - Evaluation start: 28th March - Evaluation end: 4th April - Paper submission due: 25th April - Notification to authors: 2nd May - Camera ready due: 9th May - Workshop: 31st July (co-located with ACL 2025)
**Organizers of the shared task**
- Blanca Calvo Figueras https://github.com/BlancaCalvo, HiTZ Basque Center for Language Technology - Ixa, University of the Basque Country UPV/EHU, Spain - Rodrigo Agerri https://ragerri.github.io/, HiTZ Basque Center for Language Technology - Ixa, University of the Basque Country UPV/EHU, Spain - Elena Cabrio https://www-sop.inria.fr/members/Elena.Cabrio/, University of Côte d’Azur and member of the Inria-I3S research team Wimmics - Serena Villata https://webusers.i3s.unice.fr/~villata/Home.html, University of Côte d’Azur and member of the Inria-I3S research team Wimmics
**Do you have other questions?**
Please become a member of the Google Group "critical-questions-generation-shared-task" https://groups.google.com/g/critical-questions-generation-shared-task to keep yourself updated. For specific questions, contact blanca.calvo@ehu.eus Website: https://hitz-zentroa.github.io/shared-task-critical-questions-generation/