Call for Participation

BEA 2023 Shared Task: Generating AI Teacher Responses in Educational Dialogues

https://sig-edu.org/sharedtask/2023

SHARED TASK DESCRIPTION Conversational agents offer promising opportunities for education. They can fulfill various roles (e.g., intelligent tutors and service-oriented assistants) and pursue different objectives (e.g., improving student skills and increasing instructional efficiency) (Wollny et al. 2021). Among all of these different vocations of an educational chatbot, the most prevalent one is the *AI teacher* helping a student with skill improvement and providing more opportunities to practice. Some recent meta-analyses have even reported a significant effect of chatbots on skill improvement, for example in language learning (Bibauw et al. 2022). What is more, current advances in AI and natural language processing have led to the development of conversational agents that are founded on more powerful generative language models.

Despite these promising opportunities, the use of powerful generative models as a foundation for downstream tasks also presents several crucial challenges. In the educational domain in particular, it is important to ascertain whether that foundation is solid or flimsy. Bommasani et al. (2021: pp. 67-72) stressed that, if we want to put these models into practice as AI teachers, it is imperative to determine whether they can (a) speak to students like a teacher, (b) understand students, and (c) help students improve their understanding. Therefore, Tack and Piech (2022) formulated the* AI teacher test challenge*: How can we test whether state-of-the-art generative models are good AI teachers, capable of replying to a student in an educational dialogue?

Following the AI teacher test challenge, we organize a *first shared task on the generation of teacher language in educational dialogues*. The goal of the task is to use NLP and AI methods to generate teacher responses in real-world samples of teacher-student interactions. These samples are taken from the Teacher Student Chatroom Corpus (Caines et al. 2020; Caines et al. 2022). Each training sample is composed of a dialogue context (i.e., several teacher-student utterances) as well as the teacher’s response. For each test sample, participants are asked to submit their best generated teacher response.

The purpose of the task is to *benchmark the ability of generative models to act as AI teachers, replying to a student in a teacher-student dialogue*. Submissions will be ranked according to several automated dialogue evaluation metrics, with the top submissions selected for further human evaluation. During this manual evaluation, human raters will compare a pair of teacher responses in terms of three abilities: can speak like a teacher, can understand a student, can help a student (Tack & Piech 2022). As such, we adopt an evaluation method that is akin to ACUTE-Eval for evaluating dialogue systems (Li et al. 2019).

*PARTICIPATION* The shared task is hosted on *CodaLab* (Pavao et al. 2022). Anyone participating in the shared task will be asked to:

1. Register on the CodaLab https://codalab.lisn.upsaclay.fr/ platform. 2. Fill in the registration form https://forms.gle/iAdKCq3dRS9srzjc6 with their CodaLab ID. Participants must comply with the terms and conditions of the task and the TSCC data outlined in the form. 3. Register for the CodaLab competition https://codalab.lisn.upsaclay.fr/competitions/11705 using the CodaLab ID. We will only accept people who submitted the registration form. Note that you can participate as a member of one team only.

*IMPORTANT DATES*

*Fri Mar 24, 2023* Training data release *Mon May 1, 2023* Test data release *Fri May 5, 2023* Final submissions due *Mon May 8, 2023* Results announced *Fri May 12, 2023* Human evaluation results announced *Mon May 22, 2023* System papers due *Fri May 26, 2023* Paper reviews returned *Tue May 30, 2023* Camera-ready papers due *Mon June 12, 2023* Pre-recorded video due *July 13, 2023* BEA Workshop at ACL

*ORGANIZERS*

Anaïs Tack, KU Leuven; Ekaterina Kochmar, MBZUAI; Zheng Yuan, King’s College London; Serge Bibauw, Universidad Central del Ecuador; Chris Piech, Stanford University

*Webpage*: https://sig-edu.org/sharedtask/2023 https://sig-edu.org/sharedtask/2023

Dear all,

Due to multiple requests, we are extending the BEA paper submission deadline to *Tuesday, May 2, 2023, 11:59pm UTC-12.* Best wishes, BEA Organizing Committee

==================================

Deadline EXTENDED: Tuesday, May 2, 2023, 11:59pm UTC-12

The 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)

Location: Toronto

Date: Thursday, July 13, 2023 (co-located with ACL 2023)

Website: https://sig-edu.org/bea/current

*Submission Deadline (extended): Tuesday, May 2, 2023, 11:59pm UTC-12*

WORKSHOP DESCRIPTION

The BEA Workshop is a leading venue for NLP innovation in the context of educational applications. It is one of the largest one-day workshops in the ACL community with over 100 registered attendees in the past several years. The growing interest in educational applications and a diverse community of researchers involved resulted in the creation of the Special Interest Group in Educational Applications (SIGEDU) https://www.aclweb.org/adminwiki/index.php?title=2019Q3_Reports:_SIGEDU in 2017, which currently has over 300 members.

The 18th BEA workshop will have keynotes by Susan Lottridge (Cambium Assessment) and Jordana Heller (Textio), an invited paper presentation by a member of one of the educational societies from the International Alliance to Advance Learning in the Digital Era (IAALDE), oral presentation sessions, and a large poster session. We expect that the workshop will continue to highlight novel technologies and opportunities for educational NLP in a range of languages.

We will solicit papers that incorporate NLP methods, including, but not limited to:

- automated scoring of open-ended textual and spoken responses; - automated scoring/evaluation for written student responses (across multiple genres); - AI and ML methods in educational applications; - integration of large language models (LLMs) in educational applications; - game-based instruction and assessment; - educational data mining; - intelligent tutoring; - collaborative learning environments; - peer review; - grammatical error detection and correction; - learner cognition; - spoken dialog; - multimodal applications; - annotation standards and schemas; - tools and applications for classroom teachers, learners and/or test developers; and - use of corpora in educational tools.

SHARED TASK *Webpage*: https://sig-edu.org/sharedtask/2023

BEA 2023 is hosting a shared task on generation of teacher responses in educational dialogues. Participants will be provided with teacher–student dialogue samples from the Teacher Student Chatroom Corpus (Caines et al., 2020) of real-world teacher–student interactions and will be asked to generate teacher responses using NLP and AI methods. Submissions will be ranked according to automated evaluation metrics, with the top submissions selected for further human evaluation. Given active participation in the previous BEA-hosted shared tasks, we expect to attract around 20 participating teams.

*Organizers*: Anaïs Tack, KU Leuven; Ekaterina Kochmar, MBZUAI; Zheng Yuan, King’s College London; Serge Bibauw, Universidad Central del Ecuador; Chris Piech, Stanford University

INVITED TALKS

BEA 2023 will feature invited talks from Susan Lottridge (Cambium Assessment) and Jordana Heller (Textio), as well as a speaker from one of the IAALDE https://alliancelss.com/ societies.

IMPORTANT DATES

All deadlines are 11:59pm UTC-12 (anywhere on earth).

- Submission Deadline (extended): Tuesday, May 2, 2023 - Notification of Acceptance: Monday, May 22, 2023 - Camera-ready Papers Due: Tuesday, May 30, 2023 - Workshop: Thursday, July 13, 2023

SUBMISSION INFORMATION

We will be using the ACL Submission Guidelines for the BEA Workshop this year. Authors are invited to submit a long paper of up to eight (8) pages of content, plus unlimited references; final versions of long papers will be given one additional page of content (up to 9 pages) so that reviewers’ comments can be taken into account. We also invite short papers of up to four (4) pages of content, plus unlimited references. Upon acceptance, short papers will be given five (5) content pages in the proceedings. Authors are encouraged to use this additional page to address reviewers’ comments in their final versions. Papers which describe systems are also invited to give a demo of their system. If you would like to present a demo in addition to presenting the paper, please make sure to select either “long paper + demo” or “short paper + demo” under “Submission Category” in the START submission page.

Previously published papers cannot be accepted. The submissions will be reviewed by the program committee. As reviewing will be blind, please ensure that papers are anonymous. Self-references that reveal the author’s identity, e.g., “We previously showed (Smith, 1991) …”, should be avoided. Instead, use citations such as “Smith previously showed (Smith, 1991) …”.

We have also included conflict of interest in the submission form. You should mark all potential reviewers who have been authors on the paper, are from the same research group or institution, or who have seen versions of this paper or discussed it with you.

We will be using the START conference system to manage submissions: https://www.softconf.com/acl2023/bea2023/

DOUBLE SUBMISSION POLICY

We will follow the official ACL double-submission policy https://www.aclweb.org/archive/policies/current/double-submission-policy.html. Specifically:

Papers being submitted both to BEA and another conference or workshop must:

- Note on the title page the other conference or workshop to which they are being submitted. - State on the title page that if the authors choose to present their paper at BEA (assuming it was accepted), then the paper will be withdrawn from other conferences and workshops.

ORGANIZING COMMITTEE

- Ekaterina Kochmar https://ekochmar.github.io/about/, MBZUAI https://ekochmar.github.io/about/ - Jill Burstein https://sites.google.com/site/jbursteinets/, Duolingo - Andrea Horbach https://www.ltl.uni-due.de/team/andrea-horbach/, FernUniversität in Hagen - Ronja Laarmann-Quante https://www.ltl.uni-due.de/team/ronja-laarmann-quante, Ruhr University Bochum - Nitin Madnani https://desilinguist.org/, Educational Testing Service - Anaïs Tack https://anaistack.github.io/, KU Leuven - Victoria Yaneva http://www.victoriayaneva.info/, National Board of Medical Examiners - Zheng Yuan https://www.cl.cam.ac.uk/~zy249/, King’s College London - Torsten Zesch https://www.ltl.uni-due.de/team/torsten-zesch, FernUniversität in Hagen

Workshop contact email address: bea.nlp.workshop@gmail.com