Call for Participation: The 4th Workshop on Human Evaluation of NLP Systems (HumEval’24)

Date: 21 May 2024 (full day)

Venue: Lingotto Conference Centre, Turin, Italy

Registration

Registration is mandatory for attending the workshop. Find more info about registration on the LREC-COLING 2024 website: https://lrec-coling-2024.org/

Workshop description

Human evaluation plays a central role in NLP, from the large-scale crowd-sourced evaluations to the much smaller experiments routinely encountered in conference papers. Yet there is growing unease about how human evaluations are conducted in NLP. Researchers have pointed out the less-than-perfect experimental and reporting standards that prevail (van der Lee et al., 2019 https://aclanthology.org/W19-8643/; Gehrmann et al., 2023 https://www.jair.org/index.php/jair/article/view/13715/26927), and that low-quality evaluations with crowdworkers may not correlate well with high-quality evaluations with domain experts (Freitag et al., 2021 https://aclanthology.org/2021.tacl-1.87). Only a small proportion of papers provide enough detail for reproduction of human evaluations, and in many cases the information provided is not even enough to support the conclusions drawn (Belz et al., 2023 https://aclanthology.org/2023.insights-1.1).

The HumEval workshop (previously at EACL 2021, ACL 2022, and RANLP 2023) aim to create a forum for current human evaluation research and future directions, a space for researchers working with human evaluations to exchange ideas and begin to address the issues human evaluation in NLP faces in many respects, including experimental design, meta-evaluation and reproducibility.

Programme

Find the detailed programme on the workshop website: https://humeval.github.io/2024/programme

Invited speakers

Mark Diaz (Google Research)

Sheila Castilho (ADAPT/DCU)

Organising Committee

Simone Balloccu, Charles University, CZ

Anya Belz, ADAPT Centre, Dublin City University, Ireland

Rudali Huidrom, Dublin City University, Ireland

Ehud Reiter, University of Aberdeen, UK

João Sedoc, New-York University

Craig Thomson, ADAPT Centre, Dublin City University, Ireland

For questions and comments regarding the workshop please contact Simone Balloccu at balloccu@ufal.mff.cuni.cz and humeval.ws@gmail.com.