Dear colleagues,

As announced a few weeks ago, the fourth iteration of the GEM workshop will be held as part of ACL https://2025.aclweb.org/, July 27–August 1st, 2025. This year we’re planning a major upgrade to the GEM workshop, which we dub GEM2, through the introduction of a large dataset of 1B model predictions together with prompts and gold standard references, encouraging researchers from all backgrounds to submit work on meaningful, efficient and robust evaluation of LLMs.

In this second CfP, we are happy to announce that (i) the large datasets of model predictions have been released (DOVE https://slab-nlp.github.io/DOVE/ and DataDecide https://huggingface.co/datasets/allenai/DataDecide-eval-instances), and (ii) GEM2 will host the ReproNLP https://repronlp.github.io/ shared task results. Important Dates

-

April 11: Direct paper submission deadline (ARR). -

May 5: Pre-reviewed (ARR) commitment deadline. -

May 19: Notification of acceptance. -

June 6: Camera-ready paper deadline. -

July 7: Pre-recorded videos due. -

July 31 - August 1: Workshop at ACL in Vienna.

Please check the GEM website https://gem-benchmark.com/workshop for submission links, templates, and more details. For any questions, please email gem-benchmark-chairs@googlegroups.com.

best, simon

*ADAPT Research Centre / Ionaid Taighde ADAPT* *School of Computing, Dublin City University, Glasnevin Campus / Scoil na Ríomhaireachta, Campas Ghlas Naíon, Ollscoil Chathair Bhaile Átha Cliath*