Dear Colleagues,
The evaluation period for the brand new Model Compression track https://www2.statmt.org/wmt25/model-compression.html at WMT 2025 https://www2.statmt.org/wmt25/index.html is approaching!
LATEST ANNOUNCEMENTS:
-
Test data release brought forward to June 19, 2025! Participants now have two full weeks to prepare their submissions. -
Submission upload space available upon request (see the task’s page for details)
OVERVIEW
This shared task aims to evaluate the potential of model compression techniques in reducing the size of large, general-purpose language models, with the goal of achieving an optimal balance between practical deployability and high translation quality in specific machine translation (MT) scenarios. The broader objectives of the task include:
-
fostering research into efficient, accessible, and sustainable deployment of LLMs for MT; -
establishing a common evaluation framework to monitor progress in model compression across a wide range of languages; and -
enabling meaningful comparisons with state-of-the-art MT systems through standardized evaluation protocols that assess not only translation quality but also efficiency.
Although the focus is on model compression, the task is closely aligned with the General MT shared task https://www2.statmt.org/wmt25/translation-task.html, sharing language directions, test data, and protocols for automatic MT quality evaluation. Additionally, the task follows the same timeline as the flagship WMT task.
We warmly invite participation from academic teams and industry players interested in applying existing compression methods to MT or exploring innovative, cutting-edge approaches.
THE TASK IN A NUTSHELL
-
Goal: Reduce the size of a general-purpose LLM while maintaining a balance between model compactness and MT performance. -
Languages: The first round will focus on the same language pairs as the General MT track. -
Conditions: -
Constrained: Participants work within a predefined model and language setting for directly comparable results. -
Unconstrained: Participants are free to compress any model across language directions of their choice. -
Evaluation Criteria: -
Translation quality: Automatically measured using the LLM-as-a-judge framework from the General MT task -
Model size: Defined by the memory usage -
Inference speed: Measured by total processing time over the test set
IMPORTANT DATES (UPDATED)
-
Test data released: 26th June 2025 19th June 2025 -
Translation submission deadline: 3rd July 2025 -
System description abstract paper: 10th July 2025 -
System description submission: 14th August 2025
WEBSITE: https://www2.statmt.org/wmt25/model-compression.html
ORGANIZERS:
-
Marco Gaido, Fondazione Bruno Kessler -
Matteo Negri, Fondazione Bruno Kessler -
Roman Grundkiewicz - Microsoft Translator -
TG Gowda - Microsoft Translator
CONTACTS:
-
Marco Gaido - mgaido@fbk.eu -
Matteo Negri - negri@fbk.eu