Dear all, the QE shared task 2024 is ON!
You can now submit and test your quality estimation system(s) on a set of different languages and tasks: to predict translation quality at sentence level, to detect error spans, or even to correct translations!
For information on how to access the test data and the submission platforms, visit the shared task's webpage: https://www2.statmt.org/wmt24/qe-task.html
Deadline to participate is July 31 (AoE).
Looking forward to receiving your predictions!
-- Best wishes, on behalf of the organisers.
Dear all, we are happy to invite you to participate in the Shared Task on Quality Estimation at WMT'24. The details of the task can be found at: https://www2.statmt.org/wmt24/qe-task.html
New this year:
* We introduce a new language pair (zero-shot): English-Spanish * Continuing from the previous edition, we will also analyse the robustness of submitted QE systems to a set of different phenomena which will span from hallucinations and biases to localized errors, which can significantly impact real-world applications.
* We also introduce a new task, seeking not only to detect but also to correct errors: Quality-aware Automatic Post-Editing! We invite participants to submit systems capable of automatically generating QE predictions for machine-translated text and the corresponding output corrections.
2024 QE Tasks:
Task 1 -- Sentence-level quality estimation This task follows the same format as last year but with fresh test sets and a new language pair: English-Spanish. We will test the following language pairs:
* English to German (MQM) * English to Spanish (MQM) * English to Hindi (MQM & DA) * English to Gujarati (DA) * English to Telugu (DA) * English to Tamil (DA) More details: https://www2.statmt.org/wmt24/qe-subtask1.html
Task 2 -- Fine-grained error span detection Sequence labelling task: predict the error spans in each translation and the associated error severity: Major or Minor. We will test the following language pairs:
* English to German (MQM) * English to Spanish (MQM) * English to Hindi (MQM) More details: https://www2.statmt.org/wmt24/qe-subtask2.html
Task 3 -- Quality-aware Automatic Post-editing We expect submissions of post edits correcting detected error spans of the original translation. Although the task is focused on quality-informed APE, we also allow participants to submit APE output without QE predictions to understand the impact of their QE system. Submissions w/o QE predictions will also be considered official. We will test the following language pairs:
* English to Hindi * English to Tamil More details: https://www2.statmt.org/wmt24/qe-subtask3.html
Important dates:
1. Test sets will be released on July 15th. 2. Participants can submit their systems by July 23rd on codalab. 3. System paper submissions are due by 20th August [aligned with WMT deadlines].
Note: Like last year, we aligned with the General MT and Metrics shared tasks to facilitate cross-submission on the common language pairs: English-German, English-Spanish, and English-Hindi (MQM).
We look forward to your submissions and feel free to contact us if you have any more questions!
Best wishes, on behalf of the organisers.