3rd CALL FOR PAPERS
Third International Workshop on Gender-Inclusive Translation Technologies (GITT) at MT Summit 2025
23 June 2025, Geneva, Switzerland
https://sites.google.com/tilburguniversity.edu/gitt2025
@gitt-workshop.bsky.social
Important Dates (Time zone: Anywhere on Earth)
Submission deadline: 10 March, 2025
Notification of Acceptance: 4 April, 2025
Camera Ready Copy due: 11 April, 2025
Workshop: 23 June, 2025
**Aim and scope**
The Gender-Inclusive Translation Technologies Workshop …
[View More](GITT) is set out to be the dedicated workshop that focuses on gender-inclusive language in translation and cross-lingual scenarios. The workshop aims to bring together researchers from diverse areas, including industry partners, MT practitioners, and language professionals. GITT aims to encourage multidisciplinary research that develops and interrogates both solutions and challenges for addressing bias and promoting gender inclusivity in MT and translation tools, including LMs applications for the translation task.
**Topics**
GITT invites technical as well as non-technical submissions, which consist of experimental, theoretical or methodological contributions. We explicitly welcome interdisciplinary submissions and submissions that focus on innovative, non-binary linguistic strategies and/or with sociolinguistically-informed perspectives. The topics of interest include, but are not limited to:
- Models or methods for assessing and mitigating gender bias
- New resources for inclusive language and gender translation (e.g., datasets, translation memories, dictionaries)
- Social, cross-lingual, and ethical implications of gender bias
- Qualitative and quantitative analyses on the potential limits of current approaches to gender bias in translation and MT, error taxonomies as well as best practices and guidelines
- User-centric case studies on the impact of biased language and/or mitigating approaches which can include translators, post-editors, or monolingual MT users
GITT is also open to other non-listed topics aligned with the scope of the workshop and works focusing on non-textual modalities (e.g., audiovisual translation)
**Submission**
We welcome four types of submissions, two archival and two non-archival.
ARCHIVAL
- Research papers: of at least 4 up to 10 pages (excluding references)
- Extended Abstracts: up to 2 pages (including references)
Accepted papers and extended abstracts consisting of novel work will be published online as proceedings in the ACL Anthology.
NON-ARCHIVAL
- Research Communications: up to 2 pages (including references).
We include a parallel submission policy in the form of Research Communications for papers related to the topic of GITT that were accepted in other venues in 2024 and 2025.
- Potluck Communications: short abstract up to 500 words (including references).
Potluck Communications offer a space for anyone—especially students and early career researchers—to discuss bold new ideas for collaboration, brainstorm about ongoing work, and explore future research directions.
The communications will not be included in the proceedings, but will serve to promote the dissemination of research aligned with the scope of the workshop.
All submissions should adhere to the MT Summit 2025 guidelines and style templates (PDF, LaTeX, Word) and be uploaded on Easychair (https://easychair.org/my/conference?conf=mtsummit2025)
**Workshop organizers**
Janiça Hackenbuchner, University of Ghent
Luisa Bentivogli, Fondazione Bruno Kessler
Joke Daems, University of Ghent
Chiara Manna, University of Tilburg
Beatrice Savoldi, Fondazione Bruno Kessler
Eva Vanmassenhove, University of Tilburg
[View Less]
The Data Science section at the IT University of Copenhagen has an open
position for a postdoc, on the topic of *scaling up qualitative interviews
with Large Language Models* (LLMs). The project is a collaboration
with the Center
for Social Data Science <https://sodas.ku.dk/> at the university of
Copenhagen. This is a NLP position, but some relevant background in
social sciences or psychology is a plus.
*Application deadline is March 17 2025*. Applications can be submitted via
ITU job …
[View More]portal:
Application link:
https://candidate.hr-manager.net/ApplicationInit.aspx?cid=119&ProjectId=181…
IT University of Copenhagen is the leading Danish university dedicated to
various aspects of IT technology, and Copenhagen is one of the happiest and
most livable cities in the world. The candidates will be part of NLPNorth
research group <https://nlpnorth.github.io/> (with 5 full-time faculty
working in various areas of NLP), as well as Pioneer Center for AI
<https://www.aicentre.dk/>. The postdoc will be hosted by Assoc. Prof. Anna
Rogers <https://annargrs.github.io/>, to whom inquiries about the project
can be directed (arog(a)itu.dk). The domain expert on this project is Hjalmar
Alexander Bang Carlsen (sociology, hc(a)sodas.ku.dk).
--
Anna Rogers
Associate Professor
IT University of Copenhagen
http://annargrs.github.io/
[View Less]
Dear colleagues,
The scientific journal Slovenščina 2.0 (diamond open-access; indexed in
Scopus, ERIH PLUS, COBISS.si, dLib.si, DOAJ, Sherpa Romeo, Linguistic
Bibliography, and CNKI) warmly invites you to submit abstracts for a
special issue dedicated to linguistic accessibility in legal and
administrative language. Contributions may address various aspects of
linguistic accessibility, including linguistic characteristics of legal and
administrative texts, user studies, specialized language …
[View More]resources,
technologies, and tools, as well as related social and ethical issues.
Empirical and interdisciplinary studies are particularly appreciated.
Abstracts (450–550 words) may be submitted in English and Slovenian and
should be sent by *March 31, 2025*, to senja.pollak(a)ijs.si and
spela.arharholdt(a)ff.uni-lj.si.
More information: https://journals.uni-lj.si/slovenscina2/announcement.
We look forward to your contributions!
Špela Arhar Holdt
[View Less]
CfP: Deadline Extension for Challenges Proposals at ISWC 2025 (March 4)
24th International Semantic Web Conference (ISWC 2025)
Nara, Japan
November 2-6, 2025
Follow us:
Twitter/X: @iswc_conf #iswc_conf ( https://twitter.com/iswc_conf )
LinkedIn: https://www.linkedin.com/company/iswc/
Mastodon Social: https://mastodon.social/@iswc_conf
Bsky: https://bsky.app/profile/iswc-conf.bsky.social
A great way to advance the state of the art in a given domain is to create
competition. We invite …
[View More]you to propose an ISWC 2025 Challenge, in which you
define an open competition on a problem of your choice within the Semantic
Web domain.
Topics:
For ISWC 2025, Challenge proposals are invited for any challenge involving
Semantic Web tasks, including but not limited to:
-
Ontology and knowledge graph alignment
-
Ontology and knowledge graph quality assurance (QA)
-
Knowledge graph construction and refinement
-
Graph embeddings and graph neural networks
-
Query and reasoning scalability
-
Open information extraction
-
Neurosymbolic reasoning
-
Semantic Web and data mining
-
Semantic Web, machine learning, and neuro-symbolic AI
-
Link prediction
-
Question answering
-
Stream processing and reasoning
-
Semantic table understanding
-
Agents (especially using neuro-symbolic methods) and agentic workflows
-
Retrieval Augmented Generation (RAG)
Proposal Stage:
ISWC 2025 will run multiple challenges with the aim of evaluating and
comparing software solutions for the Semantic Web and Knowledge Graphs in a
systematic way. To propose a challenge, you are required to:
-
Specify a task to be addressed.
-
Provide an evaluation dataset.
-
Define evaluation measures to compare the performance of participating
systems.
Important Date:
Challenge proposal submission due March 4th, 2025
Website: https://iswc2025.semanticweb.org/#/calls/challenges
Semantic Web Challenge Chairs:
Contact: iswc2025-challenge(a)easychair.org
Mayank Kejriwal
University of Southern California, United States
Pablo Mendes
Upwork, United States
Would you like to know about the host city, Nara? Check out this blog:
https://iswc2025.semanticweb.org/#/blogs/host
--
*Dr.-Ing. **Genet Asefa Gesese*
Head of Machine Learning Department (Abteilungsleitung Maschinelles Lernen)
FIZ Karlsruhe – Leibniz Institute for Information Infrastructure, Germany
( *https://www.fiz-karlsruhe.de/en/bereiche/lebenslauf-und-publikationen-dr-ing-genet-asefa-gesese
<https://www.fiz-karlsruhe.de/en/bereiche/lebenslauf-und-publikationen-dr-in…>*
)
AND
Karlsruhe Institute of Technology (KIT)
*( https://ise.aifb.kit.edu/21_89.php
<https://ise.aifb.kit.edu/21_89.php> )*
[View Less]
The Natural Language Processing and Information Retrieval Group at the
University of Mannheim invites applications for a short-term position for
ONE RESEARCHER (all genders welcome) IN COMPUTATIONAL POLITICAL SCIENCE
/ DATA SCIENCE / COMPUTATIONAL LINGUISTICS / STATISTICAL NLP
The position is affiliated to the interdisciplinary research project
Uncovering ideological frames in political discourses (UNCOVER)
headed by Ines Rehbein [1] and Simone Ponzetto [2].
We are looking for a highly …
[View More]motivated candidate to conduct research in
the area of NLP for political science. The candidate should have a
strong interest and background in one or several of the following areas
* computational political science
* NLP / data science / text-as-data
* computational linguistics / semantic text processing
with a strong drive for interdisciplinary research and high proficiency
in German.
Candidates should have a Masters degree in Computational Linguistics,
Natural Language Processing, Computer Science, Data Science, Political
Science or related areas. The position will be located at the Data and
Web Science Group (DWS) of the University of Mannheim, one of leading
centers for Data Science in Germany (more information can be found at
http://dws.informatik.uni-mannheim.de).
Duration: The position will be available for 6 to 9 months and can be
arranged as a full or part-time position. Salary range: according to
German public scale TV-L 13.
Applications can be made per e-mail (rehbein(a)uni-mannheim.de) and should
include a short research statement, CV, copy of university degrees and
transcripts. All documents should be e-mailed as a single PDF. All
applications sent before March, 10, 2025 will receive full
consideration. The position remains open until filled.
Please feel free to contact Ines Rehbein (rehbein(a)uni-mannheim.de) for
informal enquiries.
The University of Mannheim is committed to increase the percentage of
female scientists and encourages female applicants to apply. Among
candidates of equal aptitude and qualifications, a person with
disabilities will be given preference.
Please note that threats to confidentiality and unauthorized access by
third parties cannot be ruled out when communicating by unencrypted
e-mail. The information on the collection of personal data from the data
subject pursuant to Article 13 DS-GVO can be found on our homepage:
https://www.uni-mannheim.de/universitaet/stellenanzeigen/datenschutz-bei-be….
[1]
https://www.uni-mannheim.de/dws/people/researchers/postdoctoral-research-fe…
[2]
https://www.uni-mannheim.de/dws/people/professors/prof-dr-simone-paolo-ponz…
--
Ines Rehbein
Data and Web Science Group
University of Mannheim, Germany
[View Less]
Dear Colleagues,
The 10th annual Law & Corpus Linguistics Conference will be held on Friday, October 24, 2025 at Brigham Young University's J. Reuben Clark Law School in Provo, Utah. The keynote address will be delivered by Abbe Gluck, the Alfred M. Rankin Professor of Law and Faculty Director of the Solomon Center for Health Law and Policy at Yale Law School.
Proposals are invited for individual papers and panels. We are open to submissions on a broad range of topics, including but not …
[View More]limited to:
* applications of corpus linguistics to constitutional, statutory, contract, patent, trademark, probate, administrative, and criminal law in any state or nation;
* philosophical, normative, and pragmatic commentary on the use of corpus linguistics in the law;
* triangulation between corpus linguistics and other empirical methods in legal interpretation;
* corpus-based analysis of legal discourse or topics;
* best practices in corpus design and corpus linguistic methods in legal settings.
The proposal deadline is May 15, 2025. Proposals should include an abstract of no more than 750 words and complete contact information for presenters. Please send proposals to byulawcorpus(a)law.byu.edu<mailto:byulawcorpus@law.byu.edu>.
We hope you'll consider submitting a proposal!
Sincerely,
LCL 2025 Conference Organizing Committee (Tom Lee, Jesse Egbert, Brett Hashimoto, James Heilpern, James Phillips)
[View Less]
Second Call for Research & Innovation Papers
SEMANTiCS 2025 EU
21st International Conference on Semantic Systems
Vienna, Austria
September 3 - 5, 2025
Important Dates:
-
*Abstract Submission Deadline: April 25 , 2025*
-
*Paper Submission Deadline: May 2, 2025*
-
*Notification of Acceptance: June 13, 2025*
-
*Camera-Ready Paper Deadline: July 04, 2025*
*All deadlines are set for 11:59 pm, Anywhere On Earth time (UTC-12)*
*Submissions will be through …
[View More]Easychair and the submission link will be
provided soon.*
Proceedings of SEMANTiCS 2025 EU will be made available *open access*.
Research and Innovation Track
The SEMANTiCS 2025 conference is excited to invite submissions for the
Research and Innovation Track, welcoming groundbreaking research
contributions, innovative solutions, and experimental studies relevant to
the Semantic Web, Semantic Technologies, and AI-enabled semantics. We also
encourage submissions at the intersections of these fields with other
scientific and applied disciplines, fostering cross-disciplinary exchange
and advancement. Papers should present original work that has not been
published or is not under consideration elsewhere. All submissions must
adhere to the submission guidelines, including reference formatting and any
additional documentation as required. Each submission will undergo a
rigorous review process, with at least three independent reviews,
evaluating the novelty, technical quality, reproducibility, and practical
relevance of the work.
Topics of Interest
SEMANTiCS 2025 calls for submissions of high-quality research papers across
a broad spectrum of topics in Semantic Web, Semantic Technologies, and AI.
We are particularly interested in new and emerging trends, especially where
semantic technologies intersect with evolving fields such as large language
models, explainable AI, and trustworthy data infrastructures. Topics of
interest include, but are not limited to:
- Web Semantics & Linked (Open) Data
- Enterprise Knowledge Graphs, Graph Data Management
- Machine Learning Techniques for/using Knowledge Graphs (e.g.
reinforcement learning, deep learning, data mining and knowledge discovery)
- Generative AI and Knowledge Graphs (e.g., Retrieval-Augmented
Generation (RAG) with knowledge graph integration, generative model
grounding)
- Reasoning, Rules, and Policies on RAG
- Knowledge Engineering and Management (e.g., knowledge acquisition,
extraction, integration, and publication workflows)
- Terminology, Thesaurus & Ontology Management, Ontology engineering
- Web agents
- Natural Language Processing for/using Knowledge Graphs (e.g. entity
linking and resolution using target knowledge such as Wikidata and DBpedia,
foundation models)
- Crowdsourcing for/using Knowledge Graphs
- Data Quality Management and Assurance
- Mathematical and Logical Foundations of Knowledge-aware AI
- Multimodal Knowledge Graphs (e.g., text, image, audio fusion in graph
structures)
- Semantic-Enhanced Data Science Pipelines and Processes
- Semantics in Blockchain environments (e.g., traceability,
decentralized knowledge representation)
- Trust, Data Privacy, and Security with Semantic Technologies
- Internet of Things (IoT), Stream Processing, and Temporal Data
Management (e.g., real-time semantic processing and predictive analytics)
- Conversational AI and Dialogue Systems powered by Knowledge Graphs
- Provenance and Data Change Tracking (e.g., semantic versioning, data
updates in distributed settings)
- Semantic Interoperability (e.g., cross-domain standards, mapping
frameworks, ontology alignment)
- Linked Data storage, triple stores, graph databases
- Robust, Scalable, and Fault-Tolerant Semantic Data Systems (e.g.,
distributed querying, optimization)
- User Interfaces and Usability of Semantic Technologies (e.g.,
visualizations, intelligent user interaction)
- Explainable and Interoperable AI
- Decentralised and Federated Knowledge Graphs (e.g., federated
querying, link traversal)
Applied Semantic Technologies and AI in Real-World Scenarios, such as, but
not limited to:
- Biomedicine and Health (e.g., Knowledge Graphs for biomedical
applications, AI-driven diagnostics, personalized health)
- AI for Environmental and Climate Solutions (e.g., semantic modeling
for environmental impact, biodiversity knowledge graphs)
- Scientific Knowledge Graphs and Open Science (e.g., FAIR data
principles, enhanced scholarly communication)
- Semantic Technologies in GLAM (Galleries, Libraries, Archives, and
Museums)
- Knowledge Graphs and Hybrid AI for Industry 4.0/5.0 and Predictive
Maintenance
- Digital Humanities and Cultural Heritage Preservation
- Legal Technology, AI Ethics, and Regulatory Compliance (e.g., AI and
legal frameworks, semantic-enabled compliance with the EU AI Act)
- Economics and Governance of Data Ecosystems (e.g., data marketplaces,
semantic service interoperability, data policy)
Submissions will be through Easychair. Stay tuned for the submission link.
For *Submission Guidelines* and *Review and Evaluation Criteria* please
head to the online call for papers:
*https://2025-eu.semantics.cc/page/cfp_rev_rep*
<https://2025-eu.semantics.cc/page/cfp_rev_rep>.
We would highly appreciate it if you could disseminate this call within
your network.
*We look forward to receiving your contributions!*
Research and Innovation Track Chairs
Blerina Spahiu (University of Milano-Bicocca, IT)
Mehdi Ali (Lamarr Institute & Fraunhofer IAIS, Germany)
Kind Regards,
On behalf of the organising committee.
=========================
Dr. Beyza Yaman
ADAPT Centre, Trinity College Dublin
[View Less]
Dear all,
University of Turku, Finland, invites applications for **fully funded doctoral researcher positions in digital linguistics and NLP, in the TurkuNLP research group**.
The positions are part of the Human-Centric Artificial Intelligence for a Sustainable Future (HAIF) doctoral training programme, co-funded by the European Union’s Horizon Europe research and innovation programme’s Marie Skłodowska-Curie Actions (MSCA) (grant agreement No 101177564).
Please find more information on …
[View More]the program and application procedure below!
Best,
Veronika Laippala
HAIF is a doctoral programme for 25 doctoral researchers hosted by the University of Turku. HAIF unites research groups from the fields of computing, materials science, social sciences, law, humanities, and health sciences. HAIF doctoral researchers will conduct novel research under the guidance of scientifically distinguished and experienced supervisors. By connecting 11 research groups from 5 faculties, HAIF aims to create a cohesive, synergistic, and interdisciplinary research community.
The global network of associated partners, representing various industries and third-sector organisations, in addition to universities and research institutes, are committed to supporting the HAIF doctoral training. This training features 4–6-month secondments during the years 2027–2029 and a variety of other educational activities.
HAIF’s objective is to provide training that promotes the safe, secure, legally, and ethically sustainable use of AI, with an emphasis on humans as developers, users, and decision-subjects affected by the technology. In addition to individual and societal perspectives, HAIF’s research themes delve into the technological properties that shape interactions between humans and AI systems (e.g., transparency, interpretability, reliability, and accountability).
HAIF doctoral researchers will be selected through a transparent, merit-based process with equity and inclusivity as cross-cutting priorities.
The call for applications for the HAIF doctoral researcher positions is open until Friday, 28 February, at 15:00 (Helsinki/Europe).
An eligible applicant is an early-stage researcher (no doctoral degree) who holds a Master’s degree in a relevant discipline. Based on the EU mobility rule, the applicant must not have resided in Finland for more than 12 months in the last 36 months.
Read more about HAIF and find the guide for applicants on the website: https://haif.utu.fi/
Here is the direct link to the online application system: https://ats.talentadore.com/apply/25-doctoral-researcher-positions-in-msca-…
[View Less]
*THE 1ST EUROPEAN CL RESEARCH SUMMIT FOR JUNIOR RESEARCHERS*
We invite junior researchers in computational linguistics and NLP to
join us for an exciting week in the Swiss mountains!
This call invites participants to apply for the summit by completing a
non-binding registration form. Upon acceptance, participants will
receive a final binding registration form.
*Registration Form*
Please register via the following registration form:
https://forms.gle/kZpFnbGETwAXgxbWA <https://…
[View More]forms.gle/kZpFnbGETwAXgxbWA>
*About KnitTogether:*
Great research arises from amazing collaborations. However, establishing
collaborations is challenging, and it depends on the context of the
institution, the supervision, existing networks, and one’s personality.
With *KnitTogether25*, we invite NLP researchers, particularly junior
ones who work on *Bias in NLP and Its Societal Impacts,* to spend an
exciting week in the Swiss mountains.
In a growing community dealing with a topic that is gaining increasing
societal influence and relevance, we want to join forces and exchange
ideas. What can we learn from each other?
In particular, but not limited to, we would like to explore: a) Which
groups are under-/overrepresented or represented to their disadvantage?
b) How can we incorporate such factors into modeling? c) How can we
analyze them? We would like to hear the different perspectives of the
participants: How do you relate your work to the topic? Using innovative
and playful methods, we want to learn more about the work of our peers
and thus get to know each other better /scientifically/.
During a week full of hikes and other fun activities, the goal is to get
to know each other without hurdles, draft ideas, and discuss research.
We want to create a safe space where everyone can openly discuss their
work and ideas without academic pressure.
Program
* Workshops
* Peer-to-peer tutorials
* Keynotes
* Mentoring sessions
* Brainstorming sessions
* Social activities
*Important Dates*
* *February 28, 2025 (AoE)*: registration deadline
* *May 5-10, 2025:* Retreat Date
Workshop Location
* Wildhaus, Switzerland
*Workshop Fees*
* *Registration fee:* ~ 60 Euros (includes accommodation and food,
excludes travel costs).
* Application for *additional travel subsidies* is possible.
Website
* https://knittogether.github.io/kt25/
<https://knittogether.github.io/kt25/>
*Organizers*
* Esra Dönmez (University of Stuttgart)
* Neele Falk (University of Stuttgart)
* Florian Schneider (University of Hamburg)
* Andreas Waldis (TU Darmstadt, University of Lucerne)
*Contact*
For any questions, please contact the organizers at
/knittogethernlp(a)gmail.com/ <mailto:knittogethernlp@gmail.com>
Best Regards,
KnitTogether25 Organizers
*---------------------------------------------*
*Sponsors*
We would like to thank*IRIS*(Interchange Forum for Reflecting on
Intelligent Systems),*HSLU*(University of Lucerne) and*GSCL*(German
Society for Computational Linguistics and Language Technology) for
sponsoring this event.
[View Less]
<Apologies for cross-postings>
------------------------------------------------
Call for Participation
PROFE 2025: Language Proficiency Evaluation
IberLEF 2025 Shared Task
https://nlp.uned.es/question-answering/profe2025
PROFE 2025 reuses the exams for Spanish proficiency evaluation developed by Instituto Cervantes over many years to evaluate human students. Therefore, automatic systems will be evaluated under the same conditions as humans were. Systems will receive a set of exercises …
[View More]with their corresponding instructions without specific training material. In this way we expect Transfer Learning approaches or the use of Generative Large Language Models.
Subtasks
PROFE 2025 has three subtasks, one per exercise type. Teams can participate in any combination of them. Each subtask contains several exercises of the same type. The subtasks are:
1.
Multiple choice subtask: each exercise includes a text and a set of multiple-choice questions about the text where only one answer is correct. Given a multiple-choice question, systems must select the correct answer among the candidates.
2.
Matching subtask: each exercise contains two sets of texts. Systems must find the text in the second set that best matches the first set. There is only one possible matching per text, but the first set can contain extra unnecessary texts.
3.
Filling the gap subtask: each exercise contains a text with several gaps corresponding to textual fragments that have been removed and presented disorderly as options. Systems must determine the correct position for each fragment. There is only one correct text per gap, but there could be more candidates than gaps.
The different exercises open research on how to approach them, adapting different prompts when using generative models.
Dataset
We will use the IC-UNED-RC-ES dataset created from real examinations at Instituto Cervantes. These exams were created by human experts to assess language proficiency in Spanish. We have already collected the exams and converted them to a digital format, which is ready to be used in the task. The dataset contains exams at different levels (from A1 to C2).
The complete dataset contains 282 exams with 855 exercises. The total number of evaluation points are 6146 (among 16570 options) distributed by exercise type as:
multiple-choice: 3544 responses
matching: 2309 responses
fill-the-gap: 293 responses
In PROFE 2025 we plan to use around 50% of the exams, so the other 50% remains hidden for PROFE second edition.
We intend not to distribute the gold standard to prevent overfitting in post-campaign experiments and data contamination in LLMs.
Evaluation measures and baseline
We will use traditional accuracy (proportion of correct answers) as the main evaluation measure. Systems will receive evaluation scores from two different perspectives:
*
At the question level, where correct answers are counted individually without grouping them.
*
At the exam level, where scores for each exam are considered. Each exam contains several exercises of different types. An exam is considered to be passed if an accuracy score (accounted as the proportion of correct answers) above 0.5 is reached. Then, the proportion of passed exams is given as a global score. This perspective will only apply to those teams participating in the three subtasks.
More in detail, the exact evaluation per subtask is as follows:
*
Multiple choice subtask: we will measure accuracy as the proportion of questions correctly answered
*
Matching subtask: we will measure accuracy as the proportion of correct texts matched.
*
Fill in the gap subtask: We will measure accuracy as the proportion of correctly filled gaps.
We will use accuracy as the evaluation measure because there is only one correct option among candidates and because it is the measure applied to humans doing the same exams. Thus, we can compare the performance of automatic systems and humans under the same conditions
A preliminary baseline using ChatGPT obtains the following results for each exercise type (provided that different prompting can produce slightly different results):
*
Multiple choice accuracy: 0.64
*
Filling the gap accuracy: 0.43
*
Matching accuracy: 0.51
Schedule
February 6, 2025 Registration opens
March 10, 2025 Training data released
April 28, 2025 Test set release
May 9, 2025 Deadline for submitting runs
May 14, 2025 Release of evaluation results
June 3, 2025 Paper submission deadline
Organizers
Alvaro Rodrigo<https://www.uned.es/universidad/docentes/informatica/alvaro-rodrigo-yuste.h…>, UNED NLP & IR Group (Universidad Nacional de Educación a Distancia)
Anselmo Peñas<https://www.uned.es/universidad/docentes/informatica/anselmo-penas-padilla.…>, UNED NLP & IR Group (Universidad Nacional de Educación a Distancia)
Alberto Pérez<https://www.uned.es/universidad/docentes/informatica/alberto-perez-garcia-p…>, UNED NLP & IR Group (Universidad Nacional de Educación a Distancia)
Sergio Moreno<https://www.uned.es/universidad/docentes/en/informatica/sergio-moreno-alvar…>, UNED NLP & IR Group (Universidad Nacional de Educación a Distancia)
Javier Fruns, Instituto Cervantes
Inés Soria, Instituto Cervantes
Rodrigo Agerri<https://ragerri.github.io/>, HiTz (Universidad del País Vasco, UPV/EHU)
AVISO LEGAL. Este mensaje puede contener información reservada y confidencial. Si usted no es el destinatario no está autorizado a copiar, reproducir o distribuir este mensaje ni su contenido. Si ha recibido este mensaje por error, le rogamos que lo notifique al remitente.
Le informamos de que sus datos personales, que puedan constar en este mensaje, serán tratados en calidad de responsable de tratamiento por la UNIVERSIDAD NACIONAL DE EDUCACIÓN A DISTANCIA (UNED) c/ Bravo Murillo, 38, 28015-MADRID-, con la finalidad de mantener el contacto con usted. La base jurídica que legitima este tratamiento, será su consentimiento, el interés legítimo o la necesidad para gestionar una relación contractual o similar. En cualquier momento podrá ejercer sus derechos de acceso, rectificación, supresión, oposición, limitación al tratamiento o portabilidad de los datos, ante la UNED, Oficina de Protección de datos<https://www.uned.es/dpj>, o a través de la Sede electrónica<https://sede.uned.es/> de la Universidad.
Para más información visite nuestra Política de Privacidad<https://descargas.uned.es/publico/pdf/Politica_privacidad_UNED.pdf>.
[View Less]