August 2023 - Corpora

Research Position on Computational Models of Misunderstanding in Germany
by mroth＠coli.uni-saarland.de 28 Aug '23

28 Aug '23

The independent research group on "Computational Models of Misunderstanding for Complex Instructional Text" invites applications for one research associate. The position is funded through a grant in the Emmy Noether Programme of the German Research Foundation (DFG---Deutsche Forschungsgemeinschaft), which funds projects similar to an ERC Starting Grant or NSF CAREER Award. The group is headed by Dr. Michael Roth and currently hosted by the Institute for Natural Language Processing ("IMS") at the University of Stuttgart, Germany [1 <https://www.ims.uni-stuttgart.de/en/institute/researchgroups/mist/>]. The project is concerned with the systematic analysis and computational modelling of text passages that can lead to misunderstandings. A substantial amount of previous work has studied misunderstandings in dialogue, but suitable resources for written language are scarce because misunderstandings cannot be observed directly from a text. Since readers and writers typically do not interact, it is important for authors to ensure that texts leave no room for misinterpretation. Otherwise, for example, medical instructions may be followed incorrectly, and route directions may not guide navigators to their desired destination. The announced position plays a key role in the project's final phase, leveraging previously created resources (e.g. [2 <https://aclanthology.org/2022.lrec-1.354/>,3 <https://aclanthology.org/2020.lrec-1.702/>]) and connecting to the group's award-winning earlier work (e.g. [4 <https://aclanthology.org/2021.eacl-srw.5/>,5 <https://aclanthology.org/2022.semeval-1.146/>]). Potential areas of focus for the successful candidate include delving deeper into specific linguistic factors that may lead to misunderstandings (such as elements of implicit or underspecified language), enhancing classification models by incorporating additional information (such as commonsense knowledge or multi-modal context), and/or testing these models in practical applications (such as question answering or machine translation). The position is initially available until February 2025, with a start date as soon as possible (e.g. December 2023) and the possibility of extension (for a total of at least 2 years). Compensation will be in accordance with the German TV-L E13 salary scale at 100% (approx. 4,000 EUR *gross* per month). Successful applicants will have obtained a Ph.D. (or are close to completing their thesis) in computational linguistics, machine learning, or a closely related field, with a particular interest in semantics and pragmatics or downstream applications. Programming skills and the ability to work in a team are taken for granted. The candidate should be able to work and communicate in English (no proficiency of German is required). Applications should include a motivation letter including research interests, a CV, a list of publications and contact information of up to three references. Applications should be sent *as a single PDF file* to Michael Roth by email. Applications received by 23 September 2023 will receive full consideration, but the position will remain open until filled. Candidates who identify as female, LGBTQ+ and/or as members of any underrepresented group are particularly encouraged to apply. Feel free to contact Michael Roth (head of group) or Nicola Fanton (PhD student) for any question regarding the group or position. [1] https://www.ims.uni-stuttgart.de/en/institute/researchgroups/mist/ [2] https://aclanthology.org/2022.lrec-1.354/ [3] https://aclanthology.org/2020.lrec-1.702/ [4] https://aclanthology.org/2021.eacl-srw.5/ (best student paper) [5] https://aclanthology.org/2022.semeval-1.146/ (best task description paper) -- Dr. Michael Roth Emmy Noether Group Leader Institute for Natural Language Processing University of Stuttgart

2 1

Job : Postdoc (12 months), NLP and gender stereotypes in the French media, Universite Grenoble Alpes, France
by François Portet 25 Aug '23

25 Aug '23

Call for postdoc applications in Natural Language Processing for the automatic detection of gender stereotypes in the French media (Grenoble Alps University, France) Starting date: flexible, November 30, 2023, at the latest Duration: full-time position for 12 months Salary: according to experience (up to 4142€/ month) Application Deadline: Open until filled Location: The position will be based in Grenoble, France. This is not a remote work. Keywords: natural language processing, gender stereotypes bias, corpus analysis, language models, transfer learning, deep learning *Context* The University of Grenoble Alps (UGA) has an open position for a highly motivated postdoc researcher to joint the multidisciplinary GenderedNews project. Natural Language Processing models trained on large amount of on-line content, have quickly opened new perspectives to process on-line large amount of on-line content for measuring gender bias in a daily basis (see our project https://gendered-news.imag.fr/ <https://gendered-news.imag.fr/> ). Regarding research on stereotypes, most recent works have studied Language Models (LM) from a stereotype perspective by providing specific corpora such as StereoSet (Nadeem et al., 2020) or CrowS-Pairs (Nangia et al. 2020). However, these studies are focusing on the quantifying of bias in the LM predictions rather than bias in the original data (Choenni et al., 2021). Furthermore, most of these studies ignore named entities (Deshpande et al., 2022) which account for an important part of the referents and speakers in news. In this project, we intend to build corpora, methods and NLP tools to qualify the differences between the language used to describe groups of people in French news. *Main Tasks* The successful postdoc will be responsible for day-to-day running of the research project, under the supervision of François Portet (Prof UGA at LIG) and Gilles Bastin (prof UGA at PACTE). Regular meetings will take place every two weeks. - Defining the dimensions of stereotypes to be investigated and the possible metrics that can be processed from a machine learning perspective. - Exploring, managing and curating news corpora in French for stereotypes investigation, with a view to making them widely available to the community to favor reproducible research and comparison. - Studying and developing new computational models to process large number of texts to reveal stereotype bias in news. Make use of pretrained models for the task. - Evaluate the methods on curated focused corpus and apply it to the unseen real longitudinal corpus and analyze the results with the team. - Preparing articles for submission to peer-reviewed conferences and journals. - Organizing progress meetings and liaising between members of the team. The hired person will interact with PhD students, interns and researchers being part of the GenderedNews project. According to his/her background his/her own interests and in accordance with the project's objective, the hired person will have the possibility to orient the research in different directions. *Scientific Environment* The recruited person will be hosted within the GETALP teams of the LIG laboratory (https://lig-getalp.imag.fr/ <https://lig-getalp.imag.fr/>), which offers a dynamic, international, and stimulating environment for conducting high-level multidisciplinary research. The person will have access to large datasets of French news, GPU servers, to support for missions as well as to the scientific activities of the labs. The team is housed in a modern building (IMAG) located in a 175-hectare landscaped campus that was ranked as the eighth most beautiful campus in Europe by the Times Higher Education magazine in 2018. The person will also closely work with Gilles Bastin (PACTE, a Sociology lab in Grenoble) and Ange Richard (PhD at LIG and PACTE). The project also includes an informal collaboration with "Prenons la une" (https://prenonslaune.fr/ <https://prenonslaune.fr/>) a journalists’ association which promotes a fair representation of women in the media. *Requirements* The candidate must have a PhD degree in Natural Language Processing or computer science or in the process of acquiring it. The successful candidate should have - Good knowledge of Natural Language Processing - Experience in corpus collection/formatting and manipulation. - Good programming skills in Python - Publication record in a close field of research - Willing to work in multidisciplinary and international teams - Good communication skills - Good mastering of French is required *Instructions for applying* Applications will be considered on the fly and must be addressed to François Portet (Francois.Portet(a)imag.fr <mailto:Francois.Portet@imag.fr>). It is therefore advisable to apply as soon as possible. The application file should contain - Curriculum vitae - References for potential letter(s) of recommendation - One-page summary of research background and interests for the position - Publications demonstrating expertise in the aforementioned areas - Pre-defense reports and defense minutes; or summary of the thesis with the date of defense for those currently in doctoral studies *References* Deshpande et al. (2022). StereoKG: Data-Driven Knowledge Graph Construction for Cultural Knowledge and Stereotypes. arXiv preprint arXiv:2205.14036. Choenni et al. (2021). Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you? arXiv preprint arXiv:2109.10052. Nadeem et al. (2020) StereoSet: Measuring stereotypical bias in pretrained language models. ArXiv. Nangia et al. (2020) CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In EMNLP2020. -- François PORTET Professeur - Univ Grenoble Alpes Laboratoire d'Informatique de Grenoble - Équipe GETALP Bâtiment IMAG - Office 333 700 avenue Centrale Domaine Universitaire - 38401 St Martin d'Hères FRANCE Phone: +33 (0)4 57 42 15 44 Email:francois.portet@imag.fr www:http://membres-liglab.imag.fr/portet/

1 1

LAST CFP: ICNLSP 2023, 6th International Conference on Natural Language and Speech Processing, December 16-17, 2023
by Mourad Abbas 25 Aug '23

25 Aug '23

Last Call for Papers (06 days left for submission) 6th International Conference on Natural Language and Speech Processing <http://icnlsp.org/2023welcome> We are delighted to invite you to ICNLSP 2023, which will be held virtually from December 16th to 17th, 2023. ICNLSP 2023 offers the opportunity for attendees (researchers, academics and students, and industrials) to share their ideas and to connect to each other and make them up to date on the ongoing research in the field. ICNLSP 2023 aims to attract contributions related to natural language and speech processing. Authors are invited to present their work relevant to the topics of the conference. The following list includes the topics of ICNLSP 2023 but not limited to: Signal processing, acoustic modeling. Architecture of speech recognition system. Deep learning for speech recognition. Analysis of speech. Paralinguistics in Speech and Language. Pathological speech and language. Speech coding. Speech comprehension. Summarization. Speech Translation. Speech synthesis. Speaker and language identification. Phonetics, phonology and prosody. Cognition and natural language processing. Text categorization. Sentiment analysis and opinion mining. Computational Social Web. Arabic dialects processing. Under-resourced languages: tools and corpora. New language models. Arabic OCR. Lexical semantics and knowledge representation. Requirements engineering and NLP. NLP tools for software requirements and engineering. Knowledge fundamentals. Knowledge management systems. Information extraction. Data mining and information retrieval. Machine translation. NLP for Arabic heritage documents. *IMPORTANT DATES* Submission deadline: *31 August 2023* Notification of acceptance: *31 October 2023* Camera-ready paper due: *20 November 2023* Conference dates: *16, 17 December 2023* *PUBLICATION* 1- All accepted papers will be published in ACL Anthology ( https://aclanthology.org/venues/icnlsp/). 2- Selected papers will be published in Signals and Communication Technology (Springer) (https://www.springer.com/series/4748), indexed by Scopus and zbMATH. For more details, visit the conference website: https://www.icnlsp .org/2023welcome *CONTACT* icnlsp(at)gmail(dot)com Best regards, Mourad Abbas

1 0

[CfP] Second Workshop on Information Extraction from Scientific Publications (WIESP) at IJCNLP-AACL 2023
by Tirthankar Ghosal 25 Aug '23

25 Aug '23

*** Second Workshop on Information Extraction from Scientific Publications (WIESP) at IJCNLP-AACL 2023 *** *** Website: https://ui.adsabs.harvard.edu/WIESP/2023/ *** *** Twitter: https://twitter.com/wiesp_nlp *** Building on the success of the First WIESP at AACL-IJCNLP 2022, the Second Workshop on Information Extraction from Scientific Publications (WIESP) will provide a platform to researchers to foster discussion and research on information extraction, mining, generation, and knowledge discovery from scientific publications using Natural Language Processing and Machine Learning techniques. A lot of technological change happened in one year (since the 1st WIESP), especially with Generative Artificial Intelligence research. We are incorporating a few additional topics to stay abreast with the latest developments and research in the community. The 2nd iteration of WIESP would focus on the following topics (but not limited to): - Large Language Models (LLMs) for Science - Application of LLMs on information extraction, generation, mining and knowledge discovery from scientific publications - Probing LLMs for scientific fact checking and misinformation - Scientific document parsing - Scientific named-entity recognition - Scientific article summarization - Question-answering on scientific articles - Citation context/span extraction - Structured information extraction from full-text, tables, figures, bibliography - Novel datasets curated from scientific publications - Argument extraction and mining - Challenges in information extraction from scientific articles - Building knowledge graphs via mining scientific literature; querying scientific knowledge graphs - Novel tools for IE on scientific literature and interaction with users - Mathematical information extraction - Scientific concepts, facts extraction - Visualizing scientific knowledge - Bibliometric and Altmetric studies via information extraction from scientific articles and metadata In addition to research paper presentations, WIESP will also feature keynote talks, a panel discussion on “Large Language Models and Scientific Literature Mining'', and shared tasks. We will update the details on our website as and when they become available. We especially welcome participation from academic and research institutions, government and industry labs, publishers, and information service providers. Projects and organizations using NLP/ML techniques in their text mining and enrichment efforts are also welcome to participate. We strongly encourage participation of students, researchers, and science practitioners from diverse backgrounds, especially from underrepresented groups and communities, to be a part of WIESP events, and pro-actively make the workshop a diverse and inclusive one. ***Call for Papers*** We invite papers of the following categories: ***Long papers*** must describe substantial, original, completed, and unpublished work. Wherever appropriate, concrete evaluation and analysis should be included. Papers must not exceed eight (8) pages of content, plus unlimited pages of references. The final versions of long papers will be given one additional page of content (up to 9 pages) so that reviewers' comments can be taken into account. ***Short papers*** must describe original and unpublished work. Please note that a short paper is not a shortened long paper. Instead, short papers should have a point that can be made in a few pages, such as a small, focused contribution, a negative result, or an interesting application nugget. Short papers must not exceed four (4) pages, plus unlimited pages of references. The final versions of short papers will be given one additional page of content (up to 5 pages) so that reviewers' comments can be taken into account. In addition to papers, WIESP will also host shared tasks. More details on the WIESP shared tasks will be available on our website shortly. Also, we will publish separate CfPs on the shared tasks. Shared task authors will be invited to write their system descriptions and those will be subjected to peer review. ***Shared Task: Function of Citation in Astrophysics Literature (FOCAL)*** The citation graph is an essential tool for helping researchers find relevant literature. To further empower discovery, we aim to label the edges of the graph with the function of the citation: e.g. is the cited work necessary background knowledge, or is it used as a comparison, to the citing work? To start this process, we propose a shared task of automatically labeling citations with a function based on the textual context of the citation. A sample dataset and more instructions can be found at: https://ui.adsabs.harvard.edu/WIESP/2023/SharedTasks All accepted papers would be published in the WIESP proceedings as part of IJCNLP-AACL 2023 and indexed in the ACL Anthology. ***Important Dates*** - Paper Submission Deadline: August 25, 2023 - Notification of workshop paper/abstract acceptance: October 2, 2023 - Camera-ready Submission Deadline: October 15, 2023 - Workshop: November 2-4, 2023 (online, final date TBD) ***All submission deadlines are 11.59 pm UTC -12h ("Anywhere on Earth")*** ***Submission Website and Format*** Submission Link: TBD (please have an eye on the website) Submission will be via softconf. Submissions should follow the ACLPUB formatting guidelines (https://acl-org.github.io/ACLPUB/formatting.html) and template files (https://github.com/acl-org/acl-style-files/tree/master). Submissions (Long and Short Papers) will be subject to a double-blind peer-review process. We follow the same policies as IJCNLP-AACL 2023 regarding preprints and double submissions. The anonymity period for WIESP 2023 is from July 25 to August 25. ***Organizers*** - Tirthankar Ghosal, National Center for Computational Sciences, Oak Ridge National Laboratory, USA - Felix Grezes, Center for Astrophysics | Harvard & Smithsonian, USA - Thomas Allen, Center for Astrophysics | Harvard & Smithsonian, USA - Kelly Lockhart, Center for Astrophysics | Harvard & Smithsonian, USA - Alberto Accomazzi, Center for Astrophysics | Harvard & Smithsonian, USA -- +++++++++++++++++++++++++++++++++++ Tirthankar Ghosal https://member.acm.org/~tghosal ++++++++++++++++++++++++++++++++++++

1 1

2023 Annual Sinclair Lecture - delivered by Professor Susan Hunston OBE
by Michaela Mahlberg 25 Aug '23

25 Aug '23

Dear all, You are invited to join us at the University of Birmingham for the 2023 Annual Sinclair Lecture! This year’s lecture will be delivered by Professor Susan Hunston OBE Date: Mo 11th September 2023, 6pm Title: “From Pattern to System: an exploration in lexical grammar” Abstract: The approaches to lexis and grammar pioneered by Michael Halliday and by John Sinclair can be seen as conflicting or as complementary. For the most part they have developed independently of each other. This lecture reports on a project to bring the two approaches together and to populate systemic networks with constructions derived from pattern grammar. It will show that the concept of system is invaluable in demonstrating a hierarchy of constructions derived from patterns, and then in explicating choices available to speakers in specific areas of meaning. The paper will also consider the possible practical applications of this work. We are back to an in-person event this year – so you are invited to raise a glass with us after the lecture! The event is free, but please register here: https://www.birmingham.ac.uk/schools/edacs/departments/englishlanguage/even… We are very much looking forward to seeing many old and new corpus linguistic friends! Michaela --------------- Professor Michaela Mahlberg Director of the Centre for Corpus Research University of Birmingham Edgbaston Birmingham B15 2TT, UK Editor of the International Journal of Corpus Linguistics https://benjamins.com/catalog/ijcl Host of the Life and Language Podcast https://anchor.fm/michaela-mahlberg/ @MichaMahlberg<https://twitter.com/MichaMahlberg> http://www.birmingham.ac.uk/staff/profiles/elal/mahlberg-michaela.aspx https://www.turing.ac.uk/people/researchers/michaela-mahlberg https://www.linkedin.com/in/michaela-mahlberg/

1 0

GEM flash shared task summarisation/D2T: expression of interest
by Simon Mille 25 Aug '23

25 Aug '23

Dear colleague, We are collecting interests for the GEM v3 "flash" shared task, which will focus on text summarisation and data-to-text generation in a variety of languages. The submission deadline is expected to be around the end of September 2023, and we will provide a human evaluation of each participant’s main submission in the following weeks. Please fill out the very brief survey <https://forms.office.com/Pages/ResponsePage.aspx?id=EhAze6GHFkqLD6RgWx89f3L…> to indicate which task(s) and language(s) you would be interested in. More details on the datasets and modalities will be communicated soon. Thank you! simon More about GEM: https://gem-benchmark.com/

1 0

Employment opportunities: Research Assistant Position for corpora and data mining at Hong Kong Baptist University
by janice pan 25 Aug '23

25 Aug '23

Job title: Research Assistant for Corpora and Data Mining (PR0113/23-24) The original post can be seen via searching "PR0113/23-24" on https://hro.hkbu.edu.hk/index.php?page_id=6&menu_id=222&f=job *FACULTY OF ARTSDEPARTMENT OF TRANSLATION, INTERPRETING, AND INTERCULTURAL STUDIESResearch Assistant for Corpora and Data Mining (PR0113/23-24)* The Department invites applications for a full-time Research Assistant position for the project "Re-testing the Universals: Mining Interpreting Data through Re-Engineered Mega-Size Corpora". This pioneering project is at the intersection of corpus-based interpreting studies, natural language processing, and text mining. The project involves working with three significant corpora, namely the Chinese/ English Political Interpreting Corpus (CEPIC), the European Parliament Interpreting Corpus (EPIC), and the European Parliament Interpreting Corpus Ghent (EPICG). The successful candidate will be involved in the analysis of these corpora in languages including Chinese, English, Italian, French, Dutch. *Responsibilities:* - Primary focus on data cleaning, preparation, and initial analysis of the CEPIC, EPIC, and EPICG corpora across multiple languages; - Collaborative work on hypothesis and methodology development; and - Application of natural language processing techniques and text mining to analyse and draw insights from the corpora. *Requirements:* - A bachelor's degree or above in a related field such as Translation Studies, Linguistics, Data Science, or Computational Linguistics. Recent university graduates are also encouraged to apply; - Proficiency in text mining and programming languages such as Python and R, or tools such as SPSS, for data processing and analysis; - Basic competence in at least two of the languages used in the project: Chinese, English, Italian, French and Dutch; - Prior experience or familiarity with the CEPIC, EPIC, and EPICG corpora will be a significant advantage; and - A strong background in computational linguistics or a related field is preferred. The initial appointment will be made on a fixed-term contract for one year. Re-appointment thereafter will be subject to mutual agreement and funding availability. Salary Range: HK$20,000-24,000 per month *Salary will be commensurate with qualifications and experience.* For inquiries about the post, please contact the project investigator, Dr. Janice Pan at janicepan(a)hkbu.edu.hk. *Application Procedures:* Applicants are invited to submit their applications at the HKBU e-Recruitment System (https://jobs.hkbu.edu.hk). Those who are not invited for interview 8 weeks after the submission of the application may consider their applications unsuccessful. Details of the University's Personal Information Collection Statement can be found at https://hro.hkbu.edu.hk/pics. The University reserves the right not to make an appointment for the post advertised, and the appointment will be made according to the terms and conditions applicable at the time of the offer. *Review of applications is ongoing until the position is filled.* *Apply Now* <https://jobs.hkbu.edu.hk/hrssers/persrequest?job_no=PR0113/23-24>

1 0

everything in NLP is statistical now, or what this list is for
by Stan Szpakowicz 25 Aug '23

25 Aug '23

Hello. Gully Burns wrote this in response to a few unnecessary ad-hominem posts: /I was shocked to see a vitriolic ad-hominem attack on a colleague posted to this mailing list. It is entirely inappropriate to post this type of diatribe against an individual even though someone might disagree with either the tone or the content of an individual's messages or arguments. The fact that other members of the community chimed in to reinforce the attack is also appalling and entirely inappropriate./ I agree with Gully. Ada Wan's posts may be long, and clearly are not to everyone's liking, but they speak to a phenomenon that is troubling enough to merit a discussion. Symbolic processing in NLP is utterly outmoded. Number crunching has taken overcompletely, and linguistic reflection seems to be considered unnecessary. This may be an opening for scientific dodginess. Ada advocates, and very reasonably, for good scientific practice, so one wonders why anyone would be riled up. This may be my optimism, but the Corpora List may be the best forum that NLP has today. It is not only a place for announcing conferences and workshops, and advertising open positions. It is, and it should be, a place for an exchange of ideas. -- Stan Szpakowicz, PhD, DSc, Emeritus Professor EECS, Computer Science, University of Ottawa

4 4

[FinNLP@IJCNLP-AACL] Hybrid and New Dataset for ML-ESG-2
by CHUNG-CHI CHEN 25 Aug '23

25 Aug '23

Dear Colleagues, I would like to share two news of FinNLP@IJCNLP-AACL-2023 ( https://sites.google.com/nlg.csie.ntu.edu.tw/finnlp2023/home) with you. (1) We have decided to host FinNLP@IJCNLP-AACL in a hybrid format. While we encourage in-person attendance, we understand the need for flexibility and will also be accommodating live online presentations. The submission deadline is Sep. 9th. For more details, please refer to our call for paper: https://sites.google.com/nlg.csie.ntu.edu.tw/finnlp2023/call-for-papers (2) We're excited to announce the release of a new dataset in Japanese for the Multi-Lingual ESG Impact Type Identification (ML-ESG-2) shared task. If this area of research interests you, registration is still open. For more details, please visit: https://sites.google.com/nlg.csie.ntu.edu.tw/finnlp2023/shared-task-esg-imp… . Looking forward to seeing you at FinNLP@IJCNLP-AACL in November. Warm regards, Chung-Chi --- 陳重吉 (Chung-Chi Chen), Ph.D. Researcher Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology, Japan E-mail: c.c.chen(a)acm.org Website: http://cjchen.nlpfin.com

1 0

Paper DL extension: The 4th Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP 2023)
by Juri Opitz 24 Aug '23

24 Aug '23

Dear colleagues, due to popular demand, the deadline for regular paper submissions to the 4th Workshop on Evaluation and Comparison for NLP systems (Eval4NLP) at AACL 2023 will be extended by 1 week. The new submission deadline is **September 1**. Please note that Eval4NLP also allows submission of pre-reviewed papers *together* with their reviews and explanation of possible refinements. These can be submitted up to one month later. For details, please see below or visit the Eval4NLP webpage: https://eval4nlp.github.io/2023/cfp.html. ----------------------------------------------------------------------------------------- The 4th Workshop on Evaluation and Comparison for NLP systems (Eval4NLP), co-located at the 2023 Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (AACL 2023), invites the submission of long and short papers, with a theoretical or experimental nature, describing recent advances in system evaluation and comparison in NLP. ** Important Dates ** All deadlines are 11.59 pm UTC -12h (“Anywhere on Earth”). - Direct submission to Eval4NLP: ** September 1 ** - Submission of pre-reviewed papers to Eval4NLP (see below for details) : September 25 - Notification of acceptance: October 2 - Camera-ready papers due: October 10 - Workshop day: November 1 Please see the Call for Papers for more details: https://eval4nlp.github.io/2023/cfp.html. ** Special topic of this year’s workshop ** This year's edition of the Eval4NLP workshop puts a focus on the evaluation of and through large language models (LLMs). Notably, the workshop will feature a shared task on LLM evaluation and specifically encourages the submission of LLM evaluation focused papers. Other submissions that fit the general scope of Eval4NLP are of course also welcome. See below for more details. ** Shared Task ** This year’s version will come with a shared task on explainable evaluation of generated language (MT and summarization) with a focus on LLM prompts. Please find more information on the shared task page: https://eval4nlp.github.io/2023/shared-task.html. ** Topics ** Designing evaluation metrics: Proposing and/or analyzing metrics with desirable properties, e.g., high correlations with human judgments, strong in distinguishing high-quality outputs from mediocre and low-quality outputs, robust across lengths of input and output sequences, efficient to run, etc.; Reference-free evaluation metrics, which only require source text(s) and system predictions; Cross-domain metrics, which can reliably and robustly measure the quality of system outputs from heterogeneous modalities (e.g., image and speech), different genres (e.g., newspapers, Wikipedia articles and scientific papers) and different languages; Cost-effective methods for eliciting high-quality manual annotations; and Methods and metrics for evaluating interpretability and explanations of NLP models. Creating adequate evaluation data: Proposing new datasets or analyzing existing ones by studying their coverage and diversity, e.g., size of the corpus, covered phenomena, representativeness of samples, distribution of sample types, variability among data sources, eras, and genres; and Quality of annotations, e.g., consistency of annotations, inter-rater agreement, and bias check. Reporting correct results: Ensuring and reporting statistics for the trustworthiness of results, e.g., via appropriate significance tests, and reporting of score distributions rather than single-point estimates, to avoid chance findings; reproducibility of experiments, e.g., quantifying the reproducibility of papers and issuing reproducibility guidelines; and Comprehensive and unbiased error analyses and case studies, avoiding cherry-picking and sampling bias. ** Submission Guidelines ** The workshop welcomes two types of submission -- long and short papers. Long papers may consist of up to 8 pages of content, plus unlimited pages of references. Short papers may consist of up to 4 pages of content, plus unlimited pages of references. Please follow the ACL ARR formatting requirements, using the official templates: https://github.com/acl-org/acl-style-files. Final versions of both submission types will be given one additional page of content for addressing reviewers’ comments. The accepted papers will appear in the workshop proceedings. The review process is double-blind. Therefore, no author information should be included in the papers and the (optional) supplementary materials. Self-references that reveal the author's identity must be avoided. Papers that do not conform to these requirements will be rejected without review. ** The submission sites on Openreview ** Standard submissions: https://openreview.net/group?id=aclweb.org/AACL-IJCNLP/2023/Workshop/Eval4N… Pre-reviewed submissions: https://openreview.net/group?id=aclweb.org/AACL-IJCNLP/2023/Workshop/Eval4N… See below for more information on the two submission modes. ** Two submission modes: standard and pre-reviewed ** Eval4NLP features two modes of submissions. Standard submissions: We invite the submission of papers that will receive up to three double-blind reviews from the Eval4NLP committee, and a final verdict from the workshop chairs. Pre-reviewed: To a later deadline, we invite unpublished papers that have already been reviewed, either through ACL ARR, or recent AACL/EACL/ACL/EMNLP/COLING venues (these papers will not receive new reviews but will be judged together with their reviews via a meta-review; authors are invited to attach a note with comments on the reviews and describe possible revisions). Final verdicts will be either accept, reject, or conditional accept, i.e., the paper is only accepted provided that specific (meta-)reviewer requirements have been met. Please also note the multiple submission policy. ** Optional Supplementary Materials ** Authors are allowed to submit (optional) supplementary materials (e.g., appendices, software, and data) to improve the reproducibility of results and/or to provide additional information that does not fit in the paper. All of the supplementary materials must be zipped into one single file (.tgz or .zip) and submitted via Openreview together with the paper. However, because supplementary materials are completely optional, reviewers may or may not review or even download them. So, the submitted paper should be fully self-contained. ** Preprints ** Papers uploaded to preprint servers (e.g., ArXiv) can be submitted to the workshop. There is no deadline concerning when the papers were made publicly available. However, the version submitted to Eval4NLP must be anonymized, and we ask the authors not to update the preprints or advertise them on social media while they are under review at Eval4NLP. ** Multiple Submission Policy ** Eval4NLP allows authors to submit a paper that is under review in another venue (journal, conference, or workshop) or to be submitted elsewhere during the Eval4NLP review period. However, the authors need to withdraw the paper from all other venues if they get accepted and want to publish in Eval4NLP. Note that AACL and ARR do not allow double submissions. Hence, papers submitted both to the main conference and AACL workshops (including Eval4NLP) will violate the multiple submission policy of the main conference. If authors would like to submit a paper under review by AACL to the Eval4NLP workshop, they need to withdraw their paper from AACL and submit it to our workshop before the workshop submission deadline. ** Best Paper Awards ** We will optionally award prizes to the best paper submissions (subject to availability; more details to come soon). Both long and short submissions will be eligible for prizes. ** Presenting Published Papers ** If you want to present a paper which has been published recently elsewhere (such as other top-tier AI conferences) at our workshop, you may send the details of your paper (Paper title, authors, publication venue, abstract, and a link to download the paper) directly to eval4nlp(a)gmail.com. We will select a few high-quality and relevant papers to present at Eval4NLP. This allows such papers to gain more visibility from the workshop audience and increases the variety of the workshop program. Note that the chosen papers are considered as non-archival here and will not be included in the workshop proceedings. ----------------------------------------------------------------------------------------- Best wishes, Eval4NLP organizers Website: https://eval4nlp.github.io/2023/index.html Email: eval4nlp(a)gmail.com

1 0

2024

2023

2022

Corpora August 2023