The Third Workshop on LLM4Eval progresses the discussion from the previous series. These earlier events investigated the potential and challenges of using LLMs for search relevance evaluation, automated judgments, and retrieval-augmented generation (RAG) assessment. As modern IR systems integrate search, recommendations, conversational interfaces, and personalization, new evaluation challenges arise beyond basic relevance assessment. These applications create personalized rankings, explanations, and adapt to user preferences over time, requiring new evaluation methods.

Overview

Recent advancements in Large Language Models (LLMs) have significantly impacted evaluation methodologies in Information Retrieval (IR), reshaping the way relevance, quality, and user satisfaction are assessed. Initially demonstrating the potential for query-document relevance judgments, LLMs are now being applied to more complex tasks, including relevance label generation, assessment of retrieval-augmented generation systems, and evaluation of the quality of text-generation systems. As IR systems evolve toward more sophisticated and personalized user experiences, integrating search, recommendations, and conversational interfaces, new evaluation methodologies become necessary.

Building upon the success of our previous workshops, this third iteration of the LLM4Eval workshop at SIGIR 2025 seeks submissions exploring new opportunities, limitations, and hybrid approaches involving LLM-based evaluations.

Important Dates

• Paper submission deadline: April 23, 2025 (AoE)
• Notification of acceptance: May 21, 2025 (AoE)
• Workshop date: July 17, 2025

Topics of interest

We invite submissions on topics including, but not limited to:

• LLMs for query-document relevance assessment
• Evaluating conversational IR and recommendation systems with LLMs
• Hybrid evaluation frameworks combining LLM and human annotations
• Identifying failure modes and limitations of LLM annotations
• Prompt engineering strategies for improving LLM annotation quality
• Standardizing protocols for reliable LLM-based evaluations
• Bias, fairness, and ethical considerations in LLM evaluations
• LLM annotation robustness, reliability, and reproducibility
• User-centric evaluations, personalization, and subjective assessments with LLMs
• Case studies and lessons from industry applications of LLM-based evaluations

Submission guidelines

• Papers must follow SIGIR format and should not exceed 9 pages, excluding references.
• We accept full papers (published or unpublished), position papers, and demo papers.
• All papers will be peer-reviewed (double-blind) by the program committee and judged by their relevance to the workshop themes and potential to generate discussion.
• Previously published studies can be submitted in their original format and will be reviewed solely for their relevance to this workshop.
• All submissions must be in English (PDF format).
• Submission through EasyChair: https://easychair.org/conferences/?conf=llm4evalsigir25.

Publication options

Authors can choose between archival and non-archival options for their submissions:

• Archival: Papers will be included in the workshop proceedings.
• Non-archival: Papers may be uploaded to arXiv.org, allowing submission elsewhere as they will be considered non-archival. The workshop’s website will maintain a link to the arXiv versions of the papers.