*Venue*: ACL 2024, TextGraph workshop *Website*: https://sites.google.com/view/textgraphs2024/home/shared-task
TextGraph-17 workshop co-located with ACL-2024 features a shared task. TextGraph fosters the investigation of synergies between methods for text and graph processing. This edition focuses on the fusion of LLMs with KGs. In line with this goal, we propose a shared task on Text-Graph Representations for Knowledge Graph Question Answering (KGQA).
The shared task is to select a KG entity (out of several candidates) that corresponds to an answer given a textual question. The specificity of the task is that for each question-answer (Q-A) pair not only a textual Q-A pair is given but also a graph of shortest paths in the KG from entities in the query to the LLM-generated candidate entity (including links of the intermediate nodes). This way, participants easily may experiment with various strategies of text-graph modality fusion for the given task in a controllable manner.
Participants can submit reports about their participation to the TextGraphs workshop. An example of a previous TextGraphs shared task can be found here: https://aclanthology.org/volumes/2022.textgraphs-1/.
*Task Description*
Participants are given: - text: question with a list of Wikidata entities mentioned, - text: 5-10 answer candidates in the form of Wikidata entities, - graph: a Wikidata sub-graph composed of the shortest paths between entities in question and entities in answer is provided.
One of the candidates is correct, others are incorrect. The goal is to find the correct answer ie. perform a binary classification. Participants are provided a train and development dataset in the form of Q-A-Subgraph triples. Besides, a submission to Codalab public and private leaderboard of the test set Q-A solutions will be available.
Examples of visualization for the question "Who was formerly an actor and now a Republican senator?" can be found on our challenge website (https://sites.google.com/view/textgraphs2024/home/shared-task). The participants are provided both textual labels of candidates and such graphs so additional features can be extracted from them, such as graph density, length of paths, textual labels on the paths, etc.
* Important Dates*
- Training dataset released: 10th March 2024 - Test set released: 25 March 2024 - End of evaluation: 25 April 2024 - Submission deadline for technical reports: 17 May 2024
Contact
Please write all questions about the shared task to textgraphs17@googlegroups.com. Also, you are invited to join our Telegram group where you can connect to organizers and get updates: https://t.me/+kRTCZYTrpJ5jZGVi
Organizers: Irina Nikishina, Universität Hamburg Aida Usmanova, Leuphana University Lüneburg Angelie Kraft, Universität Hamburg Cedric Möller, Universität Hamburg Debayan Banerjee, Universität Hamburg Junbo Huang, Universität Hamburg Longquan Jiang, Universität Hamburg Rana Abdullah, Universität Hamburg Xi Yan, Universität Hamburg Andrey Sakhovskiy, KFU Elena Tutubalina, KFU Mikhail Salnikov, AIRI Alexander Panchenko, AIRI Ricardo Usbeck, Universität Hamburg
Xi Yan and Cedric Möller, On behalf of the Organizing Committee