Dear colleagues,
We are happy to announce the call for papers for our one-day special session on “Multi-Perspectivist Data and Learning 2023” at the upcoming CD MAKE 2023 conference:
https://cd-make.net/special-sessions/multi-perspectivist-data-and-learning/
*************************************************** Description, scope and aims
Many Artificial Intelligence applications are based on supervised machine learning (ML), which ultimately grounds on manually annotated data. The annotation process (i.e., ground-truthing) is often performed in terms of a majority vote and this has been proved to be often problematic, as highlighted by recent studies on the evaluation of ML models. Recently, a different paradigm for ground-truthing has started to emerge, called data perspectivism, which moves away from traditional majority aggregated datasets, towards the adoption of methods that integrate different opinions and perspectives within the knowledge representation, training, and evaluation steps of ML processes, by adopting a non-aggregation policy. This alternative paradigm obviously implies a radical change in how we develop and evaluate ML systems: such ML systems have to take into account multiple, uncertain, and potentially mutually conflicting views. This obviously brings both opportunities and difficulties: novel models or training techniques may need to be designed, and the validation phase may become more complex. Nonetheless, initial works have shown that data perspectivism can lead to better performances, and could also have important implications in terms of human-in-the-loop and interpretable AI, as well as in regard to the ethical issues or concerns related to the use of AI systems.
The scope of this special session is to attract contributions related to the management of subjective, crowd-sourced, multi-perspective, or otherwise non-aggregated data in ground-truthing, machine learning, and more generally artificial intelligence systems.
Invited contributions: full research papers and research in progress papers.
***************************************************
Topics of interest:
- Subjective, uncertain, or conflicting information in annotation and crowdsourcing processes;
- Limits and problems with standard data annotation and aggregation processes;
- Theoretical studies on the problem of learning from multi-rater and non-aggregated data;
- Participation mechanisms/incentives/gamification for rater engagement and crowdsourcing;
- Ethical and legal concerns related to annotation and aggregation processes in ground-truthing;
- Creation and documentation of multi-rater and non-aggregated datasets and benchmarks;
- Development of ML algorithms for multi-rater and non-aggregated data;
- Techniques for the evaluation of ML systems based on multi-rater and non-aggregated data;
- Applications of data perspectivism and non-aggregated data to eXplainable AI, human-in-the-loop AI and algorithmic fairness;
- Experimental and application studies of ML/AI systems on multi-rater and non-aggregated data, in possibly different application domains (e.g. NLP, medicine, legal studies, etc.)
***************************************************
Important dates:
Submission Deadline March 27, 2023 (AoE)
Author Notification June 01, 2023
Proceedings Version June 22, 2023 (AoE)
Conference August 29 – September 01, 2023
***************************************************
Special Session Chairs:
Federico Cabitza (University of Milano-Bicocca, Italy)
Andrea Campagner (University of Milano-Bicocca, Italy)
Valerio Basile (University of Turin, Italy)
Program Committee (provisional):
Nahuel Costa Cortez, University of Oviedo
Elisa Leonardelli, Fondazione Bruno Kessler (FBK)
Julian Lienen, Paderborn University
Gavin Abercrombie, Heriot-Watt University
Simona Frenda, University of Turin
Marília Barandas, Fraunhofer Portugal AICOS
Duarte Folgado, Fraunhofer Portugal AICOS
Barbara Plank, Ludwig Maximilian University of Munich
Tommaso Caselli, Rijksuniversiteit Groningen
***************************************************
Related readings
[1] Cabitza, F., Campagner, A., Basile, V. (2023)
Toward a Perspectivist Turn in Ground Truthing for Predictive Computing
Proceedings of the AAAI Conference on Artificial Intelligence
(extended preprint at: https://arxiv.org/pdf/2109.04270.pdf)
[2] V. Basile (2020)
It’s the End of the Gold Standard as we Know it. On the Impact of Pre-aggregation on the Evaluation of Highly Subjective Tasks
Proceedings of the AIxIA 2020 Discussion Papers Workshop
[3] F. Cabitza, A. Campagner, L. M. Sconfienza (2020)
As if sand were stone. New concepts and metrics to probe the ground on which to build trustable AI
BMC Medical Informatics and Decision Making
[4] Plank, B. (2022).
The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation.
arXiv preprint arXiv:2211.02570.
Dear colleagues,
This is a reminder that the call for papers is open for our one-day special session on “Multi-Perspectivist Data and Learning 2023” at the upcoming CD MAKE 2023 conference:
https://cd-make.net/special-sessions/multi-perspectivist-data-and-learning/
*************************************************** Description, scope and aims
Many Artificial Intelligence applications are based on supervised machine learning (ML), which ultimately grounds on manually annotated data. The annotation process (i.e., ground-truthing) is often performed in terms of a majority vote and this has been proved to be often problematic, as highlighted by recent studies on the evaluation of ML models. Recently, a different paradigm for ground-truthing has started to emerge, called data perspectivism, which moves away from traditional majority aggregated datasets, towards the adoption of methods that integrate different opinions and perspectives within the knowledge representation, training, and evaluation steps of ML processes, by adopting a non-aggregation policy. This alternative paradigm obviously implies a radical change in how we develop and evaluate ML systems: such ML systems have to take into account multiple, uncertain, and potentially mutually conflicting views. This obviously brings both opportunities and difficulties: novel models or training techniques may need to be designed, and the validation phase may become more complex. Nonetheless, initial works have shown that data perspectivism can lead to better performances, and could also have important implications in terms of human-in-the-loop and interpretable AI, as well as in regard to the ethical issues or concerns related to the use of AI systems.
The scope of this special session is to attract contributions related to the management of subjective, crowd-sourced, multi-perspective, or otherwise non-aggregated data in ground-truthing, machine learning, and more generally artificial intelligence systems.
Invited contributions: full research papers and research in progress papers.
***************************************************
Topics of interest:
- Subjective, uncertain, or conflicting information in annotation and crowdsourcing processes;
- Limits and problems with standard data annotation and aggregation processes;
- Theoretical studies on the problem of learning from multi-rater and non-aggregated data;
- Participation mechanisms/incentives/gamification for rater engagement and crowdsourcing;
- Ethical and legal concerns related to annotation and aggregation processes in ground-truthing;
- Creation and documentation of multi-rater and non-aggregated datasets and benchmarks;
- Development of ML algorithms for multi-rater and non-aggregated data;
- Techniques for the evaluation of ML systems based on multi-rater and non-aggregated data;
- Applications of data perspectivism and non-aggregated data to eXplainable AI, human-in-the-loop AI and algorithmic fairness;
- Experimental and application studies of ML/AI systems on multi-rater and non-aggregated data, in possibly different application domains (e.g. NLP, medicine, legal studies, etc.)
***************************************************
Important dates:
Submission Deadline March 27, 2023 (AoE)
Author Notification June 01, 2023
Proceedings Version June 22, 2023 (AoE)
Conference August 29 – September 01, 2023
***************************************************
Special Session Chairs:
Federico Cabitza (University of Milano-Bicocca, Italy)
Andrea Campagner (University of Milano-Bicocca, Italy)
Valerio Basile (University of Turin, Italy)
Program Committee (provisional):
Nahuel Costa Cortez, University of Oviedo
Elisa Leonardelli, Fondazione Bruno Kessler (FBK)
Julian Lienen, Paderborn University
Gavin Abercrombie, Heriot-Watt University
Simona Frenda, University of Turin
Marília Barandas, Fraunhofer Portugal AICOS
Duarte Folgado, Fraunhofer Portugal AICOS
Barbara Plank, Ludwig Maximilian University of Munich
Tommaso Caselli, Rijksuniversiteit Groningen
***************************************************
Related readings
[1] Cabitza, F., Campagner, A., Basile, V. (2023)
Toward a Perspectivist Turn in Ground Truthing for Predictive Computing
Proceedings of the AAAI Conference on Artificial Intelligence
(extended preprint at: https://arxiv.org/pdf/2109.04270.pdf)
[2] V. Basile (2020)
It’s the End of the Gold Standard as we Know it. On the Impact of Pre-aggregation on the Evaluation of Highly Subjective Tasks
Proceedings of the AIxIA 2020 Discussion Papers Workshop
[3] F. Cabitza, A. Campagner, L. M. Sconfienza (2020)
As if sand were stone. New concepts and metrics to probe the ground on which to build trustable AI
BMC Medical Informatics and Decision Making
[4] Plank, B. (2022).
The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation.
arXiv preprint arXiv:2211.02570.
-
Valerio Basile