User models, metrics and measures of search : a tutorial on the C/W/L evaluation framework

Azzopardi, Leif and Moffat, Alistair and Thomas, Paul and Zuccon, Guido; (2021) User models, metrics and measures of search : a tutorial on the C/W/L evaluation framework. In: CHIIR 2021 - Proceedings of the 2021 Conference on Human Information Interaction and Retrieval. CHIIR 2021 - Proceedings of the 2021 Conference on Human Information Interaction and Retrieval . ACM, New York, NY., 347–348. ISBN 9781450380553 (https://doi.org/10.1145/3406522.3446049)

[thumbnail of Azzopardi-etal-CHIIR-2021-User-models-metrics-and-measures-of-search]
Preview
 Text. Filename: Azzopardi_etal_CHIIR_2021_User_models_metrics_and_measures_of_search.pdf
Accepted Author Manuscript
Download (363kB)| Preview

Abstract

Evaluation is central to Information Retrieval, and is how we compare the quality of systems. One important principle of evaluation is that the measured score should reflect the user's experience with the system. Hence, there should be direct connection between how users interact with the system and the characteristics of the metric. In this tutorial we introduce the C/W/L approach to user modeling and show how different user models lead to different metrics. We then describe the recent innovations and approaches to evaluation that it has facilitated. The tutorial is presented as a mix of on-line synchronous lecture, pre-recorded in-depth videos, and hands-on activities using theC/W/L toolkit for participants' own evaluation tasks. A followup consultation session is also provided, to allow extended questions and individual discussion with the four presenters.

CORE (COnnecting REpositories)