ERR is not C/W/L : Exploring the relationship between expected reciprocal rank and other metrics

Azzopardi, Leif and Mackenzie, Joel and Moffat, Alistair; (2021) ERR is not C/W/L : Exploring the relationship between expected reciprocal rank and other metrics. In: ICTIR '21 : Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval. ACM, New York, NY., 231–237. ISBN 9781450386111 (https://doi.org/10.1145/3471158.3472239)

[thumbnail of Azzopardi-etal-SIGIR-2021-Exploring-the-relationship-between-expected-reciprocal-rank-and-other-metrics]
Preview
 Text. Filename: Azzopardi_etal_SIGIR_2021_Exploring_the_relationship_between_expected_reciprocal_rank_and_other_metrics.pdf
Accepted Author Manuscript
Download (1MB)| Preview

Abstract

We explore the relationship between expected reciprocal rank (ERR) and the metrics that are available under the C/W/L framework. On the surface, it appears that the user browsing model associated with ERR can be directly injected into a C/W/L arrangement, to produce system measurements equivalent to those generated from ERR. That assumption is now known to be invalid, and demonstration of the impossibility of ERR being described via C/W/L choices forms the first part of our work. Given that ERR cannot be accommodated within the C/W/L framework, we then explore the extent to which practical use of ERR correlates with metrics that do fit within the C/W/L user browsing model. In this part of the investigation we present a range of shallow-evaluation C/W/L variants that have very high correlation with ERR when compared in experiments involving a large number of TREC runs. That is, while ERR itself is not a C/W/L metric, there are other weighted-precision compu- tations that fit with the user model assumed by C/W/L, and yield system comparisons almost indistinguishable from those generated via the use of ERR.

CORE (COnnecting REpositories)