Data-driven evaluation metrics for heterogeneous search engine result pages

Azzopardi, Leif and White, Ryen W. and Thomas, Paul and Craswell, Nick; (2020) Data-driven evaluation metrics for heterogeneous search engine result pages. In: CHIIR 2020 - Proceedings of the 2020 Conference on Human Information Interaction and Retrieval. CHIIR 2020 - Proceedings of the 2020 Conference on Human Information Interaction and Retrieval . ACM, New York, 213–222. ISBN 9781450368926 (https://doi.org/10.1145/3343413.3377959)

[thumbnail of Azzopardi-etal-CHIIR-2020-Data-driven-evaluation-metrics-for-heterogeneous-search-engine-result-pages]
Preview
 Text. Filename: Azzopardi_etal_CHIIR_2020_Data_driven_evaluation_metrics_for_heterogeneous_search_engine_result_pages.pdf
Accepted Author Manuscript
Download (758kB)| Preview

Abstract

Evaluation metrics for search typically assume items are homoge- neous. However, in the context of web search, this assumption does not hold. Modern search engine result pages (SERPs) are composed of a variety of item types (e.g., news, web, entity, etc.), and their influence on browsing behavior is largely unknown. In this paper, we perform a large-scale empirical analysis of pop- ular web search queries and investigate how different item types influence how people interact on SERPs. We then infer a user brows- ing model given people’s interactions with SERP items – creating a data-driven metric based on item type. We show that the proposed metric leads to more accurate estimates of: (1) total gain, (2) total time spent, and (3) stopping depth – without requiring extensive parameter tuning or a priori relevance information. These results suggest that item heterogeneity should be accounted for when de- veloping metrics for SERPs. While many open questions remain concerning the applicability and generalizability of data-driven metrics, they do serve as a formal mechanism to link observed user behaviors directly to how performance is measured. From this approach, we can draw new insights regarding the relationship be- tween behavior and performance – and design data-driven metrics based on real user behavior rather than using metrics reliant on some hypothesized model of user browsing behavior.

CORE (COnnecting REpositories)