The influence of presentation and performance on user satisfaction

Pathak, Kanaad and Azzopardi, Leif and Halvey, Martin; (2024) The influence of presentation and performance on user satisfaction. In: CHIIR '24: Proceedings of the 2024 Conference on Human Information Interaction and Retrieval. Association for Computing Machinery, New York, pp. 77-86. ISBN 9798400704345 (https://doi.org/10.1145/3627508.3638335)

[thumbnail of Pathak-etal-CHIR-2024-The-influence-of-presentation-and-performance-on-user-satisfaction]
Preview
 Text. Filename: Pathak-etal-CHIR-2024-The-influence-of-presentation-and-performance-on-user-satisfaction.pdf
Final Published Version
License: Strathprints license 1.0
Download (6MB)| Preview

Abstract

Information Retrieval (IR) systems are designed to provide users with a ranked list of results based on their queries. The effectiveness of an IR system is gauged not just by its ability to retrieve relevant results but also by how it presents these results to users; an engaging presentation often correlates with increased user satisfaction. While existing research has delved into the link between user satisfaction, IR performance metrics, and presentation, these aspects have typically been investigated in isolation. Our research aims to bridge this gap by examining the relationship between query performance, presentation and user satisfaction. For our analysis, we conducted a between-subjects experiment comparing the effectiveness of various result card layouts for an ad-hoc news search interface. Drawing data from the TREC WaPo 2018 collection, we centered our study on four specific topics. Within each of these topics, we assessed six distinct queries with varying nDCG values. Our study involved 164 participants who were exposed to one of five distinct layouts containing result cards, such as “title”, “title+image”, or “title+image+summary”. Our findings indicate that while nDCG is a strong predictor of user satisfaction at the query level, there exists no linear relationship between the performance of the query, presentation of results and user satisfaction. However, when considering the total gain on the initial result page, we observed that presentation does play a significant role in user satisfaction (at the query level) for certain layouts with result cards such as, title+image or title+image+summary. Our results also suggest that the layout differences have complex and multifaceted impacts on satisfaction. We demonstrate the capacity to equalize user satisfaction levels between queries of varying performance by changing how results are presented. This emphasizes the necessity to harmonize both performance and presentation in IR systems, considering users’ diverse preferences. Ultimately, our insights can steer the evolution of more user-aligned IR systems, underscoring the balance between system performance and result presentation.

ORCID iDs

Pathak, Kanaad ORCID logoORCID: https://orcid.org/0000-0002-1246-8685, Azzopardi, Leif and Halvey, Martin ORCID logoORCID: https://orcid.org/0000-0001-6387-8679;
CORE (COnnecting REpositories)