Reflecting upon perceptual speed tests in information retrieval : limitations, challenges, and recommendations

Foulds, Olivia and Azzopardi, Leif and Halvey, Martin; (2020) Reflecting upon perceptual speed tests in information retrieval : limitations, challenges, and recommendations. In: CHIIR '20: Proceedings of the 2020 Conference on Human Information Interaction and Retrieval. ACM, CAN, pp. 234-242. ISBN 978-1-4503-6892-6

[img]
Preview
Text (Foulds-etal-chiirpr-2020-Reflecting-upon-perceptual-speed-tests-in-information-retrieval)
Foulds_etal_chiirpr_2020_Reflecting_upon_perceptual_speed_tests_in_information_retrieval.pdf
Accepted Author Manuscript

Download (475kB)| Preview

    Abstract

    Perceptual Speed (PS) is a cognitive ability defined by an individual’s accuracy and speed to scan information while completing visual search tasks. Prior studies using PS tests have demonstrated that PS affects multiple factors in Information Retrieval (IR), such as a user’s search performance, interaction with the system, time spent completing tasks, and subjective impression of their workload. With greater knowledge of PS, systems could be designed that accommodate users with low PS to improve their overall search experience. However, in this perspectives paper, we analyse how PS tests have been used in IR, and identify multiple uncertainties regarding PS content, administration, analysis, and reporting of findings. Consequently, we aim to stir discussion between IR researchers by drawing awareness to these issues. As a result, we further discuss challenges involved in advancing how future PS tests are used in IR. Finally, we propose recommendations that have the potential for enhancing the reliability and validity of current PS tests.