Exploring the Impact of Conversational Strategies on User Search Experience in Goal-Oriented Tasks in a Voice-Only Domain

Dubiel, Mateusz and Halvey, Martin and Azzopardi, Leif (2021) Exploring the Impact of Conversational Strategies on User Search Experience in Goal-Oriented Tasks in a Voice-Only Domain. PhD thesis, University Of Strathclyde.

[thumbnail of Dubiel-2021-Exploring-the-impact-of-conversational-strategies-on-user-search-experience]
Preview
 Text. Filename: Dubiel_2021_Exploring_the_impact_of_conversational_strategies_on_user_search_experience.pdf
Final Published Version
Download (8MB)| Preview

Abstract

Conversational search has recently become established as a new information search paradigm. Increasing numbers of people are using smart speakers and virtual personal assistants to access information online. Given the substantial increase in the usage of devices that feature conversational agents, it is timely to explore the implications that they have on user search behaviour. This thesis explores how conversational search agents impact on user search experience and outcomes for goal-oriented tasks. Our enquiry is inspired by the premise that a fully conversational agent equipped with memory could lead to a better user search experience than an agent that is based on a slot-filling paradigm which rep-resents the current state of the art. This dissertation presents three Wizard of Oz, lab-based studies that explore a series of conversational agents that differ in their ability to preserve conversational state and employ a different degree of conversational initiative. The goal of this research is to get a better understanding of how different ways of presenting information impact on user search experience in terms of cognitive load, satisfaction with the agent, time required to complete the task and overall performance. In the first study, we address the problem of an agent's statefulness, which is the ability to preserve and account for the information presented by the user without the need to repeat it. Specifically, through a series of interactive search tasks, we evaluate how agent's memory improves user search experience. In the second study, we focus on mixed-initiative in conversation, by evaluating how different ways of eliciting information and presenting it back to the user (revealment) impact upon user search experience. In the third study, we further explore the role of mixed-initiative by evaluating agent's 'proactivity' (i.e. the search involvement that goes beyond the original scope of the query) on user search experience. Finally, we compare the insights from all of the three studies and assess how differences in search task design and the mode of delivering the studies (lab-based vs. deployed online) impact upon experimental outcomes. Overall, the contribution of the research presented in the current thesis is threefold. Firstly, it provides a validation of theoretical frameworks of conversational search by empirically investigating the impact of agent's memory and mixed initiative on user search experience. Secondly, it provides insights into the efficacy of alternative ways of eliciting information and presenting search results during conversational search. Thirdly, it contributes towards a more robust evaluation of conversational agents by developing a spectrum of conversational strategies and reflecting on the impact of the design of goal-oriented search tasks.

ORCID iDs

Dubiel, Mateusz ORCID logoORCID: https://orcid.org/0000-0001-8250-3370, Halvey, Martin and Azzopardi, Leif;
CORE (COnnecting REpositories)