Agents, simulated users and humans : an analysis of performance and behaviour

Maxwell, David and Azzopardi, Leif; (2016) Agents, simulated users and humans : an analysis of performance and behaviour. In: CIKM '16 : Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. ACM, New York, NY., pp. 731-740. ISBN 9781450340731 (https://doi.org/10.1145/2983323.2983805)

[thumbnail of Maxwell-Azzopardi-ACM-ICIKM-2016-Agents-simulated-users-and-humans]
Preview
 Text. Filename: Maxwell_Azzopardi_ACM_ICIKM_2016_Agents_simulated_users_and_humans.pdf
Accepted Author Manuscript
Download (774kB)| Preview

Abstract

Most of the current models that are used to simulate users in Interactive Information Retrieval (IIR) lack realism and agency. Such models generally make decisions in a stochastic manner, without recourse to the actual information encountered or the underlying information need. In this paper, we develop a more sophisticated model of the user that includes their cognitive state within the simulation. The cognitive state maintains data about what the simulated user knows, has done and has seen, along with representations of what it considers attractive and relevant. Decisions to inspect or judge are then made based upon the simulated user's current state, rather than stochastically. In the context of ad-hoc topic retrieval, we evaluate the quality of the simulated users and agents by comparing their behaviour and performance against 48 human subjects under the same conditions, topics, time constraints, costs and search engine. Our findings show that while naive configurations of simulated users and agents substantially outperform our human subjects, their search behaviour is notably different from actual searchers. However, more sophisticated search agents can be tuned to act more like actual searchers providing greater realism. This innovation advances the state of the art in simulation, from simulated users towards autonomous agents. It provides a much needed step forward enabling the creation of more realistic simulations, while also motivating the development of more advanced cognitive agents and tools to help support and augment human searchers. Future work will focus not only on the pragmatics of tuning and training such agents for topic retrieval, but will also look at developing agents for other tasks and contexts such as collaborative search and slow search.

  • Item type:Book Section
    ID code:58955
    Dates:
    Date
    Event
    24 October 2016
    Published
    18 July 2016
    Accepted
    Notes:© 2016 Copyright held by the owner/author(s). Publication rights licensed to ACM. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Proceedings of the 25th ACM International on Conference on Information and Knowledge Management https://doi.org/10.1145/2983323.2983805.
    Subjects:Bibliography. Library Science. Information Resources > Library Science. Information Science
    Department:Faculty of Science > Computer and Information Sciences
    Depositing user: Pure Administrator
    Date deposited:05 Dec 2016 11:22
    Last modified:11 Nov 2024 15:07
    URI:https://strathprints.strath.ac.uk/id/eprint/58955
CORE (COnnecting REpositories)