CLEF 2017 dynamic search evaluation lab overview

Kanoulas, Evangelos and Azzopardi, Leif; Jones, Gareth J.F. and Lawless, Séamus and Gonzalo, Julio and Kelly, Liadh and Goeuriot, Lorraine and Mandl, Thomas and Cappellato, Linda and Ferro, Nicola, eds. (2017) CLEF 2017 dynamic search evaluation lab overview. In: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 10456 . Springer-Verlag, IRL, pp. 361-366. ISBN 9783319658131 (https://doi.org/10.1007/978-3-319-65813-1_31)

[thumbnail of Kanoulas-Azzopardi-LNCS-2017-CLEF-2017-dynamic-search-evaluation-lab]
Preview
 Text. Filename: Kanoulas_Azzopardi_LNCS_2017_CLEF_2017_dynamic_search_evaluation_lab.pdf
Accepted Author Manuscript
Download (190kB)| Preview

Abstract

In this paper we provide an overview of the first edition of the CLEF Dynamic Search Lab. The CLEF Dynamic Search lab ran in the form of a workshop with the goal of approaching one key question: how can we evaluate dynamic search algorithms? Unlike static search algorithms, which essentially consider user request’s independently, and which do not adapt the ranking w.r.t the user’s sequence of interactions, dynamic search algorithms try to infer from the user’s intentions from their interactions and then adapt the ranking accordingly. Personalized session search, contextual search, and dialog systems often adopt such algorithms. This lab provides an opportunity for researchers to discuss the challenges faced when trying to measure and evaluate the performance of dynamic search algorithms, given the context of available corpora, simulations methods, and current evaluation metrics. To seed the discussion, a pilot task was run with the goal of producing search agents that could simulate the process of a user, interacting with a search system over the course of a search session. Herein, we describe the overall objectives of the CLEF 2017 Dynamic Search Lab, the resources created for the pilot task and the evaluation methodology adopted.

CORE (COnnecting REpositories)