Picture map of Europe with pins indicating European capital cities

Open Access research with a European policy impact...

The Strathprints institutional repository is a digital archive of University of Strathclyde's Open Access research outputs. Strathprints provides access to thousands of Open Access research papers by Strathclyde researchers, including by researchers from the European Policies Research Centre (EPRC).

EPRC is a leading institute in Europe for comparative research on public policy, with a particular focus on regional development policies. Spanning 30 European countries, EPRC research programmes have a strong emphasis on applied research and knowledge exchange, including the provision of policy advice to EU institutions and national and sub-national government authorities throughout Europe.

Explore research outputs by the European Policies Research Centre...

A comparison of primary and secondary relevance judgements for real-life topics

Wakeling, Simon and Halvey, Martin and Villa, Robert and Hasler, Laura (2016) A comparison of primary and secondary relevance judgements for real-life topics. In: Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (CHIIR '16 ). ACM, New York, pp. 173-182. ISBN 9781450337519

[img]
Preview
Text (Wakeling-etal-CHIIR-2016-comparison-of-primary-and-secondary-relevance-judgements)
Wakeling_etal_CHIIR_2016_comparison_of_primary_and_secondary_relevance_judgements.pdf - Accepted Author Manuscript

Download (817kB) | Preview

Abstract

The notion of relevance is fundamental to the field of Information Retrieval. Within the field a generally accepted conception of relevance as inherently subjective has emerged, with an individual’s assessment of relevance influenced by numerous contextual factors. In this paper we present a user study that examines in detail the differences between primary and secondary assessors on a set of “real-world” topics which were gathered specifically for the work. By gathering topics which are representative of the staff and students at a major university, at a particular point in time, we aim to explore differences between primary and secondary relevance judgements for real-life search tasks. Findings suggest that while secondary assessors may find the assessment task challenging in various ways (they generally possess less interest and knowledge in secondary topics and take longer to assess documents), agreement between primary and secondary assessors is high.