Strathprints logo
Strathprints Home | Open Access | Browse | Search | User area | Copyright | Help | Library Home | SUPrimo

Evaluating epistemic uncertainty under incomplete assessments

Baillie, Mark and Azzopardi, Leif and Ruthven, Ian (2008) Evaluating epistemic uncertainty under incomplete assessments. Information Processing and Management, 44 (2). pp. 811-837. ISSN 0306-4573

[img]
Preview
PDF (strathprints003418.pdf)
strathprints003418.pdf

Download (1MB) | Preview

Abstract

The thesis of this study is to propose an extended methodology for laboratory based Information Retrieval evaluation under incomplete relevance assessments. This new methodology aims to identify potential uncertainty during system comparison that may result from incompleteness. The adoption of this methodology is advantageous, because the detection of epistemic uncertainty - the amount of knowledge (or ignorance) we have about the estimate of a system's performance - during the evaluation process can guide and direct researchers when evaluating new systems over existing and future test collections. Across a series of experiments we demonstrate how this methodology can lead towards a finer grained analysis of systems. In particular, we show through experimentation how the current practice in Information Retrieval evaluation of using a measurement depth larger than the pooling depth increases uncertainty during system comparison.

Item type: Article
ID code: 3418
Keywords: information retrieval evaluation, incompleteness, system comparison, test collections, Electronic information resources, Media Technology, Management Science and Operations Research, Library and Information Sciences, Information Systems, Computer Science Applications
Subjects: Bibliography. Library Science. Information Resources > Information resources > Electronic information resources
Department: Faculty of Science > Computer and Information Sciences
Depositing user: Strathprints Administrator
Date Deposited: 15 Jun 2007
Last modified: 15 Apr 2015 11:27
Related URLs:
URI: http://strathprints.strath.ac.uk/id/eprint/3418

Actions (login required)

View Item View Item