Evaluation of attribution bias in generator-aware retrieval-augmented large language models

Tools

Abolghasemi, Amin and Azzopardi, Leif and Hashemi, Seyyed Hadi and de Rijke, Maarten and Verberne, Suzan; Che, Wanxiang and Nabende, Joyce and Shutova, Ekaterina and Pilehvar, Mohammad Taher, eds. (2025) Evaluation of attribution bias in generator-aware retrieval-augmented large language models. In: Findings of the Association for Computational Linguistics. Proceedings of the Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics (ACL), AUT, pp. 21105-21124. ISBN 9798891762565 (https://doi.org/10.18653/v1/2025.findings-acl.1087)

[thumbnail of Abolghasemi-etal-2025-Evaluation-of-attribution-bias-in-generator-aware-retrieval-augmented-large-language-models]
Preview
 Text. Filename: Abolghasemi-etal-2025-Evaluation-of-attribution-bias-in-generator-aware-retrieval-augmented-large-language-models.pdf
Final Published Version
License: Creative Commons Attribution 4.0 logo
Download (467kB)| Preview

Abstract

Attributing answers to source documents is an approach used to enhance the verifiability of a model's output in retrieval-augmented generation (RAG). Prior work has mainly focused on improving and evaluating the attribution quality of large language models (LLMs) in RAG, but this may come at the expense of inducing biases in the attribution of answers. We define and examine two aspects in the evaluation of LLMs in RAG pipelines, namely attribution sensitivity and bias with respect to authorship information. We explicitly inform an LLM about the authors of source documents, instruct it to attribute its answers, and analyze (i) how sensitive the LLM's output is to the author of source documents, and (ii) whether the LLM exhibits a bias towards human-written or AI-generated source documents. We design an experimental setup in which we use counterfactual evaluation to study three LLMs in terms of their attribution sensitivity and bias in RAG pipelines. Our results show that adding authorship information to source documents can significantly change the attribution quality of LLMs by 3 to 18%. We show that LLMs can have an attribution bias towards explicit human authorship, which can serve as a competing hypothesis for findings of prior work that shows that LLM-generated content may be preferred over human-written contents. Our findings indicate that metadata of source documents can influence LLMs' trust, and how they attribute their answers. Furthermore, our research highlights attribution bias and sensitivity as a novel aspect of the brittleness of LLMs.

ORCID iDs

Abolghasemi, Amin, Azzopardi, Leif ORCID logoORCID: https://orcid.org/0000-0002-6900-0557, Hashemi, Seyyed Hadi, de Rijke, Maarten and Verberne, Suzan; Che, Wanxiang, Nabende, Joyce, Shutova, Ekaterina and Pilehvar, Mohammad Taher
CORE (COnnecting REpositories)