Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints

Tools

Rohanian, Omid and Jauncey, Hannah and Nouriborji, Mohammadmahdi and Chauhan, Vinod Kumar and Gonçalves, Bronner P. and Kartsonaki, Christiana and Merson, Laura and Clifton, David ISARIC Clinical Characterisation Group (2023) Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints. In: BioNLP 2023 - BioNLP and BioNLP-ST, Proceedings of the Workshop. Proceedings of the Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics (ACL), CAN, pp. 62-78. ISBN 9781959429852 (https://doi.org/10.18653/v1/2023.bionlp-1.5)

[thumbnail of 2023.bionlp-Using-Bottleneck-Adapters-to-Identify-Cancer-in-Clinical-Notes-under-Low-Resource-Constraints]
Preview
 Text. Filename: 2023.bionlp-Using-Bottleneck-Adapters-to-Identify-Cancer-in-Clinical-Notes-under-Low-Resource-Constraints.pdf
Final Published Version
License: Creative Commons Attribution 4.0 logo
Download (421kB)| Preview

Abstract

Processing information locked within clinical health records is a challenging task that remains an active area of research in biomedical NLP. In this work, we evaluate a broad set of machine learning techniques ranging from simple RNNs to specialised transformers such as BioBERT on a dataset containing clinical notes along with a set of annotations indicating whether a sample is cancer-related or not. Furthermore, we specifically employ efficient fine-tuning methods from NLP, namely, bottleneck adapters and prompt tuning, to adapt the models to our specialised task. Our evaluations suggest that fine-tuning a frozen BERT model pre-trained on natural language and with bottleneck adapters outperforms all other strategies, including full fine-tuning of the specialised BioBERT model. Based on our findings, we suggest that using bottleneck adapters in low-resource situations with limited access to labelled data or processing capacity could be a viable strategy in biomedical text mining. The code used in the experiments are going to be made available at [LINK ANONYMIZED].

ORCID iDs

Rohanian, Omid, Jauncey, Hannah, Nouriborji, Mohammadmahdi, Chauhan, Vinod Kumar ORCID logoORCID: https://orcid.org/0000-0001-8195-548X, Gonçalves, Bronner P., Kartsonaki, Christiana, Merson, Laura and Clifton, David; Demner-fushman, Dina, Ananiadou, Sophia and Cohen, Kevin
CORE (COnnecting REpositories)