Fine-tuning language models to recognize semantic relations

Roussinov, Dmitri and Sharoff, Serge and Puchnina, Nadezhda (2023) Fine-tuning language models to recognize semantic relations. Language Resources and Evaluation, 57 (4). pp. 1463-1486. ISSN 1574-0218 (https://doi.org/10.1007/s10579-023-09677-w)

[thumbnail of Roussinov-etal-LRE-2023-Fine-tuning-language-models-to-recognize-semantic-relations]
Preview
 Text. Filename: Roussinov-etal-LRE-2023-Fine-tuning-language-models-to-recognize-semantic-relations.pdf
Final Published Version
License: Creative Commons Attribution 4.0 logo
Download (988kB)| Preview

Abstract

Transformer-based pre-trained Language Models (PLMs) have emerged as the foundations for the current state-of-the-art algorithms in most natural language processing tasks, in particular when applied to context rich data such as sentences or paragraphs. However, their impact on the tasks defined in terms of abstract individual word properties, not necessary tied to their specific use in a particular sentence, has been inadequately explored, which is a notable research gap. Addressing this gap is crucial for advancing our understanding of natural language processing. To fill this void, we concentrate on classification of semantic relations: given a pair of concepts (words or word sequences) the aim is to identify the semantic label to describe their relationship. E.g. in the case of the pair green/colour, “is a” is a suitable relation while “part of”, “property of”, and “opposite of” are not suitable. This classification is independent of a particular sentence in which these concepts might have been used. We are first to incorporate a language model into both existing approaches to this task, namely path-based and distribution-based methods. Our transformer-based approaches exhibit significant improvements over the state-of-the-art and come remarkably close to achieving human-level performance on rigorous benchmarks. We are also first to provide evidence that the standard datasets over-state the performance due to the effect of “lexical memorisation.” We reduce this effect by applying lexical separation. On the new benchmark datasets, the algorithmic performance remains significantly below human-level, highlighting that the task of semantic relation classification is still unresolved, particularly for language models of the sizes commonly used at the time of our study. We also identify additional challenges that PLM-based approaches face and conduct extensive ablation studies and other experiments to investigate the sensitivity of our findings to specific modelling and implementation choices. Furthermore, we examine the specific relations that pose greater challenges and discuss the trade-offs between accuracy and processing time.

ORCID iDs

Roussinov, Dmitri ORCID logoORCID: https://orcid.org/0000-0002-9313-2234, Sharoff, Serge and Puchnina, Nadezhda;
CORE (COnnecting REpositories)