Space transformers : language modeling for space systems

Berquand, Audrey and Darm, Paul and Riccardi, Annalisa (2021) Space transformers : language modeling for space systems. IEEE Access, 9. pp. 133111-133122. ISSN 2169-3536 (https://doi.org/10.1109/ACCESS.2021.3115659)

[thumbnail of Berquand-etal-IEEEA-2021-Space-Transformers-language-modeling]
Preview
 Text. Filename: Berquand_etal_IEEEA_2021_Space_Transformers_language_modeling.pdf
Final Published Version
License: Creative Commons Attribution 4.0 logo
Download (2MB)| Preview

Abstract

The transformers architecture and transfer learning have radically modified the Natural Language Processing (NLP) landscape, enabling new applications in fields where open source labelled datasets are scarce. Space systems engineering is a field with limited access to large labelled corpora and a need for enhanced knowledge reuse of accumulated design data. Transformers models such as the Bidirectional Encoder Representations from Transformers (BERT) and the Robustly Optimised BERT Pretraining Approach (RoBERTa) are however trained on general corpora. To answer the need for domain specific contextualised word embedding in the space field, we propose Space Transformers, a novel family of three models, SpaceBERT, SpaceRoBERTa and SpaceSciBERT, respectively further pre-trained from BERT, RoBERTa and SciBERT on our domain-specific corpus. We collect and label a new dataset of space systems concepts based on space standards. We fine-tune and compare our domain-specific models to their general counterparts on a domain-specific Concept Recognition (CR) task. Our study rightly demonstrates that the models further pre-trained on a space corpus outperform their respective baseline models in the Concept Recognition task, with SpaceRoBERTa achieving significant higher ranking overall.

ORCID iDs

Berquand, Audrey, Darm, Paul and Riccardi, Annalisa ORCID logoORCID: https://orcid.org/0000-0001-5305-9450;
CORE (COnnecting REpositories)