Subtle variation in sepsis-III definitions markedly influences predictive performance within and across methods

Cohen, Samuel N. and Foster, James and Foster, Peter and Lou, Hang and Lyons, Terry and Morley, Sam and Morrill, James and Ni, Hao and Palmer, Edward and Wang, Bo and Wu, Yue and Yang, Lingyi and Yang, Weixin (2024) Subtle variation in sepsis-III definitions markedly influences predictive performance within and across methods. Scientific Reports, 14 (1). 1920. ISSN 2045-2322 (https://doi.org/10.1038/s41598-024-51989-6)

[thumbnail of Cohen-etal-SR-2024-Subtle-variation-in-sepsis-III-definitions-markedly-influences-predictive-performance]
Preview
 Text. Filename: Cohen-etal-SR-2024-Subtle-variation-in-sepsis-III-definitions-markedly-influences-predictive-performance.pdf
Final Published Version
License: Creative Commons Attribution 4.0 logo
Download (2MB)| Preview

Abstract

Early detection of sepsis is key to ensure timely clinical intervention. Since very few end-to-end pipelines are publicly available, fair comparisons between methodologies are difficult if not impossible. Progress is further limited by discrepancies in the reconstruction of sepsis onset time. This retrospective cohort study highlights the variation in performance of predictive models under three subtly different interpretations of sepsis onset from the sepsis-III definition and compares this against inter-model differences. The models are chosen to cover tree-based, deep learning, and survival analysis methods. Using the MIMIC-III database, between 867 and 2178 intensive care unit admissions with sepsis were identified, depending on the onset definition. We show that model performance can be more sensitive to differences in the definition of sepsis onset than to the model itself. Given a fixed sepsis definition, the best performing method had a gain of 1–5% in the area under the receiver operating characteristic (AUROC). However, the choice of onset time can cause a greater effect, with variation of 0–6% in AUROC. We illustrate that misleading conclusions can be drawn if models are compared without consideration of the sepsis definition used which emphasizes the need for a standardized definition for sepsis onset.

  • Item type:Article
    ID code:88870
    Dates:
    Date
    Event
    22 January 2024
    Published
    11 January 2024
    Accepted
    4 April 2023
    Submitted
    Subjects:Medicine
    Department:Faculty of Science > Mathematics and Statistics
    Depositing user: Pure Administrator
    Date deposited:22 Apr 2024 12:44
    Last modified:11 Nov 2024 14:16
    URI:https://strathprints.strath.ac.uk/id/eprint/88870
CORE (COnnecting REpositories)