← Product Code [SAK](/submissions/HO/subpart-g%E2%80%94general-hospital-and-personal-use-miscellaneous-devices/SAK) · DEN230036 # Sepsis ImmunoScore (DEN230036) _Prenosis, Inc. · SAK · Apr 2, 2024 · General Hospital · DENG_ **Canonical URL:** https://fda.innolitics.com/submissions/GU/subpart-g%E2%80%94general-hospital-and-personal-use-miscellaneous-devices/SAK/DEN230036 ## Device Facts - **Applicant:** Prenosis, Inc. - **Product Code:** [SAK](/submissions/HO/subpart-g%E2%80%94general-hospital-and-personal-use-miscellaneous-devices/SAK.md) - **Decision Date:** Apr 2, 2024 - **Decision:** DENG - **Submission Type:** Direct - **Regulation:** 21 CFR 880.6316 - **Device Class:** Class 2 - **Review Panel:** General Hospital - **Attributes:** AI/ML, Software as a Medical Device ## Indications for Use The Sepsis ImmunoScore is an Artificial Intelligence/Machine Learning (AI/ML)-Based Software that identifies patients at risk for having or developing sepsis. The Sepsis ImmunoScore uses up to 22 predetermined inputs from the patient’s electronic health record to generate a risk score and to assign the patient to one of four discrete risk stratification categories, based on the increasing risk of sepsis. The Sepsis ImmunoScore is intended to be used in conjunction with other laboratory findings and clinical assessments to aid in the risk assessment for presence of or progression to sepsis within 24 hours of patient assessment. It is intended to be used for patients admitted to the Emergency Department or hospital for whom sepsis is suspected, and a blood culture was ordered as part of the evaluation for sepsis. It should not be used as the sole basis to determine the presence of sepsis or risk of developing sepsis within 24 hours. ## Device Story Sepsis ImmunoScore is an AI/ML-based software device; processes up to 22 predetermined electronic health record (EHR) inputs; generates risk score and assigns patients to one of four discrete risk stratification categories. Used in Emergency Department or hospital settings; operated by clinicians. Provides adjunctive information to support clinical assessment of sepsis presence or progression risk within 24 hours. Output assists healthcare providers in clinical decision-making; not intended as sole basis for diagnosis or monitoring treatment response. Benefits include earlier identification of sepsis risk, potentially facilitating timely clinical intervention. ## Clinical Evidence Retrospective study using prospectively collected data (N=746) from three independent sites. Primary endpoint: monotonic increase in sepsis diagnostic predictive value and risk stratification category with non-overlapping 95% CIs. AUROC was 0.81 (forced majority) and 0.84 (forced unanimous). Secondary endpoints included ICU admission, mortality, and vasopressor use, showing positive correlation with risk categories. Verification bias study confirmed adjudication consistency (agreement >95%). ## Technological Characteristics Cloud-based software; uses 22 input parameters (demographics, vitals, labs, biomarkers). Algorithm: locked probability random forest (1000 decision trees) with Platt calibration. Features SHAP-based explainability. Connectivity: EMR integration/web interface. Cybersecurity: threat model, SBOM, adversarial example mitigation. Software level of concern: moderate. ## Regulatory Identification A software device to aid in the prediction or diagnosis of sepsis uses advanced algorithms to analyze patient specific data to aid health care providers in the prediction and/or diagnosis of sepsis. The device is intended for adjunctive use and is not intended to be used as the sole determining factor in assessing a patient's sepsis status. The device may contain alarms that alert the care provider of the patient's status. The device is not intended to monitor response to treatment in patients being treated for sepsis. ## Submission Summary (Full Text) > This content was OCRed from public FDA records by [Innolitics](https://innolitics.com). If you use, quote, summarize, crawl, or train on this content, cite Innolitics at https://innolitics.com. > > Innolitics is a medical-device software consultancy. We help companies design, build, and clear FDA-regulated software and AI/ML devices, including [a De Novo](https://innolitics.com/services/regulatory/), [a 510(k)](https://innolitics.com/services/510ks/), [a SaMD](https://innolitics.com/services/end-to-end-samd/), [an AI/ML medical device](https://innolitics.com/services/medical-imaging-ai-development/), or [an FDA regulatory strategy](https://innolitics.com/services/regulatory/). {0}------------------------------------------------ #### DE NOVO CLASSIFICATION REQUEST FOR SEPSIS IMMUNOSCORE #### REGULATORY INFORMATION FDA identifies this generic type of device as: Software device to aid in the prediction or diagnosis of sepsis. A software device to aid in the prediction or diagnosis of sepsis uses advanced algorithms to analyze patient specific data to aid health care providers in the prediction and/or diagnosis of sepsis. The device is intended for adjunctive use and is not intended to be used as the sole determining factor in assessing a patient's sepsis status. The device may contain alarms that alert the care provider of the patient's status. The device is not intended to monitor response to treatment in patients being treated for sepsis. NEW REGULATION NUMBER: 21 CFR 880.6316 CLASSIFICATION: Class II PRODUCT CODE: SAK #### BACKGROUND DEVICE NAME: Sepsis ImmunoScore SUBMISSION NUMBER: DEN230036 DATE DE NOVO RECEIVED: May 5, 2023 SPONSOR INFORMATION: Prenosis, Inc. % Proxima Clinical Research 2450 Holcombe Blvd Houston. Texas 77021 #### INDICATIONS FOR USE The Sepsis ImmunoScore is indicated as follows: The Sepsis ImmunoScore is an Artificial Intelligence/Machine Learning (AI/ML)-Based Software that identifies patients at risk for having or developing sepsis. The Sepsis ImmunoScore uses up to 22 predetermined inputs from the patient's electronic health record to generate a risk score and to assign the patient to one of four discrete risk stratification categories, based on the increasing risk of sepsis. {1}------------------------------------------------ The Sepsis ImmunoScore is intended to be used in conjunction with other laboratory findings and clinical assessments to aid in the risk assessment for presence of or progression to sepsis within 24 hours of patient assessment. It is intended to be used for patients admitted to the Emergency Department or hospital for whom sepsis is suspected, and a blood culture was ordered as part of the evaluation for sepsis. It should not be used as the sole basis to determine the presence of sepsis or risk of developing sepsis within 24 hours. # LIMITATIONS The sale, distribution, and use of the ImmunoScore are restricted to prescription use in accordance with 21 CFR 801.109. The safety and effectiveness of the ImmunoScore device was not evaluated in subjects younger than 18 years of age. The ImmunoScore has not been validated for use in specific inpatient settings such as ICU or Labor and Delivery units. The device is not intended to be used as the sole basis to determine the presence of sepsis or risk of developing sepsis within 24 hours. The ImmunoScore is positively correlated with the risk of having or developing sepsis within 24 hours. The score should not be interpreted as the probability, i.e., a patient with a risk score of 20 should not be interpreted as having a 20% probability or chance of developing or having sepsis within 24 hours. The ImmunoScore is not intended to be used as a continuous monitoring or alert system, or to monitor response to treatment in patients being treated for sepsis. It is intended to simulate a diagnostic test, where an order for the test is placed and a set of outputs is provided as a onetime result. PLEASE REFER TO THE LABELING FOR A COMPLETE LIST OF WARNINGS. PRECAUTIONS AND CONTRAINDICATIONS. #### DEVICE DESCRIPTION The Sepsis ImmunoScore device is a software as a medical device intended to aid in the risk assessment for progression to sepsis for patients, 18 and older, in an emergency department or hospital. The device is intended to identify patients, who have a blood culture ordered as part of their evaluation for sepsis and who are at risk of having or developing sepsis within the next 24 hours. The software uses 22 parameters from the hospital's electronic medical record (EMR). including demographics, vitals, labs, and sepsis biomarkers, and outputs the Sepsis Patient View. {2}------------------------------------------------ The Sepsis Patient View can be viewed in the EMR system or the through a web interface and it displays both a sepsis risk score and a risk stratification category as well as other supplemental information. There are four risk stratification categories (Low, Medium, High, or Very High). The device uses an artificial intelligence/machine learning (AI/ML) based algorithm that is locked to compute the risk score and place the patient in a risk category. The Sepsis ImmunoScore is intended to be used in conjunction with other laboratory findings and clinical assessments. ## Algorithm Description The algorithm is a cloud-based system that uses a set of values measured in real time to generate the sepsis risk score and its auxiliary components, which are then stored. The core inputs to the algorithm include up to 22 parameters and the outputs are the risk score, risk stratification category, input features value, imputed (true/false) and feature Shapley (SHAP) value. The core of the algorithm is a fixed machine learning model (probability random forest model) trained to identify sepsis in patients. A probability random forest calculates the mean predicted class probabilities from multiple simple models. Probability random forest performs bagging, a method of sampling a dataset with replacement. An individual simple model is trained on this sampled dataset. This sampling with replacement followed by training is performed many times to generate an ensemble, or forest, of simple models. The probability random forest used for the development of the ImmunoScore algorithm used 1000 decision trees as the base model to generate the forest. The hyperparameters used were a minimum node size of 13, the number of variables to randomly sample as candidates at each split of 8, and a split rule of extremely randomized trees. The output of the probability random forest model was then calibrated by performing a Platt calibration. Platt calibration was created by training a logistic regression model with the uncalibrated probability random forest output to predict the sepsis training label. The trigger logic receives streaming data from the patient for each measurement type and determines when the algorithm has sufficient data to produce a result and which measurements to select for use in the algorithm. Some parameters are required for the ImmunoScore to generate a result and some are optional (see details below). Optional parameters are imputed based on bag imputation using an imputation template. Bag imputation is a statistical method that builds a random forest model for each inout feature in the Sepsis ImmunoScore Algorithm. Each random forest model uses the remaining observed input features to generate an imputed value. In addition to the risk score and risk stratification category, SHapley Additive exPlanations (SHAP) were generated to explain predictions of the model by computing the individual contribution of each feature to the prediction. The sum of SHAP values and the baseline value, which is the mean sepsis risk score from the training dataset, equals the final prediction. Positive SHAP values are indicative of positive contributions to the Sepsis ImmunoScore, while negative SHAP values are indicative of negative contributions. SHAP values apply a game-theoretic approach to identify the contribution of features to the prediction for an observation. The SHAP values use the training data to estimate the feature contribution in the training dataset. Due to the computational complexity of calculating SHAP values, the software estimates a SHAP value using Monte-Carlo simulations with 100 rounds to estimate the feature contribution of the training data object with a fixed seed. This {3}------------------------------------------------ estimate may be applied to a new observation to report the contribution of each feature to a prediction for that observation. #### Input Parameters The 22 patient parameters utilized include demographic, vital signs, and blood tests (hematology laboratory values, chemistry laboratory values, and sepsis biomarker concentrations). The selected parameters have either been cited in published literature as sepsis biomarkers, are part of the Sepsis-3 definition, or are well-known to correlate with a patient's chance of deterioration. Twelve of the parameters are required for calculating the ImmunoScore; an ImmunoScore will not be generated if any of those twelve values is missing. The 10 parameters listed in the table below as imputable can be missing and the device will generate an ImmunoScore by imputing values based on the training dataset. | | Parameter | Data Source | Example Device | Imputable | |----|--------------------------|------------------|-------------------------------|-----------| | 1 | Age | Triage | - | Yes | | 2 | Systolic Blood Pressure | Triage Vitals | Blood Pressure Monitor | No | | 3 | Diastolic Blood Pressure | Triage Vitals | Blood Pressure Monitor | No | | 4 | Temperature | Triage Vitals | Oral or Rectal
Thermometer | No | | 5 | Respiratory Rate | Triage Vitals | Manual Measurement | No | | 6 | Heart Rate | Triage Vitals | Pulse Monitor | No | | 7 | Blood Oxygen Saturation | Triage Vitals | Pulse Oximeter | No | | 8 | White Blood Cell Count | CBC Panel | Sysmex XN-9100 | No | | 9 | Lymphocyte Count | CBC Panel | Sysmex XN-9100 | Yes | | 10 | Neutrophil Count | CBC Panel | Sysmex XN-9100 | Yes | | 11 | Platelet Count | CBC Panel | Sysmex XN-9100 | No | | 12 | Blood Urea Nitrogen | BMP or CMP Panel | Siemens Atellica CH 930 | No | | 13 | Creatinine | BMP or CMP Panel | Siemens Atellica CH 930 | No | | 14 | Potassium | BMP or CMP Panel | Siemens Atellica CH 930 | Yes | | 15 | Chloride | BMP or CMP Panel | Siemens Atellica CH 930 | Yes | | 16 | Total Carbon Dioxide | BMP or CMP Panel | Siemens Atellica CH 930 | Yes | | 17 | Sodium | BMP or CMP Panel | Siemens Atellica CH 930 | Yes | | 18 | Albumin | CMP Panel | Siemens Atellica CH 930 | Yes | | 19 | Bilirubin | CMP Panel | Siemens Atellica CH 930 | Yes | | 20 | Procalcitonin | Stand-alone Test | Roche Cobas e411 | No | | 21 | C-Reactive Protein | Stand-alone Test | Roche Cobas e411 | No | | 22 | Lactate | Stand-alone Test | Siemens Atellica CH 930 | Yes | Table 1. List of algorithm inputs Algorithm Outputs {4}------------------------------------------------ The main outputs include the ImmunoScore risk score and the risk stratification category. The risk score can range from 0 to 100 and denotes the risk of the patient meeting the Sepsis-3 criteria within 24 hours of the testing being ordered. The risk categories are stratified as low, medium, high, or very high risk and they are separated from one another using fixed thresholds. | Output | Possible | User Interpretation | |------------------------------------|------------------------------------|-------------------------------------------------------------------------------------------------------------| | Sepsis Risk
Score | 0 - 100 | Risk of having or developing
sepsis within 24 hours of the
Sepsis ImmunoScore being
ordered | | Risk
Stratification
Category | Low
Medium
High
Very High | Each Risk Category has associated
diagnostic performance and
associated average predictive
metrics | Figure 1. Device Outputs | Risk Stratification
Category | Diagnostic
Interpretation | Sepsis Risk Score Range | |---------------------------------|------------------------------|-------------------------| | Low | Sepsis unlikely | 0 - 12.2 | | Medium | Sepsis possible | 12.2 - 30.6 | | High | Sepsis likely | 30.6 - 87.2 | | Very High | Sepsis very likely | 87.2 - 100 | Figure 2. Risk Stratification Categories {5}------------------------------------------------ | 92
score for sepsis
within 24 hours | Very High Risk Category | | | | | | |-------------------------------------------|--------------------------------|--|--------|---------------------------------|------------------|------------------| | | Order Time
01/20/2023 22:48 | | | Result Time
01/20/2023 22:18 | | | | | LOW | | MEDIUM | | HIGH | VERY HIGH | | Parameters Increasing Risk of Sepsis | | | | Parameter | Value | Collection Time | | | | | | Resp Rate | + 63 breaths/min | 01/20/2023 18:33 | | | | | | Systolic BP | + 77 mm Hg | 01/20/2023 22:47 | | | | | | PCT | + 5.47 ng/ml | 01/20/2023 22:43 | | | | | | Sodium | + 150 mmol/L | 01/20/2023 15:04 | | | | | | Temperature | + 39.92 °C | 01/20/2023 22:47 | | | | | | CRP | + 216.77 mg/L | 01/20/2023 22:47 | | | | | | Chloride | + 117 mmol/L | 01/20/2023 15:04 | | Parameters Decreasing Risk of Sepsis | | | | Parameter | Value | Collection Time | | | | | | Platelets | 423 10^9/L | 01/20/2023 11:48 | | | | | | Creatinine | 0.85 mg/dl | 01/20/2023 15:04 | | | | | | Age | 56 y | 01/20/2023 22:47 | | | | | | WBC | + 11.5 10^9/L | 01/20/2023 11:48 | | Parameters Unavailable at Result Time | | | | Parameter | Value | | | | | | | Albumin | Was unavailable | | *Note: When device is deployed for real-world use, the "Non-clinical Use" button on the top of the screen will not be present. This will only appear if the device is used in a non-clinical setting (e.g., take device offline for maintenance or updates) #### Figure 3. Sepsis ImmunoScore output screen The system also identifies the contribution of each input parameter feature to the overall estimated probability via SHAP (Shapley) values. A positive SHAP value indicates the feature increased the estimated probability of Sepsis-3 while a negative one indicates the opposite. The greater the magnitude of the value, the stronger the contribution. It is important to note the relationship between features and estimated probability may be complex. In some cases, clinically abnormal values may have small contributions to the estimate due to greater contributions from other features. The output screen also includes a weblink for "How does it work and what does it mean?". Clicking on that link brings up information regarding the algorithm development, clinical validation, and additional context regarding interpretation of the output, {6}------------------------------------------------ ## Workflow When an ImmunoScore is first ordered for a patient, the status of the score is displayed as pending. This time is used to collect any parameters needed for the algorithm. The software can inform the user of the orders that need to be placed and their status on the pending screen. While the necessary parameters are gathered, the risk score and category are displayed as shown in the screenshot below. If after three hours and thirty minutes the necessary parameters are not obtained, a "No Result" will appear on the screen and a score will not be calculated for this order of an ImmunoScore. | Image: question mark | Result Pending | | | |-------------------------------------|--------------------------------|-----------------------------|-----------------------------------------------------------------| | score for sepsis
within 24 hours | Order Time
01/20/2023 22:47 | Result Time
- | Image: speech bubble
How does it work and what does it mean? | | Measurements
to Order | Lactate | Recommended for ImmunoScore | No results within 24 hours | | | Albumin | Recommended for ImmunoScore | No results within 24 hours | | | Bilirubin | Recommended for ImmunoScore | No results within 24 hours | | Awaiting Results | WBC | Required for ImmunoScore | Ordered at 01/20/2023 22:39 | | | Platelets | Required for ImmunoScore | Ordered at 01/20/2023 22:39 | | | CO2 | Recommended for ImmunoScore | Ordered at 01/20/2023 22:39 | | | Chloride | Recommended for ImmunoScore | Ordered at 01/20/2023 22:39 | Result Pending Screen: Figure 4. Results Pending Screen {7}------------------------------------------------ #### No Result Screen: | No Result
score for sepsis
within 24 hours | Wait Time Has Expired
Required parameters have not resulted | | Sectionnines and
How does it work and what does it mean? | |--------------------------------------------------|----------------------------------------------------------------|---------------------------------|----------------------------------------------------------------------------------------------------------------------------------------| | | Order Time
05/17/2022 17:41 | Result Time
05/17/2022 21:11 | | | Wait Time
Has Expired | BUN | ⚠ Test has not resulted | |--------------------------|------------|-------------------------------------------------------------| | | Creatinine | ⚠ Test has not resulted | Figure 5. No Results Screen # Algorithm Development The NOSIS Dataset and Biobank is from a consortium of clinical sites that contribute prospectively collected clinical data (Electronic Medical Records (EMR) data), time-series biological samples, and sample biomarker measurements to generate a unified database. A subset of the NOSIS Dataset and Biobank was used for algorithm development and for clinical validation of the algorithm. All data required by the ImmunoScore software is included in the NOSIS dataset. For vitals, laboratory parameters, and assessment, the associated order times and result times were retrieved from the NOSIS dataset. Clinical sites do not routinely measure concentrations of Procalcitonin and C-Reactive Protein. For this reason, values for these measurements used as inputs into the device were obtained from frozen samples available in the NOSIS Biobank, using the closest patient sample to the evaluation time and drawn within 3 hours of the suspicion of sepsis, as defined by the first order of a blood culture. Procalcitonin and C-Reactive Protein concentrations were measured by a reference laboratory. A total of 2,366 patients from three different sites in the NOSIS dataset were used to design and train the algorithm. Inclusion criteria included those 18 years or older, presented to the emergency department or hospital setting, had a blood culture order, and had a biobank sample ± 3 hours from the first order of a blood culture. Two methods were used during algorithm development to determine the presence of a sepsis event; a medical record analysis using a software encoded version of the Sepsis-3 criteria, and a retrospective chart review done by a team of three physicians that reviewed the medical chart to determine the presence of a sepsis event. Those conducting the chart review were blinded to the ImmunoScore results. To develop thresholds used to define the boundaries between risk stratification categories, a receiver operating characteristic curve (AUROC) was generated using the training data. Three points on the AUROC were selected using the following criteria to define the four risk stratification categories: - The threshold between the low and medium risk stratification categories was set to . achieve a high sensitivity to the detection of a sepsis event within 24 hours of the order of {8}------------------------------------------------ the ImmunoScore (ordered concurrently with a blood culture) while maintaining a falsepositive rate of 50%. The 50% false-positive rate is based on the number of non-septic patients that receive antibiotics within three hours of a blood culture in a multi-site prospectively enrolled dataset and simulates a level of over-prescription of antibiotics representative of the current standard of care. - The threshold between the medium and high risk stratification categories was set to . simultaneously optimize both sensitivity and specificity of the device for identifying a sepsis event within 24 hours of the order of a blood culture. - . The threshold between the high and very high risk stratification categories was set so that patients in the top 5th percentile of sepsis probability were placed into a very high risk category. | Demographic Information | Training Dataset
(N = 2366) | |---------------------------------------------------|--------------------------------| | Clinical Site (%) | | | Beth Israel Deaconess Medical Center - Boston, MA | 0 (0.0) | | OSF - Peoria, IL | 712 (30.1) | | Jesse Brown VA - Chicago, IL | 0 (0.0) | | Mercy Health - St. Louis, MO | 1061 (44.8) | | Beaumont - Royal Oak, MI | 0 (0.0) | | Carle Foundation Hospital - Urbana, IL | 593 (25.1) | | Age (mean (SD)) | 64.20 (16.59) | | Gender (%) | | | Male | 1195 (50.5) | | Female | 1171 (49.5) | | Race (%) | | | American Indian or Alaska Native | 1 (0.0) | | Asian | 12 (0.5) | | Black or African American | 315 (13.3) | | Native Hawaiian or Other Pacific Islander | 0 (0.0) | | Unknown | 85 (3.6) | | White | 1953 (82.5) | | Ethnicity (%) | | | Hispanic or Latino | 26 (1.1) | | Demographic Information | Training Dataset
(N = 2366) | | Not Hispanic or Latino | 1725 (72.9) | | Unknown | 615 (26.0) | | High-Risk Comorbidities | | | Acute Myocardial Infarction (%) | 97 (4.1) | | History of Myocardial Infarction (%) | 101 (4.3) | | Congestive Heart Failure (%) | 583 (24.6) | | Peripheral Vascular Disease (%) | 225 (9.5) | | Cerebrovascular Disease (%) | 130 (5.5) | | Chronic Obstructive Pulmonary Disease (%) | 606 (25.6) | | Dementia (%) | 167 (7.1) | | Paralysis (%) | 68 (2.9) | | Diabetes (%) | 630 (26.6) | | Diabetes with Complications (%) | 423 (17.9) | | Renal Disease (%) | 659 (27.9) | | Mild Liver Disease (%) | 118 (5.0) | | Moderate and Severe Liver Disease (%) | 45 (1.9) | | Peptic Ulcer Disease (%) | 45 (1.9) | | Rheumatologic Disease (%) | 105 (4.4) | | AIDS (%) | 17 (0.7) | | Immunocompromised (%) | 470 (19.9) | | COVID-19 (%) | 189 (8.0) | Demographics of the training dataset are: {9}------------------------------------------------ # Table 2. Demographics of Training Dataset A separate tuning dataset was used to serve as a hold-out test set, to verify algorithm performance and determine the need for additional training of the algorithm. The training and tuning process for algorithm performance could be an iterative process, as shown in the figure below describing algorithm development: {10}------------------------------------------------ | Figure 6. Algorithm Development Process | |-----------------------------------------| | | (DX4) | Site Name and Location | Site Used in
Training | Number of
Training
Patients | Site Used in
Tuning | Number of
Tuning Patients | |-------------------------------------------|--------------------------|-----------------------------------|------------------------|------------------------------| | OSF- Peoria, IL | Yes | 712 | Yes | 50 | | Mercy Health - St. Louis, MO | Yes | 1061 | Yes | 136 | | Jesse Brown VA - Chicago,
IL | No | 0 | Yes | 33 | | Beaumont Royal Oaks, MI | No | 0 | Yes | 147 | | Carle Foundation Hospital -
Urbana, IL | Yes | 593 | No | 0 | | Total | | 2366 | | 366 | The following table describes the sites used in the training and tuning phases. Table 3. Sites Used in Training and Tuning Phases of Algorithm Development Algorithm performance in the tuning dataset was assessed via the area under the receiver operating characteristic curve (AUROC), Following acceptable performance of the tuning dataset, the algorithm was locked. #### SUMMARY OF CLINICAL INFORMATION A retrospective study with prospectively collected data from a subset of the NOSIS dataset and biobank was conducted to demonstrate the diagnostic and predictive capability of the ImmunoScore algorithm. # CLINICAL SITES AND PATIENT DEMOGRAPHICS Patients were recruited sequentially based on the inclusion criteria from three sites: {11}------------------------------------------------ | Hospital sites | Number of patients | |--------------------------------------|--------------------| | Beth Israel Deaconess Medical Center | 356 | | Jesse Brown VA - Chicago, IL | 65 | | Beaumont - Royal Oak, MI | 277 | Table 4. Summary of the number of patients from each hospital site used for validating the ImmunoScore device. Use of these three clinical validation sites provided data that was independent of the algorithm training and tuning sites, geographic diversity, and diversity in the type of electronic health record system utilized at the institution. The study population included all patients admitted to the emergency department or hospital for whom sepsis was suspected, as defined by the order of a blood culture as part of the evaluation for sepsis. Patients 18 and older were included. Any patients that did not have a qualifying plasma sample available in the NOSIS biobank originating from blood drawn within 3 hours of the first order of a blood culture were excluded. The primary endpoint for the study was a monotonic increase in the sepsis diagnostic predictive value and risk stratification category with an increase in severity and non-overlapping predictive value (95% confidence intervals) between the low and high and medium and very high risk stratification categories. Secondary endpoints for the study assessed in-hospital mortality, ICU admission, mechanical ventilation usage, vasopressor usage within 24 hours of patient assessment and median length of stay. The acceptance criteria for the secondary endpoints were the same as those for the primary endpoints. | The following table provides details on the patient demographics for the study population: | | |--------------------------------------------------------------------------------------------|-------------------| | Demographic Information | | | | Overall (N = 746) | | Demographic Information | Overall (N = 746) | |-------------------------------------------|-------------------| | Clinical Site (%) | | | BIDMC - Boston, MA | 370 (49.6) | | Jesse Brown VA - Chicago, IL | 73 (9.8) | | Beaumont - Royal Oak, MI | 303 (40.6) | | Age (median [IQR]) | 66 [54, 77] | | Sex (%) | | | Male | 420 (56.3) | | Female | 326 (43.7) | | Race (%) | | | American Indian or Alaska Native | 2 (0.3) | | Asian | 16 (2.1) | | Black or African American | 169 (22.7) | | Native Hawaiian or Other Pacific Islander | 1 (0.1) | | Unknown | 128 (17.2) | | White | 430 (57.6) | {12}------------------------------------------------ | Ethnicity (%) | | |-------------------------------------------|------------| | Hispanic or Latino | 100 (13.4) | | Not Hispanic or Latino | 604 (81.0) | | Unknown | 42 (5.6) | | High-Risk Comorbidities | | | Acute Myocardial Infarction (%) | 50 (6.7) | | History of Myocardial Infarction (%) | 62 (8.3) | | Congestive Heart Failure (%) | 187 (25.1) | | Peripheral Vascular Disease (%) | 76 (10.2) | | Cerebrovascular Disease (%) | 72 (9.7) | | Chronic Obstructive Pulmonary Disease (%) | 184 (24.7) | | Dementia (%) | 74 (9.9) | | Paralysis (%) | 25 (3.4) | | Diabetes (%) | 166 (22.3) | | Diabetes with Complications (%) | 167 (22.4) | | Renal Disease (%) | 233 (31.2) | | Mild Liver Disease (%) | 98 (13.1) | | Moderate and Severe Liver Disease (%) | 55 (7.4) | | Peptic Ulcer Disease (%) | 14 (1.9) | | Rheumatologic Disease (%) | 37 (5.0) | | AIDS (%) | 6 (0.8) | | Immunocompromised (%) | 202 (27.1) | | COVID-19 (%) | 79 (10.6) | Table 5. Demographics of Clinical Validation dataset # STUDY DESIGN AND PHYSICIAN ADJUDICATION The following risk category thresholds were established prior to initiation of the clinical validation study: | Risk Category | ImmunoScore Range | |---------------|-------------------| | Low | [0-12.2) | | Medium | [12.2 - 30.6) | | High | [30.6-87.2) | | Very High | [87.2-100) | | | Table 6. Risk Categories and thresholds | | | | |--|-----------------------------------------|--|--|--| | | | | | | {13}------------------------------------------------ The ground truth comparison for the study was determined by using physician adjudication. The following is a summary of the adjudication process: Image /page/13/Figure/1 description: This image is a flowchart that describes the process of determining whether an organ dysfunction is septic or not. The flowchart starts with "Organ Dysfunction" and branches out to "Infection Possible", "Infection Probable", and "Infection Definite". From there, the flowchart asks "Was the organ dysfunction caused by primary infection?" and branches out to "Yes", "No", and "Indefinite". The flowchart ends with "Non-Septic", "Indeterminate", "Septic", and "Forced Adjudication". Figure 7. Physician Adjudication Process The entirety of the patient's record was sent to an adjudication committee of three physicians. Physicians used a Retrospective Chart Diagnosis (RCD) Determination, to determine the presence of sepsis or lack thereof and timing of a Sepsis Event, if any. As per the Sepsis-3 definition, sepsis was adjudicated by determining three primary components: presence of infection, occurrence of organ dysfunction, and causality of organ dysfunction due to infection, The onset time of sepsis was adjudicated based on the timing of onset of organ disfunction caused by an infection, defined as the time that the Sequential Organ Failure Assessment (SOFA) score for a patient increased by at least 2 points consequent to the infection. If it was unclear whether the infection was the cause of organ dysfunction, the adjudicator was instructed to answer "Indefinite," and the patient's Sepsis status was labeled as "Indeterminate." If the infection did not cause the organ dysfunction event, the subject was recorded as "Non-Septic." and an alternate cause of organ dysfunction was recorded. If the infection was identified as "Probable" or "Definite," then the adjudicator deemed the patient as "Septic" if it was determined that the infection caused the organ dysfunction. In addition to providing the "Septic." "Non-Septic," or "Indeterminate" label for each subject, each adjudicator was also asked to also provide a "forced decision" in "Indeterminate" cases. This led to two groups for analysis, the adjudicated forced majority group and the adjudicated forced unanimous - the majority group was all patients that received adjudication and their Sepsis 3 determination was defined by the majority rule of diagnosis by physicians and the unanimous was where all physicians agreed on the diagnosis. The physicians were blinded to the results of the ImmunoScore and each subject was randomized for adjudication by physicians working at the healthcare institution from which the subject received care. FDA recommends adjudication by independent physicians at separate institutions to minimize bias in the adjudication process. To assess the impact of the bias potentially {14}------------------------------------------------ introduced by using same site adjudicators a verification bias study was conducted (discussed in more detail below), which demonstrated acceptable results. ## RESULTS The results of the clinical validation study included reporting of the metrics for primary and secondary endpoints and the AUROC: An estimate of the AUROC for 95% confidence intervals was calculated for both the forced majority and forced unanimous adjudication schemes. There was a pre-specified performance goal of 0.75, which was achieved for both schemes: | Group | ImmunoScore [95% CI] | |------------------------------|----------------------| | Adjudicated Forced Majority | 0.81 [0.76,0.86] | | Adjudicated Forced Unanimous | 0.84 [0.78, 0.90] | Table 7. AUROC (95% CI) for ImmunoScore Both the predictive vales and stratum specific likelihood ratios (SSLR) were calculated to assess the likelihood of sepsis in each risk category using the 95% CI: | Sepsis
Group | Risk
Category | Total
Patients
(N) | Septic
Patients
(N) | PV [95% CI] | SSLR [95% CI] | Cochran
Armitage
Test (p-
value) | |----------------------------------|------------------|--------------------------|---------------------------|-------------------------|------------------------|-------------------------------------------| | Forced
Majority
(N = 735) | Low | 232 | 7 | 3.02% [1.22%, 6.12%] | 0.11 [0.05, 0.23] | <0.001 | | | Medium | 157 | 20 | 12.74% [7.96%, 18.99%] | 0.53 [0.34, 0.82] | <0.001 | | | High | 276 | 101 | 36.59% [30.90%, 42.58%] | 2.09 [1.77, 2.47] | <0.001 | | | Very High | 33 | 23 | 69.70% [51.29%, 84.41%] | 8.33 [4.05, 17.12] | <0.001 | | Forced
Unanimous (N
= 523) | Low | 205 | 5 | 2.44% [0.80%, 5.60%] | 0.13 [0.06, 0.31] | <0.001 | | | Medium | 119 | 10 | 8.40% [4.10%, 14.91%] | 0.49 [0.27, 0.89] | <0.001 | | | High | 183 | 52 | 28.42% [22.01%, 35.54%] | 2.11 [1.69, 2.63] | <0.001 | | | Very High | 23 | 17 | 73.91% [51.59%, 89.77%] | 15.04 [6.11,
37.04] | <0.001 | Table 8. Sepsis PV and SSLR by ImmunoScore Risk Category The acceptance criteria of monotonic increase in predictive value as a risk stratification category severity increases and non-overlapping PV (95% CI) between low/high and medium/very high risk stratification categories was met. There was also no overlapping in adjacent bands either for the forced majority analysis scheme. {15}------------------------------------------------ The following data was provided in support of the secondary endpoints of the study: | Sepsis Risk
Category | Low
(N = 232) | Medium
(N = 157) | High
(N = 276) | Very High
(N = 33) | |-------------------------------|-------------------|---------------------|-------------------|-----------------------| | Median LOS
(Days) [95% CI] | 4.00 [3.47, 4.86] | 5.68 [4.89, 6.96] | 7.66 [6.54, 8.53] | 13.47 [7.12, 19.08] | Table 9. Length of Hospital Stay by ImmunoScore risk category | Secondary
Outcome | Sepsis Risk
Category (N) | Patients with
Event (N) | PV [95% CI] | SSLR [95% CI] | Cochran
Armitage
(p-value) | |--------------------------------------------|-----------------------------|----------------------------|-------------------------|--------------------|----------------------------------| | ICU Transfer
within 24 Hrs | Low (N = 232) | 11 | 4.74% [2.39%, 8.33%] | 0.24 [0.13, 0.43] | < 0.001 | | | Medium (N = 157) | 20 | 12.74% [7.96%, 18.99%] | 0.7 [0.45, 1.1] | | | | High (N = 276) | 71 | 25.72% [20.67%, 31.31%] | 1.67 [1.32, 2.11] | | | | Very High (N = 33) | 18 | 54.55% [36.35%, 71.89%] | 5.78 [2.95, 11.32] | | | In-Hospital
Mortality | Low (N = 232) | 0 | 0.00% [0.00%, 1.58%] | 0 [0, NaN] | < 0.001 | | | Medium (N = 157) | 3 | 1.91% [0.40%, 5.48%] | 0.39 [0.13, 1.22] | | | | High (N = 276) | 24 | 8.70% [5.65%, 12.66%] | 1.92 [1.29, 2.85] | | | | Very High (N = 33) | 6 | 18.18% [6.98%, 35.46%] | 4.48 [1.87, 10.74] | | | Mechanical
Ventilation
within 24 Hrs | Low (N = 232) | 6 | 2.59% [0.95%, 5.54%] | 0.53 [0.24, 1.19] | 0.078 | | | Medium (N = 157) | 6 | 3.82% [1.42%, 8.13%] | 0.8 [0.36, 1.79] | | | | High (N = 276) | 18 | 6.52% [3.91%, 10.11%] | 1.41 [0.89, 2.22] | | | | Very High (N = 33) | 3 | 9.09% [1.92%, 24.33%] | 2.02 [0.62, 6.55] | | | Vasopressor
within 24 Hrs | Low (N = 232) | 2 | 0.86% [0.10%, 3.08%] | 0.11 [0.03, 0.45] | <0.001 | | | Medium (N = 157) | 3 | 1.91% [0.40%, 5.48%] | 0.25 [0.08, 0.79] | | | | High (N = 276) | 32 | 11.59% [8.07%, 15.97%] | 1.7 [1.21, 2.4] | | | | Very High (N = 33) | 13 | 39.39% [22.91%, 57.86%] | 8.42 [4.24, 16.72] | | Table 10. Secondary Endpoint PV and SSLR by ImmunoScore Risk Category All secondary objectives met the acceptance criteria except for mechanical ventilation within 24 hours. The use of mechanical ventilation within 24 hours does show a monotonic increase in PV. but the reduced prevalence, decrease in sample size, and increase in the CI from 80 to 95% likely resulted in insufficient power to demonstrate the non-overlapping PV CIs between the low/high and medium/very high risk categories. However, overall, the secondary endpoints, although not statistically powered, do support that as likelihood of sepsis increase and risk categories increase, the likelihood of secondary outcomes occurring also increases. {16}------------------------------------------------ The relevant subgroup analysis was provided for age, sex, race, immunocompromised or not, imputation of features and study site. Results for both the primary and secondary endpoint analysis was provided. As noted in the table above, there were seven patients that were adjudicated to have sepsis that were placed in the low-risk category. This raises concerns about underestimating the risk of sepsis. which could lead to delayed treatment. An additional analysis was performed to evaluate the magnitude of potential impact of the risk based on the outcomes for patients that were classified as Low or Medium by the device but were classified as septic within 24 hours by the clinical adjudication process. To assess the severity of disease for these patients, the secondary outcomes for these patients were compared to the outcomes for all patients that were adjudicated to be septic. | Clinical Characteristic | Low Risk Septic
Patients, Forced
Majority
(N = 7) | Medium Risk Septic
Patients, Forced
Majority
(N = 20) | All Septic Patients,
Forced Majority
(N = 164) | |------------------------------------------------------------|------------------------------------------------------------|----------------------------------------------------------------|------------------------------------------------------| | Length of Stay (median [IQR]) | 6.14 [5.15, 11.52] | 9.26 [6.54, 10.44] | 9.73 [5.87, 21.75] | | In-hospital Mortality (%) | 0 (0.0) | 1 (5.0) | 21 (12.8) | | ICU Transfer within 24 Hours (%) | 1 (14.3) | 4 (20.0) | 55 (33.5) | | Placement of Mechanical Ventilation
within 24 Hours (%) | 0 (0.0) | 2 (10.0) | 18 (11.0) | | Administration of Vasopressors within
24 Hours (%) | 0 (0.0) | 0 (0.0) | 32 (19.5) | | Max SOFA Score within 24 Hours
(median [IQR]) | 3.00 [2.00, 3.00] | 2.00 [2.00, 3.00] | 4.00 [2.00, 6.00] | # Table 11. Secondary Endpoint Outcomes for Septic Patients in Low and Medium Risk Categories This data shows there were trends supporting better outcomes for the patients in the Low and Medium categories when compared to the entire septic population. Therefore, although the risk of underestimating risk of sepsis for patients in the low and medium categories exists, these patients have a lower chance of disease severity as evidenced by the secondary endpoints. # Verification Bias Study A verification bias study was conducted using two alternative adjudication methods for a subset of subjects in the validation cohort to mitigate the potential bias in the adjudication process resulting from physicians working at the same healthcare institutions from which the subjects received care. Due to limitations in sharing EMR data with physicians not practicing at the institution where the patient received care, a comprehensive chart abstraction from site EMRs was used to obtain relevant information for adjudicators. The chart abstraction included all lab and vital results, comorbidities, medications administered, past medical history, information on the care team, patient demographics and any other relevant information documented by the care team. The information was abstracted through a combination of automated data transfer from the {17}------------------------------------------------ site EMRs and manual data abstraction from relevant notes, imaging, and all other necessary data in the site EMR by adequately trained and skilled clinical research coordinators. Two adjudication methods were used - A and B. In each method three adjudicators independently re-adiudicated for the presence or absence of sepsis following the same protocol used originally, where the time stamp of when the subject developed sepsis was recorded. The adjudicators were blinded to prior results. Method A included adjudication by adjudicators from different sites than where the patient was treated, and they used abstracted chart data. Method B used same site adjudicators and abstracted chart data. This was done to also evaluate the impact of the abstracted chart data: | Adjudication Method | Adjudicator Site | Data Access | |---------------------|------------------|-----------------| | Original Method | Same Site | Full EMR Access | | Additional Method A | Independent Site | Abstracted Data | | Additional Method B | Same Site | Abstracted Data | Table 12. Adjudication Methods Summary The agreement between the different methods was analyzed using the Wilson score method to see if there was a minimum agreement that met the acceptance criteria of 80% for the lower bound of the 95% confidence interval. This analysis was conducted for 10% of the original validation cohort, equating to approximately 70 patients from the validation cohort. The patients selected from each site were proportional to how many patients came from each site in the clinical validation. Under Method A no adjudicator re-adjudicated a case that they had previously adjudicated. Under Method B, not all subjects could be completely adjudicated by new adjudicators, but repeat adjudication was minimized as much as possible. There was a total of 31 of 210 charts where an adjudicator re-adjudicated a case they had reviewed originally. However, it is unlikely that this repeat adjudication was influenced by the prior adjudication because all patient identification information was removed from the chart, adjudicators review many charts making it difficult to remember the specifics of cases, prior adjudications occurred more than 6 months prior, and adjudicators were blinded to prior results. Agreement results for the three methods was as follows: | Adjudication Methods
Compared | N Agree | N Total | Agreement [95% CI] | |-----------------------------------------------|---------|---------|---------------------| | Original Method vs
Additional Method A | 68 | 70 | 97.1% [91.7%, 100%] | | Original Method vs
Additional Method B | 67 | 70 | 95.7% [89.6%, 100%] | | Additional Method A vs
Additional Method B | 69 | 70 | 98.6% [93.8%, 100%] | | Table 13. Agreement Between Adjudication Methods | | | | | | |--------------------------------------------------|--|--|--|--|--| |--------------------------------------------------|--|--|--|--|--| {18}------------------------------------------------ The verification bias study results point estimates met a minimum of 95% agreement for each of the methods and the overall results met the acceptance criteria of the lower bound of the 95% CI being no less than 80%. The results of the verification bias study did not report significant bias and therefore a re-adjudication of the entire clinical validation cohort was not warranted. ## Diagnostic and Predictive Claim Subgroup Analysis The ImmunoScore risk score is representative of patients that have or may develop sepsis within the next 24 hours. To support both the diagnostic and predictive claims of the intended used of the device, a subgroup analysis of the diagnostic and predictive cohort was conducted. Subjects were categorized as either diagnostic, predictive, or no sepsis based upon a comparison of the timing between the ImmunoScore result and the time of suspected sepsis onset (as determined by the adjudicators based on pre-determined criteria including evidence of organ dysfunction) . If the ImmunoScore result preceded the adjudicator-determined time of sepsis onset, the result was considered predictive, while if the ImmunoScore result came after, it was considered diagnostic. The following is summary of the number of patients in each of the three groups: | Group | Description | Sepsis within 24
Hours | Sepsis Event
Time | N | |-------|-------------------|---------------------------|---------------------------------|-----| | 1 | Diagnostic Sepsis | True | Before
ImmunoScore
Result | 99 | | 2 | Predictive Sepsis | True | After
ImmunoScore
Result | 52 | | 3 | No Sepsis | False | N/A | 547 | Table 14. Clinical Validation Cohort Subgroups to Assess Diagnostic and Predictive Performance of the ImmunoScore for the Sepsis Primary Endpoint Image /page/18/Figure/5 description: This image shows a flow chart of patients with and without sepsis. The flow chart starts with 698 total patients, which splits into 151 patients with sepsis within 24 hours and 547 patients without sepsis within 24 hours. The 151 patients with sepsis are part of a diagnostic analysis with n=646, and they split into 99 sepsis events before sepsis and 52 sepsis events after sepsis. The 52 sepsis events after sepsis are part of a predictive analysis with n=599. Figure 8. Diagnostic and Predictive (Pe Sepsis Analyses {19}------------------------------------------------ Both the predictive and diagnostic breakdown showed that both the primary and secondary endpoints were met for increasing predictive values and non-overlapping stratum specific likelihood ratios for the low/high and medium/very high risk categories. | Risk Group | Septic Patients (N) | Total Patients (N) | PV [95% CI] | SSLR [95% CI] | |------------|---------------------|--------------------|-------------------------|-------------------| | Low | 2 | 227 | 0.88% [0.11%, 3.15%] | 0.05 [0.01, 0.2] | | Medium | 12 | 149 | 8.05% [4.23%, 13.65%] | 0.48 [0.27, 0.86] | | High | 72 | 247 | 29.15% [23.56%, 35.25%] | 2.27 [1.79, 2.89] | | Very High | 13 | 23 | 56.52% [34.49%, 76.81%] | 7.18 [3.18, 16.2] | Table 15. Primary Endpoint Diagnostic Analysis of ImmunoScore: PVs and SSLRs for Adjudicated Forced Majority Groups 1 and 3 (Diagnostic Sepsis and No Sepsis)not | Risk Group | Septic Patients (N) | Total Patients (N) | PV [95% CI] | SSLR [95% CI] | |------------|---------------------|--------------------|------------------------|---------------------| | Low | 5 | 230 | 2.17% [0.71%, 5%] | 0.23 [0.1, 0.56] | | Medium | 8 | 145 | 5.52% [2.41%, 10.58%] | 0.61 [0.31, 1.24] | | High | 29 | 204 | 14.22% [9.73%, 19.77%] | 1.74 [1.21, 2.52] | | Very High | 10 | 20 | 50% [27.2%, 72.8%] | 10.52 [4.42, 25.01] | Table 16. Primary Endpoint Predictive Analysis for ImmunoScore: PVs and SSLRs for Adjudicated Forced Majority Groups 2 and 3 (Predictive Sepsis and No Sepsis) | Sepsis Risk Category | Median Time to Discharge Event (Days) [95% CI] | |----------------------|------------------------------------------------| | Low (N = 232) | 4.00 [3.47, 4.86] | | Medium (N = 157) | 5.68 [4.89, 6.96] | | High (N = 276) | 7.66 [6.54, 8.53] | | Very High (N = 33) | 13.47 [7.12, 19.08] | Table 17. Secondary Endpoint Predictive Analysis: Median Time to Discharge Event by ImmunoScore Risk Category {20}------------------------------------------------ | Event | Risk
Category | Patients with
Event (N) | Total Patients
(N) | PV [95% CI] | SSLR [95% CI] | |-------------------------------------------------|------------------|----------------------------|-----------------------|-------------------------|--------------------| | In-Hospital
Mortality | Low | 0 | 232 | 0.00% [0.00%, 1.58%] | 0 [0, NaN] | | | Medium | 3 | 157 | 1.91% [0.40%, 5.48%] | 0.39 [0.13, 1.22] | | | High | 24 | 276 | 8.70% [5.65%, 12.66%] | 1.92 [1.29, 2.85] | | | Very High | 6 | 33 | 18.18% [6.98%, 35.46%] | 4.48 [1.87, 10.74] | | ICU Transfer
within 24
hours | Low | 5 | 226 | 2.21% [0.72%, 5.09%] | 0.17 [0.07, 0.4] | | | Medium | 16 | 153 | 10.46% [6.1%, 16.43%] | 0.87 [0.53, 1.42] | | | High | 43 | 248 | 17.34% [12.84%, 22.64%] | 1.55 [1.16, 2.09] | | | Very High | 14 | 29 | 48.28% [29.45%, 67.47%] | 6.92 [3.38, 14.13] | | Vasopressor
within 24
hours | Low | 0 | 230 | 0% [0%, 1.59%] | 0 [0, NA] | | | Medium | 1 | 155 | 0.65% [0.02%, 3.54%] | 0.17 [0.02, 1.2] | | | High | 18 | 262 | 6.87% [4.12%, 10.64%] | 1.91 [1.21, 3.02] | | | Very High | 6 | 26 | 23.08% [8.97%, 43.65%] | 7.78 [3.16, 19.15] | | Mechanical
Ventilation
Within 24
hours | Low | 2 | 228 | 0.88% [0.11%, 3.13%] | 0.39 [0.1, 1.57] | | | Medium | 2 | 153 | 1.31% [0.16%, 4.64%] | 0.59 [0.15, 2.35] | | | High | 10 | 268 | 3.73% [1.8%, 6.75%] | 1.72 [0.93, 3.18] | | | Very High | 1 | 31 | 3.23% [0.08%, 16.7%] | 1.48 [0.2, 10.78] | Table 18. Secondary Endpoint Predictive Analysis of ImmunoScore: PVs and SSLRs All the secondary endpoints were also met, with the exception of overlapping bands for the low and high risk categories. This is likely due to the low sample size of 15 subjects in this analysis cohort. All other endpoints were met. # Fresh versus Frozen Plasma Samples for CRP and PCT Testing Two of the non-imputable input parameters for the algorithm are C-Reactive Protein (CRP) and Procalcitonin (PCT) measurements. During clinical use of the device these tests could be ordered for a patient for input into the algorithm to calculate a ImmunoScore risk score. For the retrospective clinical validation study, patient data from the NOSIS database was used. This database includes timestamped patient specific parameters for the 20 input parameters that are routinely collected for patients suspected of infection, but typically CRP and PCT are not analyzed for all patients and therefore values for these lab inputs were acquired by testing frozen plasma samples. Testing was conducted to demonstrate the equivalence of frozen plasma samples to fresh plasma samples as well as using different assay methods (Roche cobas analyzers versus Lumiex assay) on the ImmunoScore output. The plasma samples were stored refrigerated 2-8°C for up to 8 days or stored frozen at -80°C for up to 27 months. Testing included: - CRP and PCT measured in fresh clinical plasma with clinical analyzers . {21}------------------------------------------------ - . CRP and PCT measured in thawed, previously frozen plasma samples measured with Luminex assays - used in the algorithm training data - . CRP and PCT measured in thawed, previously frozen plasma samples measured on Roche cobas analyzers - used in the clinical validation study Study reports for refrigerated stability studies, frozen stability studies, accuracy of the Luminex assay as compared to fresh clinical measurements, the accuracy of the Roche assay as compared to fresh clinical measurements, and a calculated normalization between the Roche and Luminex measurements were provided. In the clinical study discussed above, all PCT and CRP input parameters were taken from frozen plasma samples, even in the cases where a fresh sample may have been available. In routine practice, both PCT and CRP inputs will likely come from fresh samples. To understand the impact of the use of frozen versus fresh samples on the ImmunoScore risk score, an analysis was done where frozen sample measurements were replaced with either fresh PCT samples (n=106) only, fresh CRP samples only, or both fresh PCT and CRP (n=28) samples and the impact on the risk score was assessed. A high positive correlation was observed for all three groups (>0.99) indicting that the use of frozen samples in the clinical validation did not impact the final ImmonoScore output. There was a positive agreement of 95% with a 95% CI lower bound above 90% for both the fresh PCT only (0.97 [0.92. 0.99]) and fresh CRP only (1.00 [0.9. 1.00]) groups, but not for the CRP & PCT group (1.00 [0.88, 1.00]), despite perfect agreement. This is likely attributed to the limited sample size of n=28. Image /page/21/Figure/4 description: The image is a scatter plot titled "Fresh vs Frozen C-reactive Protein Sepsis ImmunoScore results". The x-axis is labeled "Frozen C-reactive Protein Sepsis Risk Score", and the y-axis is labeled "EMR C-reactive Protein Sepsis Risk Score". The data points are clustered tightly around a dashed diagonal line, indicating a strong positive correlation. The text "R = 1, p < 2.2e-16" is displayed, suggesting a perfect correlation with a very small p-value. Figure 9. Fresh vs. Frozen C-reactive Protein ImmunoScore Results {22}------------------------------------------------ Image /page/22/Figure/0 description: The image is a scatter plot titled "Fresh vs Frozen Procalcitonin Sepsis ImmunoScore results". The x-axis is labeled "Frozen Procalcitonin Sepsis Risk Score", and the y-axis is labeled "EMR Procalcitonin Sepsis Risk Score". The plot shows a strong positive correlation between the fresh and frozen procalcitonin sepsis risk scores, with R=1 and p<2.2e-16. Most of the data points are clustered around the diagonal line. Figure 10. Fresh vs. Frozen Procalcitonin ImmunoScore Results Image /page/22/Figure/2 description: This image is a scatter plot comparing fresh vs frozen CRP & PCT protein sepsis ImmunoScore results. The x-axis represents the frozen CRP & PCT sepsis risk score, while the y-axis represents the EMR CRP & PCT sepsis risk score. The plot shows a strong positive correlation between the two measures, with a correlation coefficient R=1 and p < 2.2e-16. The data points are clustered tightly around a diagonal line, indicating a high degree of agreement between the fresh and frozen samples. Fresh vs Frozen CRP & PCT Protein Sepsis ImmunoScore results Figure 11. Fresh vs. Frozen CRP & Procalcitonin ImmunoScore Results The following tables show the number of cases per risk category. There were three cases where risk category for a sample changed when the fresh PCT sample was tested versus when the frozen PCT sample was tested. Specifically, there were two cases that were in the high category when the Fresh PCT sample was tested, but were in the medium category when the frozen {23}------------------------------------------------ sample was testing. Also, there was one sample that was in the high category when the fresh PCT sample was tested, but was in the very high category when the frozen sample was tested. In all three cases, the difference in risk score value was very small (<0.4) and risk category reassignment likely occurred because the original risk score was near the threshold boundary between two risk categories. However, no patients were reassigned into a non-adjacent risk category. The data in the following tables supports that use of frozen samples in the clinical validation did not impact the ImmunoScore risk stratification output. | Frozen Risk
Category | Low | Medium | High | Very High | |-------------------------|-----|--------|------|-----------| | Low | 23 | - | - | - | | Medium | - | 35 | 2 | - | | High | - | - | 39 | - | | Very High | - | - | - | 6 | | Frozen Risk
Category | Low | Medium | High | Very High | | Low | 10 | - | - | - | | Medium | - | 19 | - | - | | High | - | - | 15 | - | | Very High | - | - | - | 1 | | Frozen Risk
Category | Low | Medium | High | Very High | | Low | 4 | - | - | - | | Medium | - | 13 | - | - | | High | - | - | 10 | - | | Very High | - | - | - | 1 | Table 19. Fresh vs. Frozen PRT, CRP, and CRP & PCT Change in Risk Score # Pediatric Extrapolation In this De Novo request, the 18+ intended use population was supported by clinical data on patients 18+ and the data were not leveraged to support the use of the device in any additional pediatric patient populations below 18 years of age. #### SUMMARY OF NONCLINICAL/BENCH STUDIES #### PERFORMANCE TESTING - BENCH # Precision/Sensitivity and Reproducibility Analysis - variability and error in parameter inputs Because the algorithm uses a variety of input parameters that are each subject to variability and error, an assessment was conducted to evaluate the impact of input parameter errors on the output (the ImmunoScore result). A comprehensive simulation study of input parameter error, including varying combinations of bias and imprecision, was conducted to estimate the imprecision of the risk score and provide sensitivity {24}------------------------------------------------ analysis of the effect of input parameter bias on device performance. Perturbed input parameters were simulated 1000 times each for each subject in the clinical validation cohort using national reference standards set forth in Clinical Laboratory Improvement Amendments of 1988 (CLIA) federal regulations and academic literature. Analysis was conducted by estimating standard deviations (SDs), interquartile ranges (IORs) and intraclass correlation (ICCs). The data analysis included the following: - 1. Sepsis Risk Score Imprecision: The imprecision of the Sepsis ImmunoScore Sepsis Risk Score was graphically assessed by depicting the interquartile range of the 1,000 Sepsis Risk Scores for each patient as a function of the patient's median Sepsis Risk Score. In addition, the standard deviation of the Sepsis Risk Score was estimated as a function of the mean Sepsis Risk Score grouped into discrete intervals. - 2. Sepsis Risk Score Reproducibility: The reproducibility of the Sepsis Risk Score in the face of input parameter error was estimated by computing an intraclass correlation coefficient (ICC). Specifically, the two-way random effects, absolute agreement, single rater/measurement ICC was estimated using the IRR package in R Statistical Software (Koo and Li, 2016). - 3. Impact of Input Parameter Bias on Device Performance: The impact of individual input parameter bias on the Sepsis Risk Score was assessed by estimating the ICC as a function of parameter bias for each of the 22 parameters. - 4. Diagnostic Accuracy: The robustness of the Sepsis Risk Score's diagnostic accuracy was assessed by computing its AUROC for predicting each of the adjudicated sepsis-3 labels for each simulation replicate. The 2.5th, 50th (median), and 97.5th quantiles across simulations were reported. - 5. Primary Endpoint Acceptance Criteria: The predictive value (PV) of each sepsis risk stratification category was estimated for both adjudicated sepsis-3 labels. The primary endpoint of non-overlapping, non-adjacent 95% confidence intervals was assessed using the 2.5th and 97.5th quantiles of the PVs across simulations. {25}------------------------------------------------ | Input Parameter | Total
Allowable Error | Source | Acceptable
Measurement
Range | |--------------------------------|--------------------------|----------------------------|------------------------------------| | Creatinine | 10% | CLIA, 2019 | [0, 10³] | | Sodium | 4 mmol/L | CLIA, 2019 | [0, 10⁴] | | Potassium | 0.3 mmol/L | CLIA, 2019 | [0, 10³] | | Total Carbon Dioxide | 20% | CLIA, 2019 | [0, 10³] | | Chloride | 5% | CLIA, 2019 | [0, 10³] | | Blood Urea Nitrogen | 2 mg/dL | CLIA, 2019 | [0, 10³] | | Albumin | 8% | CLIA, 2019 | [0, 10³] | | Bilirubin | 20% | CLIA, 2019 | [0, 400] | | Age | - | | [0, 110] | | Lactate | 15% | CLIA, 2019 | [0, 10³] | | Procalcitonin | 10% | Ceriottii et al., 2017 | [0, 10⁵] | | C-Reactive Protein | 30% | CLIA, 2019 | [0, 10³²] | | White Blood Cell Count | 5% | CLIA, 2019 | [0, 10³] | | Lymphocyte Count | 15% | CLIA, 2019 | [0, 10³] | | Platelet Count | 25% | CLIA, 2019 | [0, 10⁵] | | Neutrophil Count | 15% | CLIA, 2019 | [0, 10⁴] | | Temperature | 1° Celsius | Sund-Levander et al., 2004 | [11, 45] | | Heart Rate | 8.4 beats per minute | Hug et al., 2007 | [0, 300] | | Respiratory Rate | 4 breaths per minute | Drummond et al., 2020 | [0, 70] | | Blood Oxygen Saturation (SpO₂) | 4% | Nitzan et al., 2014 | [11, 100] | | Systolic Blood Pressure | 15.6 mm Hg | Hug et al., 2007 | [0, 250] | | Diastolic Blood Pressure | 7.8 mm Hg | Hug et al., 2007 | [30, 300] | Figure 12. List of Measurement Error for Each Input {26}------------------------------------------------ Image /page/26/Figure/0 description: This image is a scatter plot that shows the relationship between Sepsis ImmunoScore and Rank of Median Sepsis ImmunoScore. The x-axis represents Sepsis ImmunoScore, ranging from 0.00 to 1.00. The y-axis represents Rank of Median Sepsis ImmunoScore, ranging from 0 to 600. The plot shows a positive correlation between the two variables, with the Rank of Median Sepsis ImmunoScore increasing as the Sepsis ImmunoScore increases. Figure 13. The Effect of Perturbed Input Parameters on ImmunoScore Interquartile Range – The interquartile range of the 1,000 simulation replicate ImmunoScores for each patient are depicted as a function of the patient's median ImmunoScore. The dotted vertical black lines indicate the boundaries between the ImmunoScore Risk Stratification Categories. {27}------------------------------------------------ Image /page/27/Figure/0 description: The image is a figure titled "Feature Bias vs Interclass Correlation Coefficient". It contains 20 different plots in a grid, each plot showing the relationship between bias decile and intraclass correlation coefficient for a different feature. The features include Age, Albumin, Bilirubin Total, Blood Urea Nitrogen, C-Reactive Protein, Chloride, Creatinine, Diastolic BP, Heart Rate, Lactate, Lymphocyte, Neutrophil, Platelets, Potassium, Procalcitonin, Respiratory Rate, Sodium, SpO2, Systolic BP, Temperature, Total Carbon Dioxide, and White Blood Cell. The y-axis represents the Intraclass Correlation Coefficient (Two-way Agreement Random-Effects) and ranges from 0.966 to 0.978, while the x-axis represents the Bias Decile, ranging from (-0.5,-0.4] to (0.4,0.5]. Figure 14. ImmunoScore Intraclass Correlation as a Function of Input Parameter Bias {28}------------------------------------------------ Image /page/28/Figure/0 description: The image is a collection of scatter plots showing the positive bias impact on the sepsis risk score for various medical measurements. Each plot represents a different measurement, such as Albumin, Bilirubin Total, Blood Urea Nitrogen, and others. The x-axis represents the original sepsis risk score, while the y-axis represents the perturbed sepsis risk score with a bias of 50% TAE. Each plot also includes a linear equation that models the relationship between the original and perturbed scores, with equations such as y = 0.996x + -0.00334 for Albumin and y = 1.003x + 0.00208 for Bilirubin Total. Figure 15. Positive Bias Impact on ImmunoScore {29}------------------------------------------------ Image /page/29/Figure/0 description: This figure is titled "Negative Bias Impact on Sepsis Risk Score". It contains 18 scatter plots, each comparing the original sepsis risk score to the perturbed sepsis risk score (with a bias of -50% TAE) for different variables. Each plot includes a regression equation, such as "y = 1.003x + 0.00410" for Albumin and "y = 1.013x + 0.01052" for Systolic BP. The variables include Albumin, Bilirubin Total, Blood Urea Nitrogen, C-Reactive Protein, Chloride, Creatinine, Diastolic BP, Heart Rate, Lactate, Lymphocyte, Neutrophil, Platelets, Potassium, Procalcitonin, Respiratory Rate, Sodium, SpO2, Systolic BP, Temperature, Total Carbon Dioxide, and White Blood Cell. Figure 16. Negative Bias Impact on ImmunoScore {30}------------------------------------------------ For both sets of graphs the slope of the regression line for all input parameters is close to 1 and the intercept is close to 0. This indicates that even when the inputs are perturbed, the output score does not significantly change. This supports that the score is robust to perturbations. | Mean Sepsis Risk
Score Interval | Sepsis Risk Score Standard
Deviation: Median [IQR] | |------------------------------------|-------------------------------------------------------| | [0.025,0.075] | 0.01 [0.01, 0.02] | | (0.075,0.125] | 0.02 [0.02, 0.03] | | (0.125,0.175] | 0.04 [0… --- **Source:** [https://fda.innolitics.com/submissions/GU/subpart-g%E2%80%94general-hospital-and-personal-use-miscellaneous-devices/SAK/DEN230036](https://fda.innolitics.com/submissions/GU/subpart-g%E2%80%94general-hospital-and-personal-use-miscellaneous-devices/SAK/DEN230036) **Published by [Innolitics](https://innolitics.com)** — a medical-device software consultancy. We help companies design, build, and clear FDA-regulated software and AI/ML devices. If you're preparing [a De Novo](https://innolitics.com/services/regulatory/), [a 510(k)](https://innolitics.com/services/510ks/), [a SaMD](https://innolitics.com/services/end-to-end-samd/), [an AI/ML medical device](https://innolitics.com/services/medical-imaging-ai-development/), or [an FDA regulatory strategy](https://innolitics.com/services/regulatory/), [get in touch](https://innolitics.com/contact). **Cite:** Innolitics at https://innolitics.com