Predicting outcomes using neural networks in the intensive care unit

Abstract

Patients in intensive care units (ICUs) require rapid critical decision making. Modern ICUs are data rich, where information streams from diverse sources. Machine learning (ML) and neural networks (NN) can leverage the rich data for prognostication and clinical care. They can handle complex nonlinear relationships in medical data and have advantages over traditional predictive methods. A number of models are used: (1) Feedforward networks; and (2) Recurrent NN and convolutional NN to predict key outcomes such as mortality, length of stay in the ICU and the likelihood of complications. Current NN models exist in silos; their integration into clinical workflow requires greater transparency on data that are analyzed. Most models that are accurate enough for use in clinical care operate as ‘black-boxes’ in which the logic behind their decision making is opaque. Advances have occurred to see through the opacity and peer into the processing of the black-box. In the near future ML is positioned to help in clinical decision making far beyond what is currently possible. Transparency is the first step toward validation which is followed by clinical trust and adoption. In summary, NNs have the transformative ability to enhance predictive accuracy and improve patient management in ICUs. The concept should soon be turning into reality.

Key Words: Large language models; Hallucinations; Supervised learning; Unsupervised learning; Convoluted neural networks; Black-box; Workflow

Core Tip: Healthcare workers in intensive care units undertake swift and critical decisions, based on physiological and clinical data recorded in digital form, leading to information overload. Neural network models and machine learning can analyse the dense information and can potentially aid in decision making by patient triage, preventing treatment errors and providing insights into possible outcomes. Practical, legal and ethical issues need to be addressed as with other areas of healthcare. But research and its quick translation strongly suggests its imminent incorporation into routine clinical workflow.

INTRODUCTION

In the intensive care unit (ICU) prioritizing the patients and starting appropriate treatment could mean the difference between life and death. Large amounts of data on patients’ clinical condition are streamed; a heavy patient load makes it difficult for the clinician to take a considered judgement based on many rapidly changing data. This leads to patients receiving delayed treatment and worse outcomes. Therefore the physician must be sensitive to the time between the arrival of the critically ill patient and initiation of treatment[1]. Neural network (NN) and machine learning (ML) have been employed to aid the clinician undertaking these decisions. Although Alan Turing, the father of artificial intelligence (AI) proposed the skeleton of ML[2] to enable computers to learn from analyzing existing data, it took several decades for advances in computational power and creation of databases to enable practical implementation of ML in clinical care[3].

NN AND STATISTICAL METHODS

Earlier, outcomes in ICU were predicted by the use of statistical methods in the form of scoring systems. How are NN related to conventional statistical techniques (Figure 1)? There is considerable overlap because they are both interlinked[4]; only their applications vary. NNs align with discriminant analysis and regression; traditional methods are supplanted by NNs in areas of prediction and classification, though there are certain conceptual differences as well as similarities[4]. NNs have the advantage of being able to automatically approximate nonlinear mathematical function, which is particularly useful when the relation between the variables is complex or unknown. In general NN models equal or even outperform other methods[4]. Dreiseitl and Ohno-Machado[5] provide a technical review of the similarities in pattern recognition between NN (k-nearest neighbors, decision trees and support vector machines) and statistical pattern recognition. NNs were developed as ‘generalizations of mathematical models of human cognition through biological neurons’[6].

Figure 1  Concept of neural networks.

ML

Broadly, ML is an automated process to discern or learn patterns of data to classify and predict[7]. It comes under the subfield of AI. Weidener and Fischer[8] referred to the 1955 proposal for introducing the concept of AI to McCarthy et al[9] as ‘the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it’. It was put forward by an interdisciplinary group of scientists who went on to become household names in the field of AI and ML: (1) Shannon CE was a mathematician at the Bell Telephone Laboratories; (2) Minsky ML was a Harvard Junior Fellow in Mathematics and Neurology; (3) Rochester N was the Manager of Information Research at the IBM Corporation; and (4) McCarthy J was Assistant Professor of Mathematics at Darmouth College[9].

Classification of ML

Broadly, ML can be classified into supervised learning techniques, unsupervised learning and reinforcement learning[7].

Supervised learning refers to the use of data with known outcomes to classify or predict outcomes from new data[10].

In unsupervised learning, the model works on its own to identify patterns and information in unlabeled data[11]. It is possible to uncover unanticipated connections among different features in a dataset.

Reinforced learning (RL), allows machines to learn through trial and error. This method has been gaining traction. RL agents map the optimal paths to action, which obtains the highest reward. Though they may not affect the current reward, they can affect the subsequent rewards. In short, RL tries imitate human learning[12].

A number of other ML algorithms were developed, viz decision tree, support vector machine, k-nearest neighbor, random forest, regularized regression, naïve Bayes and convoluted NN. These are classified according to their complexity. A review of these methods was published in 2023[13]. The different models are not used in isolation. For optimal outcome, a number of them are combined, e.g., convoluted NN for image processing [e.g. computed tomography (CT) images of the lungs], k-nearest neighbour for clustering of physiological variables (such as blood pressure and oxygen saturation). This integrated model was was useful to predict the outcome of coronavirus disease 2019 (COVID-19) infection admissions.

In all these methods, four characteristics are necessary for efficient data mining processing: (1) High quality data; (2) Accurate data; (3) Adequate sample size; and (4) The right tool[14].

Training of datasets in AI

The critical component in the development of NN and AI models is the data on which they are trained. Biases tend to creep in at this stage; it is challenging to get data that sufficiently represent all patients. Efforts must be made to include under-represented groups to improve performance and generalizability of AI models. Failure to do so can propagate societal biases, which results in misdiagnosing certain patient groups, which are underrepresented in datasets, thereby amplifying inequalities. Prevention of bias can be addressed by: (1) Participant-centered development of AI algorithms with a focus on representative participants; (2) Sharing data and incorporating data standards that support interoperability; and (3) Sharing code, including that of AI algorithms that allows synthesis of underrepresented data. Lack of such care in the training set reinforces bias, which can lead to misdiagnoses, lack of generalization and even death.

The following procedures address the issue of potential bias in training datasets.

Recognition of bias sources: The sources of bias are explicitly identified, such as socioeconomic and biological differences, disparities in resource access, and biases in data collection (e.g., missing or skewed data). Also highlighted algorithmic biases stemming from errors in design, overadjustment, or evaluation processes.

Class imbalance handling: To mitigate class imbalance, methods like skewness-based transformations and balanced random forest algorithms are selected. These techniques ensure that minority classes (e.g., rare conditions or less-represented demographic groups) are adequately represented during training.

Benchmarking and validation: Benchmarking frameworks are used to validate models against diverse datasets. This includes external validation in heterogeneous clinical settings to assess generalizability and reduce the risk of overfitting to biased training data.

Feature selection and preprocessing: Preprocessing steps include careful feature engineering and normalization to avoid introducing biases inherent in raw data. Features are selected with domain knowledge to minimize irrelevant or biased influences.

Transparent model development: The need for transparency in model development is stressed, including sharing performance metrics like calibration and discrimination to identify and address bias.

Call for broader and representative datasets: It is essential to collect more diverse and representative datasets to reduce systemic biases and improve the fairness of models.

Incorporation of feedback loops: Iterative refinement using feedback from clinical use cases helps identify and correct biases that manifest during real-world applications.

CONCEPT OF AI IN ICU

Before AI, there was human intelligence—one of the remarkable outcomes of evolution. AI models are built to represent complexity for predicting outcomes[15]. Advances in AI and ML dealing with a large number of variables can exceed human performance in some areas of medicine. Complex non-linear relationships between independent and dependent variables are best detected by NNs[16].

Attractive as the potential use of AI and NN in ICUs appears to be, there are certain principles to adhere to, viz benchmarking. These are dependent, but not circumscribed by real world factors (e.g. socioeconomic and biological differences, access to resources), application (e.g. prejudice against the application, reinforcement of existing defects and unfair application), digital data (bias in measurement and recording, missing data), bias in preparation of variables and biases in AI algorithm design (e.g. errors in design, over and under adjustment and evaluation bias)[15]. Published studies on benchmarking in critical care outcomes were reviewed by Atallah et al[17] in 2023. Advances are required in feature types, model selection, preprocessing and validation. Further research must address class imbalance, generalizability, improved calibration, fairness and long-term validation[17].

In the ICU, traditional outcome quality indicators are mortality, complications, length of stay, readmission rate to ICU, ventilator outcomes and patient-reported outcomes. Other outcomes include medication adherence, social support or mobility before admission into ICU.

Currently, raw data that are used for model development are obtained by hand-crafting demographics, input diagnoses, labs and vital signs. Specific models employing skewness-based transformations[18] and balanced random forest algorithms[19] can correct class imbalance[17].

Table 1 shows the models employed in the studies conducted; Table 2 is a more comprehensive list of models that are available.

Table 1 Neural network models discussed in the manuscript.

Model/scoring system	Primary use case	Strengths	Limitations
Convolutional neural networks	Image-based tasks (e.g., computed tomography scans and X-rays)	High accuracy in spatial feature extraction	Computationally expensive
Recurrent neural networks	Time-series predictions (e.g., sepsis progression)	Captures temporal dependencies effectively	Potentially high computational cost
Multilayer perceptron	Nonlinear relationship modeling (e.g., ICU mortality)	Flexible, integrates with hybrid systems	Prone to overfitting if not regularized
Balanced random forests	Handling imbalanced datasets	Interpretable, robust to class imbalance	Requires careful tuning of hyperparameters
Sequential Organ Failure Assessment	Assessing organ failure severity	Widely validated, clinically interpretable	Limited to scoring; no predictive modeling
Acute Physiology and Chronic Health Evaluation	Evaluating ICU patient mortality risk	Comprehensive, includes chronic health factors	Limited in real-time adaptability

ICU: Intensive care unit.

Table 2 List of currently available neural network models.

Model	Variations	Use cases	Strengths	Weaknesses
Multilayer perceptron	N/A	Classification and regression tasks	Simple architecture and good for baseline models	Not ideal for spatial or sequential data. Can overfit with high dimensional data
Convolutional neural networks	AlexNet	Image recognition	Captures spatial hierarchies	Computationally intensive
	VGGNet	Object detection	Effective for image processing	Requires large datasets
	ResNet	Complex computer vision tasks	Residual learning avoids vanishing gradient	Requires higher computation
	Inception	Image recornition with lower computations	Efficient use of resources	Architecture complexity
	MobileNet	Mobile and embedded vision applications	Lightweight and efficient	Trade-off in accuracy for efficiency
RNN	LSTM	Language modeling	Handles sequential data	Vanishing gradient problem
	Gated recurrent unit	Time series forecasting	Simplified version of LSTM	Less powerful for complex tasks
	Bidirectional RNN	Speech recognition	Considers past and future context	Computationally expensive
GAN	DCGAN	Image generation	Generates high quality data	Training instability
	CycleGAN	Unsupervised image-to-image translation	Advances data augmentation	Mode collapse issues
	StyleGAN	Synthetic image creation for design tasks	Generates photorealistic images	Computationally expensive
Autoencoders	Variational autoencoders	Dimensionality reduction, generative tasks	Effective for feature extraction	Blurry reconstructions
	Denoising autoencoders	Anomaly detection and noise reduction	Robust against noisy inputs	Limited generative capability
Transformers	Bidirectional Encoder Representations from Transformers	Contextual embeddings for natural language processing tasks	Captures long-range dependencies	High computational requirements
	General Purpose Transformers series	Generative tasks (e.g. text generation)	Powerful generative abilities	Requires vast amounts of training data
	T5	Text summarization, translation	Task-agnostic and flexible	Computationally intensive
Graph neural networks	Graph convolutional networks	Social network analysis, biological modeling	Handles graph-structured data	Scalability issues
	Graph attention networks	Recommendation systems	Captures relational information	Complex architecture
	GraphSAGE	Molecular modeling, protein interactions	Effective for inductive learning	Requires large-scale graph sampling
Self organizing maps	N/A	Data visualization	Intuitive mapping and visualization	Less effective for high-dimensional data
Boltzmann machines	Restricted boltzmann machines	Collaborative filtering, dimensionality reduction	Probabilistic feature learning	Difficult to train
	Deep belief networks	Feature learning and pretraining	Effective for unsupervised learning	Computationally expensive
Deep reinforcement learning models	Deep Q networks	Game playing (e.g., AlphaGo)	Learns optimal policies	Sample inefficiency
	Proximal policy optimization	Robotics, autonomous navigation	Handles high-dimensional inputs	Requires hyperparameter tuning
	Actor critic methods	Autonomous systems	Balances policy and value learning	May require extensive exploration

GAN: Generative adversarial networks; LSTM: Long short-term memory; RNN: Recurrent neural networks; N/A: Not applicable.

In addition to employing balanced random forest algorithms to address class imbalance, here and in earlier investigations, additional measures such as the ones listed below to ensure the outlier cases were addressed systematically, enhancing the model’s reliability and applicability in the ICU setting.

Data augmentation

Synthetic data generation techniques, such as Synthetic Minority Over-sampling Technique and its variants, were used to create synthetic samples for under-represented rare cases. This helped balance the dataset while preserving the characteristics of the minority class.

Cross-validation

A rigorous cross-validation strategy ensured the model's generalizability and minimized overfitting. Stratified k-fold cross-validation was specifically chosen to maintain the distribution of classes across training and validation sets.

Feature selection and engineering

Careful feature selection avoided overfitting by reducing noise and irrelevant features. Additionally, domain-specific feature engineering enhanced the representation of rare cases without artificially inflating their significance.

Regularization techniques

Models were configured with regularization techniques such as L1 and L2 penalties to discourage overly complex models that might overfit to the majority class.

Evaluation metrics

Performance was monitored using a range of metrics beyond accuracy, such as precision, recall, F1-score, and area under the receiver operating characteristic and precision-recall curves. This ensured the model’s effectiveness in identifying rare cases.

Ensemble methods

In addition to the balanced random forest algorithm, ensemble techniques like AdaBoost and Gradient Boosting were evaluated to combine the strengths of multiple models and reduce bias toward the majority class.

Threshold tuning

Decision thresholds were carefully adjusted post-training to optimize sensitivity and specificity, particularly for rare cases. This was guided by clinical priorities and outcome-specific requirements.

Validation on external datasets

The model was validated on external datasets, where available, to confirm its robustness and effectiveness in generalizing to unseen data.

COMPARISON OF CONVENTIONAL METHODS VS AI IN PREDICTING ICU SURVIVAL

Two reports were published in 2022 comparing conventional methods and AI in predicting survival in the ICU. Mirzakhani et al[20] used a retrospective study of data from patients admitted in ICU (n = 840). Data from medical records were obtained about conventional severity classification (Acute Physiology and Chronic Health Evaluation (APACHE) II and APACHE IV, Sequential Organ Failure Assessment (SOFA) score and Simplified Acute Physiology Score (SAPS II). These scores were developed using statistical methods, with their inherent assumptions and limitations. They are meant to predict patient survival in ICU based on the severity of the illness, as assessed by the severity of physiological instability, and the severity of vital organ dysfunction[20]. SOFA model assesses the function of respiratory, hepatic, renal, cardiovascular, coagulation and nervous systems. SAPS II is based on 12 physiological variables, age, type of admission and three more related to underlying diseases. APACHE II employs physiological variables along with age and chronic diseases in patients admitted to ICU. APACHE IV evolved from a reformulation of the earlier equations.

The NN model consisted of the multilayer perceptron NN and classification and regression tree[20]. Variables were chosen using the univariate logistic regression; those showing statistically significant relation with the outcome (‘hospital mortality’) as the dependent variable were entered as the selected variable in the AI model. By dividing the sample to training (70%) and test (30%) set, AI models [multilayer Perceptron (MLP) NNs and highly active antiretroviral therapy, Distress Thermometers] were developed. The best model was selected based on the performance. To develop the architecture of NN, a feedforward network with a back propagation learning method with two connected hidden connected layers was used. Both conventional scores and NN models were equally good in predicting the ICU outcome, but MLP NN models outperformed others in external validation[20]. This was attributed to the greater efficiency of NN to develop nonlinear models compared to logistic regression. Further work must be done by carrying out prospective multi-centric studies in more than one centre to unravel the black-box approach of MLP NN that is currently used[20].

Barboi et al[21] from Indianapolis and Chicago (2022) performed a literature review and meta-analysis of articles which compared binary models of classification using ML with severity of disease scores for predicting mortality in ICU. They determined which model showed superior performance so that clinicians receive guidance on their performance and validity. A systematic search was carried out on publications between 2000 and 2020. Among 461 abstracts that were screened, full text was assessed in 66 (14.3%) articles. The review included 20 (4.3%) studies. They concluded that ML based models can predict ICU mortality and serve as an alternative to traditional scoring methods. Although the range of performance of ML models was superior, there was much heterogeneity, which did not allow generalization of the results. This needs externally validated models that are tested in clinical practice and updated to the patient population and the practice environment[21].

The following criteria were suggested for model developers: (1) Statement of purpose, i.e, whether they are intended for clinical practice; (2) If so, full transparency must be provided including clinical setting, steps of model development and external validation of models to allow generalizability; and (3) Metrices of models performance must be shared, including measures of calibration, discrimination and classification[21].

APPLICATION OF AI MODELS IN ICU

Predictive models using NN were used in a number of scenarios both in critical care units and in surgical procedures.

COVID-19 infection

COVID-19 infection swept through the globe suddenly and without warning, catching the healthcare world unawares. The need for quick decision making in managing the patients was never more urgent. Predictive models were developed to aid in the process. Beginning with elementary NN models based on a small sample size[22], more sophisticated prototypes were developed. Staging of patients based on imaging of the lung and clinical and biochemical parameters used AI to improve assessment of the disease outcome. An automatic method for disease quantification was obtained from computerized tomographic images of solid organs (lung, breast and heart) and integrated with known clinical and biochemical markers. CNN model was combined with multi-omic signature of COVID-specific parameters; the outcome was short term and long term prognosis[23]. This identified patients with severe disease, so that healthcare resources were optimally utilized.

The selection of ML models is based on their appropriateness for the particular characteristics of each clinical problem, and a tricky balancing act targeting predictive accuracy, generalizability, and interpretability. The areas considered for this selection include.

Data type and structure: (1) CNNs were preferred for COVID-19-related problems as they lent themselves well to image-based data, such as CT scans of the lungs, and are more efficient at extracting spatial features; and (2) The adopted RNNs modelled various problems on time-series data, including but not limited to patient recovery trajectory monitoring and sepsis progression prediction, because of their capability of temporal dependency capturing.

Complexity of the problem: (1) Most problems where there was a complex incorporation of nonlinear relationships, such as in the case of predicting mortality in ICUs, were best addressed by multilayer perceptron networks or hybrids that combined aspects of neural and decision-tree approaches; and (2) Simple tasks, where simplicity and interpretability are needed, such as preliminary feature selection or class imbalance correction, were done using Decision Trees and Random Forests.

Outcome specificity and target variables: The model types were chosen based on a particular clinical outcome. For instance, integrating CNNs with multi-omics data on COVID-19 prognosis was essential; similarly, integration of imaging with molecular data was also important.

Generalizability needs: For scenarios where adaptation is crucial across different ICU settings, such as arrhythmia detection or in the reduction of alarm fatigue, models are developed based on architectures such as CNNs and their hybrids.

Performance benchmarks: Selection was performed basing the choice on various comparative metrics, including accuracy, sensitivity, and specificity. The models selected indicated better predictive capabilities in the selected clinical context.

Interpretability requirements: Tasks requiring actionable insights, such as sepsis detection, were modelled using interpretability mechanisms ranging from attention-based RNNs to transparent hybrid methods.

A comparative assessment of artificial NN with clinical scoring assessed the risk of patients with COVID-19 getting admitted to ICU. In a prospective multi-center study 296 subjects with COVID-19 pneumonia were enrolled, and were split into general ward care group (n = 238) and ICU-admission group (n = 58). The NN model had similar predictive ability compared to the traditional scoring system (Patient Satisfaction Index), but needed fewer input variables. However another complex NN model predicted outcomes with higher accuracy, which can be used as a better prediction model[24]. But it is unlikely to find widespread clinical application because of the ‘black box’ approach of the model.

Cardiac arrhythmia prediction in ICU

It is critical to detect cardiac arrhythmias in the ICU for proper timely care. Traditional monitors give a high rate of false alarm due to physical displacement of sensors, resulting in alarm fatigue in healthcare workers. NN models were used to improve the detection of life-threatening arrhythmias[25]. Attempts were made to lower the rate by the use of electrocardiograph (ECG) signs alone or together with arterial blood pressure signal employing wavelet transform, data mining and ML approaches. These were effective when the type of alarm was known, but failed with unknown arrhythmias. The authors therefore used a hybrid-CNN method that combined conventional features obtained by physicians with features learned from CNN. The advantage included using the best of both methods. In all 953 independent alarms were annotated from 410 critical care subjects. The hybrid system outperformed either only CNN or only feature-based methods; additionally, this can be adapted to multiple modules and is flexible to work on different duration signals.

To ensure the generalizability of hybrid CNN models for arrhythmia detection in ICU settings, the article highlights several strategies.

Incorporation of diverse features: The hybrid model combines conventional features extracted by clinicians with features learned from the CNN. This leverages human expertise with data-driven insights, allowing the model to adapt to scenarios, including unknown arrhythmias.

Use of flexible architectures: It employs a modular hybrid structure that can integrate multiple types of input signals (e.g., ECG and arterial blood pressure). This flexibility allows the model to adapt to varying ICU setups and alarm types, accommodating diverse clinical contexts.

Extensive and diverse training data: To prepare for unknown arrhythmias, the model was trained on a comprehensive dataset that includes a variety of arrhythmia types and scenarios. This improves its ability to recognize new patterns not explicitly encountered during training.

Post-hoc model interpretation: Interpretability tools can be applied to understand the decision-making process when unknown arrhythmias are detected. This aids clinicians in evaluating the model's outputs and making informed adjustments.

Validation in heterogeneous settings: The hybrid CNN model is validated across multiple ICU environments and patient cohorts to assess its robustness and reliability. This ensures that it can perform consistently under different operational and patient conditions.

Continuous learning and updates: Incorporating mechanisms for ongoing learning allows the model to update itself with new data, ensuring it evolves to handle emergent arrhythmias over time.

Together, they collectively enhance the model's ability to maintain high performance while being adaptable to unknown or novel arrhythmias in various ICU settings.

Sepsis

Sepsis is a serious condition, leading to high risk of death, because of difficulty in diagnosing and delay in treatment. Antibiotics must be initiated early, for which it must be identified early. ML method was used to differentiate patients with different trajectories to sepsis.

Patients with sepsis have altered pharmacokinetics resulting in unpredictable responses to administered antibiotics; in addition they are also likely to be resistant to antibiotics. These can be approached by therapeutic drug monitoring to ensure that antibiotic concentrations remain at target exposures throughout treatment[26]. Widespread application of therapeutic drug monitoring is hampered by limited availability, complexities in operations as well as costs. Therefore physicians rely on clinical judgement to decide on the patient groups likely to derive the greatest benefit from their employment.

Large-dimensional and heterogeneous data are difficult to process. ML helps navigate through these complex situations. Information from analysis of the multidimensional temporal data aids in monitoring recovery trajectory and responses to treatment. This is achieved by measuring, assessing and adjusting drug levels to achieve the desired result and to avoid serious side effects. Parameters can be refined to obtain better predictive ability[26].

INTERPRETING THE ‘BLACK BOX’

Unless the logic behind the processing in the black box is known, they cannot be used in clinical practice.

A confluence of features extracted by CNN classifier on baseline CT images was employed. This along with laboratory and clinical data was fed into the model; a multidimensional scoring system was able to give clinical decision support to healthcare workers[27]. The following variables were used in building the system: (1) Sex; (2) Age; (3) Body mass index; (4) Comorbidities; (5) Vital signs at admission; (6) Arterial blood gas analysis; (7) Complete blood count; and (8) Any additional laboratory results. It was interpretable at two levels: (1) The global; and (2) The single patient level. An understanding of the logic that goes behind the decision process is required before it can be routinely introduced into clinical workflow.

Bio-statistical methods can give scores to quantify the likelihood of adverse outcomes and assess the effectiveness of treatment, deep learning (DL) is more capable in object recognition, which is useful to detect patterns in patient data and predict outcomes. This has limitated interpretability, leading to a trade off between predictive accuracy and interpretability. A multi-scale deep convolutional architecture was developed to improve the predictability in ICU using more ‘transparent’ methods of analysis. This was a visually interpretable method to predict mortality in the ICU, named ISeeU[28]. Employing input variables such as type of admission, chronic disease, Glasgow Coma Scale, PO₂ and diastolic blood pressure it performed well. Performance between training and validation set was close, showing that it has good generalization properties without serious overfitting[28].

ISeeU model is used as a visually interpretable method to predict mortality in ICU. It balances interpretability and complexity by employing a multi-scale deep convolutional architecture. This enhances transparency while maintaining accuracy. Calibration and discrimination matrices, along with performance comparison between training and validation datasets, ensure generalizability and reduce overfitting. These benchmarks assess the model's interpretability without compromising predictive strength.

Ho et al[29] presented a method to understand the model’s decision making process by assessing which input features were responsible for predictions in a recurrent NN model with the use of electronic medical data. They employed a Learned Binary Masks to identify inputs contributing to the predictions of a many-to-many RNN model. The lean body mass (LBM) and Mernel SHAP methods were considered complementary, not competing because evaluations depend on clinical insights and experience, which are often not quantifiable. This proof of concept study provided information on which input features gave the most significant contributions to assessing the risk of mortality predictions of a known RNN model using electromagnetic radiation of children who were critically ill[29].

More recent studies improved the interpretability of the ‘black box’ analysis of predictive models. Strickler et al[30] described a global interpretation mechanism for DL networks to predict sepsis by understanding the human-interpretability of the algorithms in sepsis detection. In principle, a balance must be struck between model complexity and accuracy. While there is an inverse relation between model complexity and human-interpretability, a trade-off is necessary. The authors proposed a post-hoc, model-agnostic interpretable mechanism to comprehensively understand sepsis-related concepts during training by a black-box ML method[30]. The authors propose to extend their work by utilizing other datasets and exploring supervised fine-tuning vs a model trained from scratch.

Both LBM and SHAP were valuable tools for interpreting RNN predictions, with LBM being more effective for capturing temporal dependencies critical to ICU events and SHAP providing a broader feature-importance perspective. These complementary strengths underline their potential for joint application in future studies.

Comparison framework: Both LBM and SHAP were evaluated on their ability to identify key features contributing to predictions of critical ICU events (e.g., sepsis onset, acute respiratory distress).

The evaluation focused on their alignment with clinical intuition and their ability to highlight actionable insights for clinicians.

Effectiveness in capturing temporal dependencies: (1) LBM: By masking irrelevant features at specific time points, LBM excelled in emphasizing temporal patterns and dynamic changes in patient data. This was particularly useful in identifying sequences of events leading up to critical conditions, such as a gradual decline in oxygen saturation before an acute event; and (2) SHAP: SHAP provided a more global perspective on feature importance across the entire dataset but was less precise in capturing time-specific dependencies compared to LBM.

Clinical interpretability: (1) LBM: Clinicians reported that LBM outputs were more intuitive for understanding time-series patterns, as the masks highlighted critical windows of observation. This facilitated real-time decision-making during ICU monitoring; and (2) SHAP: While SHAP offered detailed insights into feature contributions, the static nature of the explanations sometimes made it challenging to interpret the progression of critical events over time.

Quantitative metrics: Both methods were evaluated for their predictive accuracy improvement when used as part of the interpretation pipeline. Models with LBM interpretations demonstrated a slightly higher alignment with expert-annotated critical event sequences (85% vs 82% for SHAP) in benchmark tests. Computational efficiency was also considered. LBM required less processing time for time-series data, whereas SHAP was computationally intensive due to its reliance on generating numerous perturbed samples.

Complementary use: While LBM and SHAP were individually effective in specific aspects, combining their outputs provided a more holistic understanding. LBM highlighted critical temporal windows, while SHAP contextualized the contribution of specific features within those windows.

In 2024, Zilker et al[31] proposed a more accurate predictive tool with interpretable and actionable understanding. The authors introduced an ML framework termed PatWay-Net to make interpretable predictions at admission into ICU with features of sepsis. A novel type of recurrent NN was combined with multi-layer perceptrons for processing patient pathways and giving interpretable predictive results. In addition, the end user is provided a comprehensive dashboard to visualize the patients’ health trajectories. It was proposed as a valuable addition to health care decision making[31]. While explainable ML uses flexible ML models with high predictive ability, which need subsequent post-hoc explanation methods to convert complex math functions to easier to comprehend explanations, interpretable ML is different: Here an intrinsically interpretable model is developed so that it affords a better understanding of how predictions are made. Generalized Additive Models are more advanced intrinsically interpretable ML models. Here, input features are independently modeled in a non-linear manner to generate univariate shape functions which remain fully interpretable[31].

The advantages of PatWay-Net model is its ability to support analysis of patient pathways using patients data, thus avoiding subjectivity. It has a high performance while allowing flexible decision support applications in complex healthcare environments. It can simultaneously improve decision making both at the individual level and for administrative decisions[31].

CONCLUSION

Despite the many potential applications of NN in the ICU to ease workflow and improve outcomes, there are caveats to consider. First, AI systems are a complex and challenging multiphase process[32]. Second, AI based applications are considered to be medical devices and thereby subject to less rigorous reviewing and authorization criteria. Combined with dearth of raw data and financial constraints, decisions on whether or not to use these tools are often based on cost savings, not so much in improved patient outcomes. Finally, once a device is released into the market, no further fine-tuning to assess its performance is mandated. In addition to this is the manner in which healthcare workers respond to the alert is another variable, particularly with the possibility of ‘alert fatigue’[32]. Biases in the training set are limited in how widely the model or device is incorporated into clinical practice, all related to legal aspects and ethics[33]. In spite of the limitations, NN methods are commonly being used in the ICUs[34]. As the glitches are ironed out, they promise to aid the clinician in improving the quality of care, and to ease the burden of handling multidimensional data. Apart from the implementation of NN in the ICU setting, the concept of neural interfaces in the promises to usher in a revolutionary era that extends beyond traditional healthcare[35,36]. They in turn require, apart from technological advances, a robust ethical framework, multi-disciplinary collaboration, and regulatory oversight[35].