Artificial intelligence software for assessing brain ischemic penumbra/core infarction on computed tomography perfusion: A real-world accuracy study

Abstract

BACKGROUND

With the increasingly extensive application of artificial intelligence (AI) in medical systems, the accuracy of AI in medical diagnosis in the real world deserves attention and objective evaluation.

AIM

To investigate the accuracy of AI diagnostic software (Shukun) in assessing ischemic penumbra/core infarction in acute ischemic stroke patients due to large vessel occlusion.

METHODS

From November 2021 to March 2022, consecutive acute stroke patients with large vessel occlusion who underwent mechanical thrombectomy (MT) post-Shukun AI penumbra assessment were included. Computed tomography angiography (CTA) and perfusion exams were analyzed by AI, reviewed by senior neurointerventional experts. In the case of divergences among the three experts, discussions were held to reach a final conclusion. When the results of AI were inconsistent with the neurointerventional experts’ diagnosis, the diagnosis by AI was considered inaccurate.

RESULTS

A total of 22 patients were included in the study. The vascular recanalization rate was 90.9%, and 63.6% of patients had modified Rankin scale scores of 0-2 at the 3-month follow-up. The computed tomography (CT) perfusion diagnosis by Shukun (AI) was confirmed to be invalid in 3 patients (inaccuracy rate: 13.6%).

CONCLUSION

AI (Shukun) has limits in assessing ischemic penumbra. Integrating clinical and imaging data (CT, CTA, and even magnetic resonance imaging) is crucial for MT decision-making.

Key Words: Artificial intelligence; Acute ischemic stroke; Penumbra; Core infarction; Computed tomography perfusion

Core Tip: Shukun CerebralDoc system is a Chinese artificial intelligence (AI) post-processing software for computed tomography angiography (CTA) and cerebral perfusion. It is currently the most widely used vascular post-processing AI software in hospitals in mainland China. There is still a lack of relevant laws or regulations in China regarding the clinical application of AI. We found that Shukun AI computed tomography (CT) perfusion imaging has certain limitations in assessing the ischemic penumbra with an inaccuracy rate of 13.6%. We highlight that comprehensive evaluation of the ischemic penumbra in combination with the clinical symptoms, signs, and imaging findings (CT, CTA, and even magnetic resonance imaging), rather than totally relying on AI results, is a key imperative prior to decision-making for mechanical thrombectomy.

INTRODUCTION

Stroke is the second most common cause of death worldwide and the most common cause of death and disability in China[1]. Ischemic stroke accounts for approximately 85% of all cases of acute stroke, of which acute ischemic stroke (AIS) caused by large vessel occlusion (LVO) accounts for approximately 20% and is the leading cause of moderate to severe disability and death[2,3]. The use of alteplase for intravenous thrombolysis (IVT) is an effective treatment for AIS within the thrombolytic window[4]. However, for AIS with LVO (AIS-LVO), the recanalization rate is low, and the efficacy of IVT is unsatisfactory. A meta-analysis of five randomized trials on mechanical thrombectomy (MT) in 2016 indicated that for patients with anterior circulation AIS-LVO within 6 hours of onset, IVT combined with MT was associated with a higher immediate vascular recanalization rate and significantly reduced disability at 90 days compared with IVT, irrespective of the patient characteristics or geographical location[5]. In 2018, on the basis of the DAWN and DEFUSE3 studies, the therapeutic window for MT in patients with anterior circulation AIS-LVO was extended to 16-24 hours[6,7]. IVT combined with MT is currently the standard treatment for AIS-LVO[8,9]. Guidelines recommend computed tomography perfusion (CTP) imaging to evaluate the ischemic penumbra to select patients suitable for MT[8,9]. In the DEFUSE3 study, patients were screened based on perfusion images processed by RAPID artificial intelligence (AI)[7].

The RAPID system has indeed improved clinical accuracy and efficiency. However, many district hospitals, especially those in economically underdeveloped regions in China, are unable to afford the cost of purchasing RAPID software. Therefore, other similar and more affordable products can serve as alternative choices. Shukun (CerebralDoc, Beijing Shukun Technology Co., Ltd., Beijing, China), a Chinese AI processing software for computed tomography (CT) images of arteries and cerebral perfusion, was approved by the State Drug Administration in February 2021. In a comparative study, Shukun AI processing software was found to be superior to volume computed tomographic digital subtraction angiography (DSA) in displaying the carotid artery and extracranial segments of the vertebral artery[10]. In a study focused on head and neck arteries, the Shukun AI system achieved 100% accuracy in diagnosing aneurysms and arterial stenosis, as confirmed by DSA[11]. However, there are no reports on the reliability of Shukun AI in evaluating the ischemic penumbra to screen eligible patients for MT. Therefore, we performed a single-center study to investigate the accuracy of the Shukun software in evaluating the ischemic penumbra.

MATERIALS AND METHODS

Participants

This was a retrospective single-center clinical study conducted at Huizhou Central People’s Hospital. Consecutive AIS patients with anterior circulation LVO who underwent MT after assessment of the ischemic penumbra using Shukun AI software between November 2021 and March 2022 were included.

Inclusion criteria

Patients were required to meet the following conditions: Age ≥ 18 years; National Institute of Health Stroke Scale (NIHSS) score > 2; time from the last normal time to femoral artery puncture 6-24 hours; CTA findings indicative of occlusion of large vessels in the anterior circulation; core infarct volume [defined as cerebral blood flow (CBF) < 30%] < 70 mL; low perfusion volume [time to maximum (Tmax) > 6 seconds] > 15 mL; and low perfusion volume (Tmax > 6 seconds)/core infarct volume (CBF < 30%) > 1.8.

Exclusion criteria

According to the contraindications for intravascular diagnosis and treatment of acute large vessel occlusive ischemic stroke in the Chinese Guidelines for Stroke Prevention and Treatment (2021 version)[12], patients with the following conditions were excluded from the study: (1) A history of intracranial hemorrhage in the preceding 3 weeks, or the presence of cerebral artery malformations or aneurysms without interventional or surgical treatment; (2) Intractable hypertension not controlled by medication (systolic blood pressure > 185 mmHg or diastolic blood pressure > 110 mmHg); (3) Known allergy to iodized contrast media; (4) Blood glucose < 2.8 mmol/L or > 22.0 mmol/L; (5) Acute hemorrhagic constitution, including coagulation factor deficiency, an international normalized ratio > 3.0 or a platelet count < 40 × 10⁹/L; (6) A history of arterial puncture at an incompressible site within the last 7 days, a history of major surgery or severe trauma within the preceding 14 days, gastrointestinal or urethral bleeding within the preceding 21 days, a history of any disease in the preceding 3 months that increases the risk of bleeding (such as severe craniocerebral trauma, severe liver diseases, or ulcerative gastrointestinal diseases), and operations, biopsy of solid organs, or active bleeding within one month; (7) Suspected septic emboli or bacterial endocarditis; (8) Life expectancy < 90 days; and (9) Severe renal dysfunction.

Research protocol

One-stop multimode CT examination (256 slices Revolution CT Xtream, GE) was performed as soon as possible for these AIS patients. The images were postprocessed with Shukun (AI) software. Regions with a CBF < 30% represent the core infarct volume, and regions with a Tmax > 6 seconds represent a low perfusion volume. Iodofol (Jiangsu Hengrui Pharmaceutical Co. Ltd., China) was used as the contrast agent, with a total volume of 45 mL injected at a rate of 4.5 mL/second. Scanning was started 5 seconds after injection of the contrast agent, with a total scanning time of 60 seconds and a scanning thickness of 5 mm. The analysis output of AI (Shukun) was independently reviewed by three or more senior neurointerventional experts, and a comprehensive analysis based on the patient’s clinical manifestations, cranial CTA or magnetic resonance angiography was conducted to calculate the accuracy of AI (Shukun). In the cases of divergences among the three experts, discussions were held to reach a final conclusion. When the results of AI were inconsistent with the neurointerventional experts’ diagnosis, the diagnosis of AI was considered inaccurate.

Statistical analysis

All the data were analyzed and processed with SPSS 22.0 software. Continuous variables are expressed as the means ± SD, whereas categorical variables are expressed as frequencies (%).

RESULTS

A total of 22 patients (20 males, 2 females; mean age: 59 years) who met the inclusion criteria and underwent MT were enrolled. The mean NIHSS score was 10.7, and the vascular recanalization rate was 90.9%. At the 3-month follow-up, 63.6% of patients had a modified Rankin scale (mRS) score of 0-2. In 3 patients, the ischemic penumbra results assessed by the Shukun system were inconsistent with the signs, symptoms, or imaging findings. This corresponded to an inaccuracy rate of 13.6%.

Cases misdiagnosed by AI

Case 1: A 53-year-old man complained of speech disfluency and right limb weakness for 7 hours. Physical examination revealed mixed aphasia, right central facial and lingual paralysis, right limb muscle strength of Grade 1, right Babinski sign (+), and an NIHSS score of 17. One-stop multimode CT examination was performed immediately. The patient had some head movement during the examination. On plain CT scan, mild cortical sulcal effacement of the left cerebral hemisphere was found. AI did not suggest left cerebral hemisphere core infarction (Figure 1A). Preoperative CT perfusion analyzed by AI indicated that the volume of the core infarcted area of the right cerebral hemisphere was 4.0 mL (Figure 1A). CTA revealed occlusion of the M1 segment of the left middle cerebral artery (Figure 1B). After the intervention, the left middle cerebral artery was completely recanalized. On Day 14 after the operation, cranial diffusion-weighted imaging (DWI) revealed that the left cerebral hemisphere had a core infarction of 30 mL (Figure 1C). AI resulted in an interpretative error of the core infarction in this patient.

Figure 1 Imaging findings regarding cerebral perfusion, core infarction, and cerebral arteries of case 1. A: Shukun brain perfusion computed tomography image processing software suggested that the core infarction volume was 4 mL located in the right cerebral hemisphere (pink area), and the low perfusion volume of 246.1 mL mainly in the left hemisphere (blue area); B: Digital subtraction angiography revealed M1 segment occlusion of the left middle cerebral artery (arrow); C: Diffusion-weighted imaging revealed infarction in the left cerebral hemisphere 14 days after mechanical thrombectomy (triangle).

Case 2: A 93-year-old man who had a history of atrial fibrillation but not anticoagulant therapy was admitted to the hospital due to “being found unconscious for 4 hours”, with 11 hours elapsed since he was last seen well. On physical examination, the patient was lethargic, with a Glasgow coma scale (GCS) score of 9, mixed aphasia, right limb muscle strength of Grade 0, right Babinski sign (+), and an NIHSS score of 19. CTP revealed that the volume of the core infarct area was 5.7 mL, the mismatch volume was negative, the low perfusion volume was 4.2 mL, and the mismatch ratio was 0.7 (Figure 2A). CTA revealed occlusion of the M1 segment of the left middle cerebral artery (Figure 2B). The patient had impaired consciousness, with an NIHSS score of 19, and disappearance of the sulci and gyri of the left cerebral hemisphere on CT. All of these findings were suggestive of moderate to severe stroke. Considering that AI made an interpretative error, MT was performed immediately. The left middle cerebral artery was completely recanalized, and the forward blood flow recovered to MT in cerebral infarction (mTICI) level 3. The low-density volume of the left cerebral hemisphere was approximately 50 mL on day 7 after the operation (Figure 2C). AI severely underestimated the core infarct volume and low perfusion volume of this patient and even misjudged that the volume of the low perfusion area was smaller than the core infarct volume.

Figure 2 Imaging findings regarding cerebral perfusion, core infarction, and cerebral arteries of case 2. A: Shukun brain perfusion computed tomography (CT) image processing software suggested that the core infarction volume was 5.7 mL (pink area), the low perfusion volume was 4.2 mL (blue area), and the mismatch volume was negative; B: Digital subtraction angiography of the head and neck indicated M1 segment occlusion of the left middle cerebral artery (arrow); C: Cranial CT image showing a large area of cerebral infarction in the left hemisphere 7 days after mechanical thrombectomy (triangle).

Case 3: A 64-year-old man was transferred from a local hospital to our hospital due to left limb weakness and unclear speech for 12 hours. Physical examination revealed lethargy, a GCS score of 14, gaze to the right, a flattened nasolabial fold on the left, Grade 1 muscle strength of the left limb, and Babinski sign (+) on the left side. The patient’s NIHSS score was 13. CT revealed a low-density focus in the right basal ganglia. AI revealed that the core infarction volume was 30.4 mL, which was located mainly in the left cerebral hemisphere. The volume of the hypoperfusion area was 119.3 mL, and the mismatch ratio was 3.9 (Figure 3A). CTA revealed occlusion of both the initial segment of the right internal carotid artery and the M1 segment of the right middle cerebral artery (Figure 3B). The right internal carotid artery and right middle cerebral artery were completely recanalized after emergency MT, and the forward blood flow recovered to mTICI grade 3. A cranial CT scan performed 5 days after the operation revealed that the infarcted area was located in the right basal ganglia and that there was no obvious infarction in the left cerebral hemisphere (Figure 3C). AI misjudged the location of the core infarction in this patient.

Figure 3 Imaging findings regarding cerebral perfusion, core infarction, and cerebral arteries of case 3. A: Computed tomography (CT) image of brain perfusion (Shukun) suggested that the core infarction volume was 30.4 mL mainly in the left cerebral hemisphere (pink area), but a low perfusion volume of 119.3 mL in the right hemisphere (blue area); B: Head and neck CT angiography indicated right internal carotid artery occlusion and right middle cerebral artery occlusion (arrow); C: Brain CT image showing right basal ganglia infarction 5 days after mechanical thrombectomy (triangle).

DISCUSSION

The RAPID software developed by Stanford University in 2010 defines the infarct core area with an apparent diffusion coefficient threshold of 600 × 10^-6 mm²/second, and a PWI Tmax > 6 second defines hypoperfusion. The software automatically applies these criteria to assess DWI/PWI mismatch in patients with AIS within 5 to 7 minutes, with 100% sensitivity and 91% specificity[13]. However, there are several disadvantages of selecting patients based on MR mismatch, as follows: (1) In acute stroke patients, the MRI scan time is 10-20 minutes longer than that with CT, which is completed in 10-15 minutes; (2) Most stroke centers do not have a 24 hours × 7 days facility for MRI; and (3) Metal implants (such as pacemakers) and claustrophobia are contraindications to MRI examination.

In contrast, CT perfusion image processing software can quickly and accurately assess LVO, the vascular path, collateral compensation, and the ischemic penumbra, and it has become the main imaging modality for the preoperative evaluation of LVO before MT. In our cohort, the proportion of patients with post-MT mTICI Grade 2b or 3 anterior circulation LVO who had undergone Shukun CT perfusion imaging was 90.9%, which is greater than that reported in the Defuse3 trial (69%)[7]. The percentage of patients with favorable functional outcomes, defined as an mRS score of 0-2 at 90 days, was 63.6%, which was also higher than that reported in the Defuse3 trial (44%)[7]. This discrepancy may be attributable to the continuous update of thrombectomy equipment at our center in recent years, the use of the Penumbra thrombus suction system for patients with cardiogenic cerebral embolism, greater institutional experience, and the strict screening and selection of patients via multimodality imaging. Shukun (China) automated CT perfusion imaging software is safe and effective in selecting patients who require MT. Furthermore, it is inexpensive and easy to popularize.

Although the diagnosis of core infarction and penumbra made by AI (Shukun) is mostly reliable, it still has a certain interpretative error rate. The interpretative error rate in our single-center study was 13.6%, indicating the presence of certain limitations and pitfalls of AI. The diagnosis of core infarction and ischemic penumbra by AI depends on the setting of the parameters. Not surprisingly, the use of different AI parameters to define core infarcts results in different core infarcts and penumbras, sometimes leading to false ischemic penumbras[14]. The baseline core infarct area suggested by AI is not necessarily the final infarct size. The earlier the recanalization, the smaller is the final infarct size. With late or no recanalization, the final infarct size will be larger than the baseline core infarct area assessed by AI. In the Shukun cerebral perfusion CT image processing software, the area with a CBF < 30% was defined as the core infarction, and a Tmax > 6 seconds was defined as the ischemic area. Some studies have suggested that for patients who have undergone thrombectomy within 90 minutes of onset, areas with a CBF < 20% are much closer to the final infarct size after surgery[15]. Even if optimized with an increasing interpretative rate of AI in the future, clinical physicians and radiologists should use AI as a medical device rather than completely relying on the data provided by AI[16].

The CTP scanning time can also affect the judgment of the AI system pertaining to the volume of the ischemic penumbra. AI software takes into account the entire process from the inflow of contrast agent from arteries to outflow through veins to obtain the arterial inflow and venous outflow curves for accurately calculating the blood volume in the ischemic area. For patients with LVO, the time for contrast agents to flow into the arteries is relatively delayed; thus, the already established inappropriate CTP scanning time is liable to have a certain impact on the lesion volume. If the scanning time is too short, the volume of the ischemic penumbra cannot be accurately estimated, whereas if the scanning time is too long, the patient will be exposed to unnecessary additional radiation. Currently, the recommended optimal scanning time for CTP is 60-70 seconds[17]. The scanning time in the present study was 60 seconds. However, the scanning time may be shortened by the operator if a patient is irritable or does not cooperate with the examination. Additionally, a low contrast medium concentration or low injection rate is also likely to cause errors in the CT perfusion results.

Finally, the CT perfusion results generated by AI can also be affected by artifacts. The essence of perfusion imaging is enhanced CT examination. Since CT perfusion examination is a multiphase examination, any situation that causes CT artifacts can cause inaccurate CT perfusion results. Stroke patients with anterior circulation LVO often have dysphoria and cannot cooperate with the CT perfusion examination in a quiet state. Therefore, the appropriate application of sedatives is recommended to reduce the occurrence of artifacts in restless patients.

CONCLUSION

Although CTP imaging of the brain is a clinically useful tool in multimodality imaging of acute stroke, there are limitations and pitfalls of AI. Clinical physicians and radiologists should not rely completely on the data provided by AI. In addition to the CTP results provided by AI, a comprehensive analysis of the clinical picture, plain CT findings, CTA, and even MRI should be performed before deciding whether to perform MT. In addition, it is critical to pay attention to factors that may lead to AI inaccuracies, such as head movement during the CT examination and the speed of contrast agent injection.

ACKNOWLEDGEMENTS

We are grateful to all the participants in this study.