Interobserver reliability between pediatric radiologists and residents in ultrasound evaluation of intraventricular hemorrhage in premature infants

Abstract

BACKGROUND

Germinal matrix intraventricular hemorrhage (IVH) may contribute to significant morbidity and mortality in premature infants. Timely identification and grading of IVH affect decision-making and clinical outcomes. There is possibility of misinterpretation of the ultrasound appearances, and the interobserver variability has not been investigated between radiology resident and board-certified radiologist.

AIM

To assess interobserver reliability between senior radiology residents performing bedside cranial ultrasound during on-call hours and pediatric radiologists.

METHODS

From June 2018 to June 2020, neonatal cranial ultrasound examinations were performed in neonatal intensive care unit. Ultrasound findings were recorded by the residents performing the ultrasound and the pediatric attending radiologists.

RESULTS

In total, 200 neonates were included in the study, with a mean gestational age of 30.9 wk. Interobserver agreement for higher grade (Grade III & IV) IVH was excellent. There was substantial agreement for lower grade (Grade I & II) IVH.

CONCLUSION

There is strong agreement between radiology residents and pediatric radiologists, which is higher for high grade IVHs.

Key Words: Ultrasound head; Neonatal cranial ultrasound; Cranial ultrasound; Intraventricular hemorrhage; Neonatal intraventricular hemorrhage

Core Tip: While possibility of interobserver variability exists in all imaging modalities, it is the highest in ultrasound. Interobserver variability in ultrasound may result from technical errors such as inadequate gain/depth settings, incomplete anatomic interrogation, or error in misinterpretation. During ultrasound examination, both the image acquisition and interpretive skills improve with increasing experience. Differences in identification and grading of intraventricular hemorrhage may affect the clinical outcome, and the subsequent management options.

INTRODUCTION

Intraventricular hemorrhage (IVH) is a major neurological complication of prematurity. In neonates weighing less than 1500 g, the incidence of IVH reaches up to 27% whereas, in extremely preterm infants weighing 500-750 g, the prevalence is about 45%[1]. A substantial subgroup of premature infants with moderate to severe IVH develops neurologic sequelae including an elevated risk of post-hemorrhagic hydrocephalus, cerebral palsy, and mental retardation, while infants with mild IVH are at risk of developmental disabilities[2-4]. IVH and its neurologic and psychiatric sequelae are a major public health concern worldwide[5].

The multifaceted etiology of IVH is primarily attributed to the intrinsic fragility of the germinal matrix vasculature and the disturbance in the cerebral blood flow. The germinal matrix exhibits rapid angiogenesis in contrast to other brain regions causing its high vascular density. Hemorrhages occurring in the germinal matrix often rupture through the ependyma into the lateral ventricle and are then referred to as IVH[6].

The development of IVH is attributable to a number of risk factors including vaginal delivery, low Apgar score, severe respiratory distress syndrome, pneumothorax, hypoxia, hypercapnia, seizures, patent ductus arteriosus, thrombocytopenia, infection, and others[7-9]. Dysregulation of cerebral blood flow by these risk factors induces IVH.

While the possibility of interobserver variability exists in all imaging modalities, it is the highest in ultrasound. Interobserver variability in ultrasound may result from technical errors such as inadequate gain/depth settings, incomplete anatomic interrogation, or misinterpretation errors[10]. During ultrasound examination, both the image acquisition and interpretive skills improve with increasing experience.

Few studies have examined the reliability of cranial ultrasound interpretation, despite the ostensibly important role of accurate interpretation. Variations in the identification and grading of IVH may affect the morbidity, clinical outcomes, and subsequent treatment options. This study aims to assess interobserver reliability between senior residents performing bedside cranial ultrasounds during on-call hours and board-certified pediatric radiologists.

MATERIALS AND METHODS

This cross-sectional study was carried out in the Department of Radiology at Aga Khan University Hospital, Karachi, Pakistan. The Institutional Ethical Review Committee approved the study with a waiver for informed consent. The study period was two years, from June 2016 to June 2018. All premature infants (less than 37 wk of gestational age) or infants with very low birth weight (birth weight equal to or less than 1500 g) and infants in the Neonatal Intensive Care Unit who underwent a cranial ultrasound were included in the study. Patients who were born at term, and had prior brain computed tomography (CT) or brain magnetic resonance imaging (MRI) , or patients with known cerebral malformations were excluded. To prevent potential selection bias, patients with prior neuroimaging were excluded.

Prematurity was defined as infants born alive before 37 wk of gestation with further subcategorization as: (1) Extremely preterm (less than 28 wk); (2) very preterm (28 - 32 wk); and (3) moderate to late preterm (32 - 37 wk).

The gestational age of all infants was determined from a chart review of the mother. The weight of all infants included in the study was measured with a digital weighing scale.

Cranial ultrasound in all cases was performed through the anterior fontanelle in both coronal and sagittal planes using the Mindray M7 Diagnostic Ultrasound System with a 5- to 10-MHz transducer. All ultrasound examinations were performed by a senior resident (Year III and year IV) and reviewed by an attending board-certified radiologist with at least five years’ experience in pediatric imaging. The findings of both the resident and the pediatric radiologist with regards to the presence and absence of intraventricular hemorrhage and its grading were recorded on a structured proforma by a year III resident, blinded to additional clinical information. In addition to IVH, all scans were recorded for the presence or absence of hydrocephalus, periventricular leukomalacia, and brain malformations. If hydrocephalus was noted to be present, it was graded as mild, moderate, or severe based on the measurement of transverse atrial width.

The Volpe grading system was used for sonographic grading of IVH/germinal matrix hemorrhage[11]. (1) Grade I: Bleeding confined to the periventricular area (germinal matrix). An example of grade I IVH is shown in Figure 1, in which abnormal echogenicity is apparent in caudothalamic groove on parasagittal view; (2) Grade II: Intraventricular bleeding (10%-50% of the ventricular area on sagittal view). An example of grade II IVH is shown in Figure 2, in which abnormal echogenicity is seen extending into left lateral ventricle on coronal view; (3) Grade III: Intraventricular bleeding (> 50% of the ventricular area or distends ventricle. An example of grade III IVH is shown in Figure 3, in which hemorrhage in right lateral ventricle is seen on coronal view, with mild associated ventricular dilatation; and (4) Grade IV: Germinal matrix hemorrhage grade I, II or III with extension into brain parenchyma. An example of grade IV IVH is seen in Figure 4, in which the hemorrhage in bilateral lateral ventricles is seen extending into periventricular region bilaterally on coronal view.

Figure 1 Grade I germinal matrix hemorrhage. Parasagittal and cornal views of the right lateral ventricle, in which abnormal echogenicity in caudothalamic groove was consistent with grade I intraventricular hemorrhage/germinal matrix hemorrhage as indicated by arrows.

Figure 2 Grade II intraventricular hemorrhage. Coronal view ultrasound image reveals abnormal echogenicity in the left caudothalamic groove extending into left lateral ventricle. There was no associated ventricular dilatation (white arrow).

Figure 3 Grade III intraventricular hemorrhage. Coronal view ultrasound image reveals right-sided intraventricular hemorrhage with associated ventricular dilatation (white arrow).

Figure 4 Grade IV bilateral intraventricular hemorrhage. Coronal view ultrasound image reveals bilateral intraventricular hemorrhage filling the dilated bilateral lateral ventricles, and extending into adjacent parenchyma (white arrows).

Interobserver agreement was calculated using Kappa statistics (Table 1). Data was entered and analyzed using Statistical Package for Social Sciences version 20 software.

Table 1 Interobserver agreement with Kappa values in cranial ultrasound interpretation between radiology residents and attending pediatric radiologists.

Findings	Senior resident, %	Attending radiologist, %	Kappa value
Normal ultrasound	68	74	0.94
Abnormal ultrasound	32	26	0.94
Grade I IVH	72	62	0.82
Grade II IVH	12	12	0.86
Grade III IVH	6	7	0.90
Grade IV IVH	10	19	0.92
Right-sided IVH	38	27	0.92
Left-sided IVH	20	21
Bilateral IVH	42	52

IVH: Intraventricular hemorrhage.

RESULTS

The study included 200 neonates with a gestational age of 30.9 wk (range 20-36, SD ± 3.8). There were 120 (60%) male neonates and 80 (40%) female neonates. A total of 78 (39%) babies were delivered vaginally while 122 (61%) were delivered via lower segment cesarean section. The mean weight was 1.2 kg (range 0.5-1.5, SD ± 0.3). Based on the clinical indication on the radiology slip, 83 (41%) of the neonates had sepsis, 60 (30%) had respiratory distress, and 5 (2.5%) had a pneumothorax. The mean duration of hospital stay was 9 days (range 1-46).

The radiology resident reported 136 (68%) cases as normal and 64 (32%) as abnormal. The pediatric radiologist reported 148 (74%) cases as normal and 52 (26%) as abnormal. Twenty-four patients had IVH on the right side, 13 patients had IVH on the left side, and 27 had bilateral IVH according to resident interpretations. Fourteen patients had IVH on the right side, 11 patients had IVH on the left side, and 27 patients had bilateral IVH according to the pediatric radiologist. The presence of IVH and its grading by the resident and attending along with the Kappa values are shown in Table 1. We did not measure the interobserver agreement on additional findings encountered in the study.

DISCUSSION

On making the diagnosis of rheumatic fever by auscultation, Alvan Feinstein wrote in his book Clinical Judgment “The main problems of observer variability were neither in the eyes nor the ears of the observers. We all saw and heard essentially the same things, but each observer used different ingredients in his criteria for description and interpretation of the observations”[12]. The same can be said about medical imaging in which interobserver variability remains a critical issue[13].

Bedside cranial ultrasound is the neuroimaging standard of care for the detection of IVH[14]. Cranial sonography is cost-effective, does not require sedation, and is portable, allowing for the evaluation of critical patients at the bedside. In a study by Maalouf et al[15], ultrasound has a predictive probability of 0.85 (0.76-0.94) for the presence of IVH on MRI.

The interobserver agreement for findings on neonatal head ultrasound varies from poor to excellent among radiologists[16]. Although there is a possibility of intra-observer agreement, this is quite low. While numerous studies have explored interobserver variability in neonatal cranial ultrasonography, ours is the first to study the differences in interpretation between senior residents and board-certified pediatric radiologists.

In our study, there was excellent agreement between the senior resident and the attending for intracranial hemorrhage. There was substantial agreement on grade I and grade II intraventricular hemorrhage, whilst agreement on grade III and grade IV intraventricular hemorrhage was almost perfect.

Among radiologists, experienced neonatologists, and less experienced neonatologists involved in a study by Hagmann et al[17], the interobserver agreement in the interpretation of cranial ultrasound ranged from poor to good. Hintz et al[18] found excellent interobserver agreement on severe intraventricular hemorrhage, but poor agreement on periventricular leukomalacia between experienced board-certified radiologists with special expertise in cranial ultrasound. Among radiologists, pediatric neurologists, and neonatologists experienced in neonatal ultrasounds, Pinto et al[19] obtained excellent interobserver agreement for major findings such as parenchymal hemorrhage, but rather poor agreement for less severe pathologies such as germinal matrix hemorrhage. A study by Corbett et al[16] found excellent agreement on high-grade hemorrhage but poor agreement on interpretation of ventricular size.

Even though we did not find any prior studies on interobserver variability between senior residents and radiologists, our results are comparable to previous studies in that there is excellent agreement on major abnormalities such as grade III and IVH.

The board-certified radiologist's experience is most likely to be responsible for the study's relatively low interobserver agreements for grade I and II IVH. Because of greater image gain/depth, improved probe handling, and knowledge of pertinent anatomy, experience with doing cranial ultrasonography can result in better imaging quality.

This experience also manifests itself in improved ultrasound interpretation performed by others. Due to inexperience, the resident in our study initially missed the additional findings of hydrocephalus, choroid plexus abnormalities, and periventricular echogenicity as indicated in Table 2. We plan to conduct a follow-up study to investigate abnormalities other than IVH on neonatal cranial ultrasound which can have significant impact on disease prognosis.

Table 2 Additional ultrasound examination findings initially missed by the resident.

Additional findings on neonatal cranial ultrasound (%)
Ventricular abnormalities	12 (6)
Mild hydrocephalus	11 (5.5)
Severe hydrocephalus	1 (0.5)
Choroid plexus abnormalities	4 (2)
Increased periventricular echogenicity	4 (2)

This study has some limitations. The residents and pediatric radiologists were compared for interobserver agreement only on one variable, i.e., IVH, but no cross-sectional neuroimaging such as CT or MRI was performed for confirmation. The interobserver agreement was not calculated for additional findings such as parenchymal hemorrhage, hydrocephalus, and venous infarctions, which also have implications for neonatal neurodevelopment.

CONCLUSION

Interobserver agreement regarding detection of intraventricular hemorrhage is high for low-grade hemorrhage and almost perfect for high-grade hemorrhage between residents and board-certified pediatric radiologists.

ARTICLE HIGHLIGHTS

Research background

Neonatal cranial ultrasound examinations were evaluated in neonatal intensive care unit (NICU) patients. Ultrasound findings were recorded for the resident performing the ultrasound and the pediatric attending radiologist.

Research motivation

Despite the ostensibly important role of accurate cranial ultrasound interpretation, few studies have investigated the reliability of interpretation of cranial ultrasound. Differences in the identification and grading of intraventricular hemorrhage (IVH) may affect the clinical outcome and the subsequent management options. This is the reason the study was undertaken.

Research objectives

To assess interobserver reliability between senior radiology residents performing bedside cranial ultrasounds during on-call hours and board-certified pediatric radiologists.

Research methods

A total of 200 neonatal cranial ultrasound examinations were evaluated in NICU patients. Ultrasound findings were recorded for both the resident performing the ultrasound and the pediatric attending radiologist. Interobserver agreement was calculated.

Research results

The mean gestational age was 30.9 wk. Interobserver agreement for higher grade (Grade III & IV) IVH was excellent. There was substantial agreement for lower grade (Grade I & II) IVH.

Research conclusions

Interobserver agreement for detection of IVH is high for low-grade hemorrhage and almost perfect for high-grade hemorrhage between radiology residents and board certified pediatricians.

Research perspectives

Our study results are limited by the cross sectional nature of the study. Additionally, we did not compare agreement on the interpretation of periventricular leukomalacia, incidental findings, and degree of ventriculomegaly if it was present which can have significant impact on disease prognosis. This may be explored in a future study.