Observational Study Open Access
Copyright ©The Author(s) 2022. Published by Baishideng Publishing Group Inc. All rights reserved.
World J Radiol. Nov 28, 2022; 14(11): 367-374
Published online Nov 28, 2022. doi: 10.4329/wjr.v14.i11.367
Interobserver reliability between pediatric radiologists and residents in ultrasound evaluation of intraventricular hemorrhage in premature infants
Kiran Hilal, Noman Khan, Naila Nadeem, Faheemullah Khan, Zara Za Sheer, Ayimen Khalid, Muhammad Danish Barakzai
Muhammad Danish Barakzai, Department of Radiology, The Hospital for Sick Children, Toronto 555, Canada
Muhammad Danish Barakzai, Ayimen Khalid, Faheemullah Khan, Naila Nadeem, Noman Khan, Kiran Hilal, Department of Radiology, Aga Khan University Hospital, Karachi 74800, Pakistan
Zara Za Sheer, Community Health Sciences, Aga Khan University Hospital, Karachi 74800, Pakistan
ORCID number: Muhammad Danish Barakzai (0000-0001-8867-2124); Faheemullah Khan (0000-0001-6369-743X); Noman Khan (0000-0002-8343-6445); Kiran Hilal (0000-0003-0737-3419).
Author contributions: Barakzai MD, Khalid A, Sheer ZZ, Nadeem N and Hilal K designed the research study; Barakzai MD, Nadeem N, Khan N and Khan F performed the research; Khalid A, Sheer ZZ, Khan N and Hilal K contributed new reagents and analytic tools; Khalid A, Sheer ZZ, Khan F and Khan N analyzed the data and wrote the manuscript; All authors have read and approve the final manuscript.
Institutional review board statement: Exemption from ethical approval was granted by Ethics Review Committee.
Informed consent statement: Individual informed consent was waived off by the Institutional Review Committee due to minimal risk to the subjects, observational nature of the study and anonymized imaging review.
Conflict-of-interest statement: All authors of this manuscript have no conflict of interest to disclose.
Data sharing statement: Technical appendix, statistical code, and dataset available from the corresponding author at kiran.hilal@aku.edu. Participants gave informed consent for anonymized data sharing.
STROBE statement: The authors have read the STROBE Statement—checklist of items, and the manuscript was prepared and revised according to the STROBE Statement—checklist of items.
Open-Access: This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: https://creativecommons.org/Licenses/by-nc/4.0/
Corresponding author: Kiran Hilal, FCPS, MBBS, Assistant Professor, Department of Radiology, Aga Khan University Hospital, Stadium Road, Karachi 74800, Pakistan.kiran.hilal@aku.edu
Received: March 13, 2022
Peer-review started: March 13, 2022
First decision: May 12, 2022
Revised: May 25, 2022
Accepted: September 21, 2022
Article in press: September 21, 2022
Published online: November 28, 2022

Abstract
BACKGROUND

Germinal matrix intraventricular hemorrhage (IVH) may contribute to significant morbidity and mortality in premature infants. Timely identification and grading of IVH affect decision-making and clinical outcomes. There is possibility of misinterpretation of the ultrasound appearances, and the interobserver variability has not been investigated between radiology resident and board-certified radiologist.

AIM

To assess interobserver reliability between senior radiology residents performing bedside cranial ultrasound during on-call hours and pediatric radiologists.

METHODS

From June 2018 to June 2020, neonatal cranial ultrasound examinations were performed in neonatal intensive care unit. Ultrasound findings were recorded by the residents performing the ultrasound and the pediatric attending radiologists.

RESULTS

In total, 200 neonates were included in the study, with a mean gestational age of 30.9 wk. Interobserver agreement for higher grade (Grade III & IV) IVH was excellent. There was substantial agreement for lower grade (Grade I & II) IVH.

CONCLUSION

There is strong agreement between radiology residents and pediatric radiologists, which is higher for high grade IVHs.

Key Words: Ultrasound head, Neonatal cranial ultrasound, Cranial ultrasound, Intraventricular hemorrhage, Neonatal intraventricular hemorrhage

Core Tip: While possibility of interobserver variability exists in all imaging modalities, it is the highest in ultrasound. Interobserver variability in ultrasound may result from technical errors such as inadequate gain/depth settings, incomplete anatomic interrogation, or error in misinterpretation. During ultrasound examination, both the image acquisition and interpretive skills improve with increasing experience. Differences in identification and grading of intraventricular hemorrhage may affect the clinical outcome, and the subsequent management options.



INTRODUCTION

Intraventricular hemorrhage (IVH) is a major neurological complication of prematurity. In neonates weighing less than 1500 g, the incidence of IVH reaches up to 27% whereas, in extremely preterm infants weighing 500-750 g, the prevalence is about 45%[1]. A substantial subgroup of premature infants with moderate to severe IVH develops neurologic sequelae including an elevated risk of post-hemorrhagic hydrocephalus, cerebral palsy, and mental retardation, while infants with mild IVH are at risk of developmental disabilities[2-4]. IVH and its neurologic and psychiatric sequelae are a major public health concern worldwide[5].

The multifaceted etiology of IVH is primarily attributed to the intrinsic fragility of the germinal matrix vasculature and the disturbance in the cerebral blood flow. The germinal matrix exhibits rapid angiogenesis in contrast to other brain regions causing its high vascular density. Hemorrhages occurring in the germinal matrix often rupture through the ependyma into the lateral ventricle and are then referred to as IVH[6].

The development of IVH is attributable to a number of risk factors including vaginal delivery, low Apgar score, severe respiratory distress syndrome, pneumothorax, hypoxia, hypercapnia, seizures, patent ductus arteriosus, thrombocytopenia, infection, and others[7-9]. Dysregulation of cerebral blood flow by these risk factors induces IVH.

While the possibility of interobserver variability exists in all imaging modalities, it is the highest in ultrasound. Interobserver variability in ultrasound may result from technical errors such as inadequate gain/depth settings, incomplete anatomic interrogation, or misinterpretation errors[10]. During ultrasound examination, both the image acquisition and interpretive skills improve with increasing experience.

Few studies have examined the reliability of cranial ultrasound interpretation, despite the ostensibly important role of accurate interpretation. Variations in the identification and grading of IVH may affect the morbidity, clinical outcomes, and subsequent treatment options. This study aims to assess interobserver reliability between senior residents performing bedside cranial ultrasounds during on-call hours and board-certified pediatric radiologists.

MATERIALS AND METHODS

This cross-sectional study was carried out in the Department of Radiology at Aga Khan University Hospital, Karachi, Pakistan. The Institutional Ethical Review Committee approved the study with a waiver for informed consent. The study period was two years, from June 2016 to June 2018. All premature infants (less than 37 wk of gestational age) or infants with very low birth weight (birth weight equal to or less than 1500 g) and infants in the Neonatal Intensive Care Unit who underwent a cranial ultrasound were included in the study. Patients who were born at term, and had prior brain computed tomography (CT) or brain magnetic resonance imaging (MRI) , or patients with known cerebral malformations were excluded. To prevent potential selection bias, patients with prior neuroimaging were excluded.

Prematurity was defined as infants born alive before 37 wk of gestation with further subcategorization as: (1) Extremely preterm (less than 28 wk); (2) very preterm (28 - 32 wk); and (3) moderate to late preterm (32 - 37 wk).

The gestational age of all infants was determined from a chart review of the mother. The weight of all infants included in the study was measured with a digital weighing scale.

Cranial ultrasound in all cases was performed through the anterior fontanelle in both coronal and sagittal planes using the Mindray M7 Diagnostic Ultrasound System with a 5- to 10-MHz transducer. All ultrasound examinations were performed by a senior resident (Year III and year IV) and reviewed by an attending board-certified radiologist with at least five years’ experience in pediatric imaging. The findings of both the resident and the pediatric radiologist with regards to the presence and absence of intraventricular hemorrhage and its grading were recorded on a structured proforma by a year III resident, blinded to additional clinical information. In addition to IVH, all scans were recorded for the presence or absence of hydrocephalus, periventricular leukomalacia, and brain malformations. If hydrocephalus was noted to be present, it was graded as mild, moderate, or severe based on the measurement of transverse atrial width.

The Volpe grading system was used for sonographic grading of IVH/germinal matrix hemorrhage[11]. (1) Grade I: Bleeding confined to the periventricular area (germinal matrix). An example of grade I IVH is shown in Figure 1, in which abnormal echogenicity is apparent in caudothalamic groove on parasagittal view; (2) Grade II: Intraventricular bleeding (10%-50% of the ventricular area on sagittal view). An example of grade II IVH is shown in Figure 2, in which abnormal echogenicity is seen extending into left lateral ventricle on coronal view; (3) Grade III: Intraventricular bleeding (> 50% of the ventricular area or distends ventricle. An example of grade III IVH is shown in Figure 3, in which hemorrhage in right lateral ventricle is seen on coronal view, with mild associated ventricular dilatation; and (4) Grade IV: Germinal matrix hemorrhage grade I, II or III with extension into brain parenchyma. An example of grade IV IVH is seen in Figure 4, in which the hemorrhage in bilateral lateral ventricles is seen extending into periventricular region bilaterally on coronal view.

Figure 1
Figure 1 Grade I germinal matrix hemorrhage. Parasagittal and cornal views of the right lateral ventricle, in which abnormal echogenicity in caudothalamic groove was consistent with grade I intraventricular hemorrhage/germinal matrix hemorrhage as indicated by arrows.
Figure 2
Figure 2 Grade II intraventricular hemorrhage. Coronal view ultrasound image reveals abnormal echogenicity in the left caudothalamic groove extending into left lateral ventricle. There was no associated ventricular dilatation (white arrow).
Figure 3
Figure 3 Grade III intraventricular hemorrhage. Coronal view ultrasound image reveals right-sided intraventricular hemorrhage with associated ventricular dilatation (white arrow).
Figure 4
Figure 4 Grade IV bilateral intraventricular hemorrhage. Coronal view ultrasound image reveals bilateral intraventricular hemorrhage filling the dilated bilateral lateral ventricles, and extending into adjacent parenchyma (white arrows).

Interobserver agreement was calculated using Kappa statistics (Table 1). Data was entered and analyzed using Statistical Package for Social Sciences version 20 software.

Table 1 Interobserver agreement with Kappa values in cranial ultrasound interpretation between radiology residents and attending pediatric radiologists.
Findings
Senior resident, %
Attending radiologist, %
Kappa value
Normal ultrasound68740.94
Abnormal ultrasound32260.94
Grade I IVH72620.82
Grade II IVH12120.86
Grade III IVH670.90
Grade IV IVH10190.92
Right-sided IVH38270.92
Left-sided IVH2021
Bilateral IVH4252
RESULTS

The study included 200 neonates with a gestational age of 30.9 wk (range 20-36, SD ± 3.8). There were 120 (60%) male neonates and 80 (40%) female neonates. A total of 78 (39%) babies were delivered vaginally while 122 (61%) were delivered via lower segment cesarean section. The mean weight was 1.2 kg (range 0.5-1.5, SD ± 0.3). Based on the clinical indication on the radiology slip, 83 (41%) of the neonates had sepsis, 60 (30%) had respiratory distress, and 5 (2.5%) had a pneumothorax. The mean duration of hospital stay was 9 days (range 1-46).

The radiology resident reported 136 (68%) cases as normal and 64 (32%) as abnormal. The pediatric radiologist reported 148 (74%) cases as normal and 52 (26%) as abnormal. Twenty-four patients had IVH on the right side, 13 patients had IVH on the left side, and 27 had bilateral IVH according to resident interpretations. Fourteen patients had IVH on the right side, 11 patients had IVH on the left side, and 27 patients had bilateral IVH according to the pediatric radiologist. The presence of IVH and its grading by the resident and attending along with the Kappa values are shown in Table 1. We did not measure the interobserver agreement on additional findings encountered in the study.

DISCUSSION

On making the diagnosis of rheumatic fever by auscultation, Alvan Feinstein wrote in his book Clinical Judgment “The main problems of observer variability were neither in the eyes nor the ears of the observers. We all saw and heard essentially the same things, but each observer used different ingredients in his criteria for description and interpretation of the observations”[12]. The same can be said about medical imaging in which interobserver variability remains a critical issue[13].

Bedside cranial ultrasound is the neuroimaging standard of care for the detection of IVH[14]. Cranial sonography is cost-effective, does not require sedation, and is portable, allowing for the evaluation of critical patients at the bedside. In a study by Maalouf et al[15], ultrasound has a predictive probability of 0.85 (0.76-0.94) for the presence of IVH on MRI.

The interobserver agreement for findings on neonatal head ultrasound varies from poor to excellent among radiologists[16]. Although there is a possibility of intra-observer agreement, this is quite low. While numerous studies have explored interobserver variability in neonatal cranial ultrasonography, ours is the first to study the differences in interpretation between senior residents and board-certified pediatric radiologists.

In our study, there was excellent agreement between the senior resident and the attending for intracranial hemorrhage. There was substantial agreement on grade I and grade II intraventricular hemorrhage, whilst agreement on grade III and grade IV intraventricular hemorrhage was almost perfect.

Among radiologists, experienced neonatologists, and less experienced neonatologists involved in a study by Hagmann et al[17], the interobserver agreement in the interpretation of cranial ultrasound ranged from poor to good. Hintz et al[18] found excellent interobserver agreement on severe intraventricular hemorrhage, but poor agreement on periventricular leukomalacia between experienced board-certified radiologists with special expertise in cranial ultrasound. Among radiologists, pediatric neurologists, and neonatologists experienced in neonatal ultrasounds, Pinto et al[19] obtained excellent interobserver agreement for major findings such as parenchymal hemorrhage, but rather poor agreement for less severe pathologies such as germinal matrix hemorrhage. A study by Corbett et al[16] found excellent agreement on high-grade hemorrhage but poor agreement on interpretation of ventricular size.

Even though we did not find any prior studies on interobserver variability between senior residents and radiologists, our results are comparable to previous studies in that there is excellent agreement on major abnormalities such as grade III and IVH.

The board-certified radiologist's experience is most likely to be responsible for the study's relatively low interobserver agreements for grade I and II IVH. Because of greater image gain/depth, improved probe handling, and knowledge of pertinent anatomy, experience with doing cranial ultrasonography can result in better imaging quality.

This experience also manifests itself in improved ultrasound interpretation performed by others. Due to inexperience, the resident in our study initially missed the additional findings of hydrocephalus, choroid plexus abnormalities, and periventricular echogenicity as indicated in Table 2. We plan to conduct a follow-up study to investigate abnormalities other than IVH on neonatal cranial ultrasound which can have significant impact on disease prognosis.

Table 2 Additional ultrasound examination findings initially missed by the resident.
Additional findings on neonatal cranial ultrasound (%)
Ventricular abnormalities12 (6)
Mild hydrocephalus11 (5.5)
Severe hydrocephalus1 (0.5)
Choroid plexus abnormalities4 (2)
Increased periventricular echogenicity4 (2)

This study has some limitations. The residents and pediatric radiologists were compared for interobserver agreement only on one variable, i.e., IVH, but no cross-sectional neuroimaging such as CT or MRI was performed for confirmation. The interobserver agreement was not calculated for additional findings such as parenchymal hemorrhage, hydrocephalus, and venous infarctions, which also have implications for neonatal neurodevelopment.

CONCLUSION

Interobserver agreement regarding detection of intraventricular hemorrhage is high for low-grade hemorrhage and almost perfect for high-grade hemorrhage between residents and board-certified pediatric radiologists.

ARTICLE HIGHLIGHTS
Research background

Neonatal cranial ultrasound examinations were evaluated in neonatal intensive care unit (NICU) patients. Ultrasound findings were recorded for the resident performing the ultrasound and the pediatric attending radiologist.

Research motivation

Despite the ostensibly important role of accurate cranial ultrasound interpretation, few studies have investigated the reliability of interpretation of cranial ultrasound. Differences in the identification and grading of intraventricular hemorrhage (IVH) may affect the clinical outcome and the subsequent management options. This is the reason the study was undertaken.

Research objectives

To assess interobserver reliability between senior radiology residents performing bedside cranial ultrasounds during on-call hours and board-certified pediatric radiologists.

Research methods

A total of 200 neonatal cranial ultrasound examinations were evaluated in NICU patients. Ultrasound findings were recorded for both the resident performing the ultrasound and the pediatric attending radiologist. Interobserver agreement was calculated.

Research results

The mean gestational age was 30.9 wk. Interobserver agreement for higher grade (Grade III & IV) IVH was excellent. There was substantial agreement for lower grade (Grade I & II) IVH.

Research conclusions

Interobserver agreement for detection of IVH is high for low-grade hemorrhage and almost perfect for high-grade hemorrhage between radiology residents and board certified pediatricians.

Research perspectives

Our study results are limited by the cross sectional nature of the study. Additionally, we did not compare agreement on the interpretation of periventricular leukomalacia, incidental findings, and degree of ventriculomegaly if it was present which can have significant impact on disease prognosis. This may be explored in a future study.

Footnotes

Provenance and peer review: Invited article; Externally peer reviewed.

Peer-review model: Single blind

Specialty type: Radiology, nuclear medicine and medical imaging

Country/Territory of origin: Pakistan

Peer-review report’s scientific quality classification

Grade A (Excellent): A

Grade B (Very good): B

Grade C (Good): C

Grade D (Fair): 0

Grade E (Poor): 0

P-Reviewer: Aydin S, Turkey; Moshref RH, Saudi Arabia; Schoenhagen P, United States S-Editor: Liu JH L-Editor: Ma JY P-Editor: Liu JH

References
1.  Allen KA. Treatment of intraventricular hemorrhages in premature infants: where is the evidence? Adv Neonatal Care. 2013;13:127-130.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 7]  [Cited by in F6Publishing: 8]  [Article Influence: 0.8]  [Reference Citation Analysis (0)]
2.  Sherlock RL, Anderson PJ, Doyle LW; Victorian Infant Collaborative Study Group. Neurodevelopmental sequelae of intraventricular haemorrhage at 8 years of age in a regional cohort of ELBW/very preterm infants. Early Hum Dev. 2005;81:909-916.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 124]  [Cited by in F6Publishing: 131]  [Article Influence: 7.3]  [Reference Citation Analysis (0)]
3.  Murphy BP, Inder TE, Rooks V, Taylor GA, Anderson NJ, Mogridge N, Horwood LJ, Volpe JJ. Posthaemorrhagic ventricular dilatation in the premature infant: natural history and predictors of outcome. Arch Dis Child Fetal Neonatal Ed. 2002;87:F37-F41.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 205]  [Cited by in F6Publishing: 218]  [Article Influence: 10.3]  [Reference Citation Analysis (0)]
4.  Pinto-Martin JA, Whitaker AH, Feldman JF, Van Rossem R, Paneth N. Relation of cranial ultrasound abnormalities in low-birthweight infants to motor or cognitive performance at ages 2, 6, and 9 years. Dev Med Child Neurol. 1999;41:826-833.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 88]  [Cited by in F6Publishing: 89]  [Article Influence: 4.0]  [Reference Citation Analysis (0)]
5.  Wilson-Costello D, Friedman H, Minich N, Fanaroff AA, Hack M. Improved survival rates with increased neurodevelopmental disability for extremely low birth weight infants in the 1990s. Pediatrics. 2005;115:997-1003.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 465]  [Cited by in F6Publishing: 501]  [Article Influence: 27.4]  [Reference Citation Analysis (0)]
6.  Ballabh P, Xu H, Hu F, Braun A, Smith K, Rivera A, Lou N, Ungvari Z, Goldman SA, Csiszar A, Nedergaard M. Angiogenic inhibition reduces germinal matrix hemorrhage. Nat Med. 2007;13:477-485.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 108]  [Cited by in F6Publishing: 118]  [Article Influence: 7.2]  [Reference Citation Analysis (0)]
7.  Osborn DA, Evans N, Kluckow M. Hemodynamic and antecedent risk factors of early and late periventricular/intraventricular hemorrhage in premature infants. Pediatrics. 2003;112:33-39.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 184]  [Cited by in F6Publishing: 203]  [Article Influence: 9.7]  [Reference Citation Analysis (0)]
8.  Babnik J, Stucin-Gantar I, Kornhauser-Cerar L, Sinkovec J, Wraber B, Derganc M. Intrauterine inflammation and the onset of peri-intraventricular hemorrhage in premature infants. Biol Neonate. 2006;90:113-121.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 28]  [Cited by in F6Publishing: 30]  [Article Influence: 1.8]  [Reference Citation Analysis (0)]
9.  Vural M, Yilmaz I, Ilikkan B, Erginoz E, Perk Y. Intraventricular hemorrhage in preterm newborns: risk factors and results from a University Hospital in Istanbul, 8 years after. Pediatr Int. 2007;49:341-344.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 16]  [Cited by in F6Publishing: 18]  [Article Influence: 1.1]  [Reference Citation Analysis (0)]
10.  Muhammad A, Waheed AA, Alvi MI, Khan N, Sayani R. Interobserver Agreement on Focused Assessment with Sonography for Trauma in Blunt Abdominal Injury. Cureus. 2018;10:e2592.  [PubMed]  [DOI]  [Cited in This Article: ]  [Reference Citation Analysis (0)]
11.  Volpe JJ. Intraventricular hemorrhage and brain injury in the premature infant. Diagnosis, prognosis, and prevention. Clin Perinatol. 1989;16:387-411.  [PubMed]  [DOI]  [Cited in This Article: ]
12.  Feinstein AR  Clinical Judgement. 1976.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 22]  [Cited by in F6Publishing: 23]  [Article Influence: 0.5]  [Reference Citation Analysis (0)]
13.  Benchoufi M, Matzner-Lober E, Molinari N, Jannot AS, Soyer P. Interobserver agreement issues in radiology. Diagn Interv Imaging. 2020;101:639-641.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 99]  [Cited by in F6Publishing: 108]  [Article Influence: 49.5]  [Reference Citation Analysis (0)]
14.  Lowe LH, Bailey Z. State-of-the-art cranial sonography: Part 1, modern techniques and image interpretation. AJR Am J Roentgenol. 2011;196:1028-1033.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 35]  [Cited by in F6Publishing: 39]  [Article Influence: 3.2]  [Reference Citation Analysis (0)]
15.  Maalouf EF, Duggan PJ, Counsell SJ, Rutherford MA, Cowan F, Azzopardi D, Edwards AD. Comparison of findings on cranial ultrasound and magnetic resonance imaging in preterm infants. Pediatrics. 2001;107:719-727.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 267]  [Cited by in F6Publishing: 281]  [Article Influence: 12.7]  [Reference Citation Analysis (0)]
16.  Corbett SS, Rosenfeld CR, Laptook AR, Risser R, Maravilla AM, Dowling S, Lasky R. Intraobserver and interobserver reliability in assessment of neonatal cranial ultrasounds. Early Hum Dev. 1991;27:9-17.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 17]  [Cited by in F6Publishing: 13]  [Article Influence: 0.5]  [Reference Citation Analysis (0)]
17.  Hagmann CF, Halbherr M, Koller B, Wintermark P, Huisman T, Bucher HU; Swiss Neonatal Network. Interobserver variability in assessment of cranial ultrasound in very preterm infants. J Neuroradiol. 2011;38:291-297.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 12]  [Cited by in F6Publishing: 9]  [Article Influence: 1.1]  [Reference Citation Analysis (0)]
18.  Hintz SR, Slovis T, Bulas D, Van Meurs KP, Perritt R, Stevenson DK, Poole WK, Das A, Higgins RD; NICHD Neonatal Research Network. Interobserver reliability and accuracy of cranial ultrasound scanning interpretation in premature infants. J Pediatr. 2007;150:592-596, 596.e1.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 70]  [Cited by in F6Publishing: 76]  [Article Influence: 4.7]  [Reference Citation Analysis (0)]
19.  Pinto J, Paneth N, Kazam E, Kairam R, Wallenstein S, Rose W, Rosenfeld D, Schonfeld S, Stein I, Witomski T. Interobserver variability in neonatal cranial ultrasonography. Paediatr Perinat Epidemiol. 1988;2:43-58.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 28]  [Cited by in F6Publishing: 31]  [Article Influence: 0.8]  [Reference Citation Analysis (0)]