Update report on the quality of gliomas radiomics: An integration of bibliometric and radiomics quality score

Abstract

BACKGROUND

Despite the increasing number of publications on glioma radiomics, challenges persist in clinical translation.

AIM

To assess the development and reporting quality of radiomics in brain gliomas since 2019.

METHODS

A bibliometric analysis was conducted to reveal trends in brain glioma radiomics research. The Radiomics Quality Score (RQS), a metric for evaluating the quality of radiomics studies, was applied to assess the quality of adult-type diffuse glioma studies published since 2019. The total RQS score and the basic adherence rate for each item were calculated. Subgroup analysis by journal type and research objective was performed, correlating the total RQS score with journal impact factors.

RESULTS

The radiomics research in glioma was initiated in 2011 and has witnessed a surge since 2019. Among the 260 original studies, the median RQS score was 11, correlating with a basic compliance rate of 46.8%. Subgroup analysis revealed significant differences in domain 1 and its subitems (multiple segmentations) across journal types (P = 0.039 and P = 0.03, respectively). The Spearman correlation coefficients indicated that the total RQS score had a negative correlation with the Journal Citation Report category (-0.69) and a positive correlation with the five-year impact factors (0.318) of journals.

CONCLUSION

Glioma radiomics research quality has improved since 2019 but necessitates further advancement with higher publication standards.

Key Words: Glioma; Adult-type; Radiomics; Bibliometrics; Quality score

Core Tip: Bibliometric and radiomics quality scores were used to assess glioma radiomics research, and the results showed the quality of glioma radiomics research and reporting requires further improvement.

INTRODUCTION

Gliomas are the most common primary tumors of the central nervous system. While preoperative non-invasive imaging can provide critical information on differential diagnosis, molecular typing, grading, prognosis, and treatment response of gliomas, conventional analysis methods often underperform in delivering adequate results[1].

Radiomics, an artificial intelligence (AI)-assisted technology, offers a novel approach by extracting high-dimensional feature data from medical images[2]. Therefore, radiomics has attracted great interest among clinicians and researchers[1], with its application in glioma studies including differential diagnosis[3,4], grading and classification[5-7], survival analysis[8,9], and treatment response[10]. Despite its potential, radiomics is not yet widely adopted in clinical practice, primarily due to the lack of large-scale data validation, methodological rigor, and high-quality reporting. Radiomics Quality Score (RQS), an evaluation system developed by Lambin, aims to address these challenges by assessing the effectiveness and integrity of radiomics research[11]. Based on 16 criteria, the RQS rewards and penalizes methodological and analytical aspects to promote best scientific practices. Previous studies have utilized the RQS to evaluate the quality of radiomics in neuro-oncology, revealing a notable absence of high-level evidence[12]. Our present study aims to update the insights into the field of glioma radiomics by focusing on articles published since February 2019, which were not covered in prior assessments[13].

Bibliometrics, as a method of information visualization, enables a quantitative analysis of publication characteristics such as timing, authorship, and citations, providing a comprehensive view of research trends and hotspots within a specific domain[14-16]. It can achieve quantitative analysis of literature in a specific research field in a worldwide context through statistical methods and visualizing the results with the helps of software such as VOSviewer and CiteSpace, which can analyze and visualize literature data, general network data, and text data through co-occurrence clustering to present the scientific knowledge map[14-16]. In this study, we apply bibliometric analysis to delineate the development and trends in glioma radiomics research and employ the RQS to evaluate the quality of recent studies on adult-type diffuse gliomas. Our findings aim to offer valuable guidance for future research endeavors and facilitate clinical translation in the field of glioma radiomics.

MATERIALS AND METHODS

Literature retrieval and manual screening

We designed a search strategy tailored to our research focus on radiomic applications in glioma, with specific inclusion criteria including: (1) English-language papers; (2) Focused on glioma; (3) Integrated radiomic analysis; and (4) Published between 2011 and 2023. The final search was updated on April 22, 2023. A team of three authors (Cao X, Xiong M, and Yang J) screened potentially relevant papers, resolving any uncertainties through discussion. Unlike systematic review, bibliometric analysis primarily requires abstract screening, with full-text review reserved for select cases.

According to the screening criteria, a total of 773 articles were included for bibliometric analysis in the study after retrieval from PubMed and Embase databases and integration with the WOS database (the literature retrieval method can be found in Supplementary material). Then two authors (Liu Z and Zhang LQ) further screened out articles that could be used for RQS scoring. Articles that met the following criteria were excluded: (1) Review or Meta-analysis; (2) Original research articles published in 2019 and before; (3) Focused on localized glioma and pediatric and diffuse midline gliomas, and (4) Radiomics or radiogenomics studies using only correlations and without a performance measurement. Figure 1 shows the inclusion and exclusion criteria of bibliometric and RQS analyses in this study.

Figure 1  Flow chart of inclusion and exclusion criteria for bibliometrics and Radiomics Quality Score analyses.

Bibliometric analysis and visualization

We employed VOSviewer (version 1.6.18) and CiteSpace (version 6.2.R4) for bibliometric visualization, examining cooperation networks and keyword co-occurrences[17-20]. Scimago Graphica (version 1.0.35) supported the visualization of country/region analysis[21,22]. CiteSpace was utilized to assess leading journals' collaboration and field-specific interrelationships. Microsoft Excel 2019 facilitated the analysis of targeted files and publication trends. The top ten most cited or productive entities in each category were analyzed and tabulated. The top ten most cited or productive countries/regions, institutions, and journals were also analyzed.

RQS data extraction and analysis

Three experienced reviewers (Yang J, Kan YB, and Cao X, with 5, 8, and 10 years in neuro-oncology imaging, respectively) scored RQS based on full-text articles. A workshop was conducted to ensure a unified understanding of RQS assessment criteria. Data extraction was guided by six predefined domains, with details of the 16 RQS components provided in Supplementary Table 1 in the Supplementary material. Each article was independently evaluated by two assessors, with discrepancies resolved through consensus discussions moderated by Cao X.

Statistical analysis

Basic adherence for RQS items, domains, or total scores was set at a minimum score of 1. Adherence rates were calculated as ratios of compliant articles. The Kruskal-Wallis test was applied to compare RQS scores across research types (differential diagnosis, grading or molecular typing, prognosis, and treatment response) and journals (imaging, clinical and comprehensive journals) subgroups, with Bonferroni correction for multiple comparisons. Statistical significance was set at P < 0.05. All analyses were performed using SPSS software package (version 22, SPSS Inc., Chicago, Illinois).

RESULTS

Annual publication trends

Our bibliometric analysis encompassed 773 papers, including 667 original articles and 106 review articles (Figure 1). The inception of glioma radiomics research in 2011 marked a gradual increase, with a significant surge since 2019, constituting nearly 80% of the total decade's output (Figure 2). The collective citation count stands at 15525, with an average of 20.71 citations per paper.

Figure 2  Global trend of publications on glioma radiomics from 2011 to 2023.

Productive countries/regions

Fifty-three countries/regions contributed to the field, with the United States (274 publications), China (242), and Germany (73) leading in publication volume, representing 35.45%, 31.31%, and 9.44% of the total, respectively. The United States also dominated in total citations and average citations per paper, at 7959 and 29.05, respectively, surpassing China's 4039 and 16.70 citations (Supplementary Table 2). The cooperative network among countries could be divided into six cooperative network clusters by VOS clustering and showed that the United States and China had more cooperative relations with other countries (Figure 3).

Figure 3 Inter-country cooperation network map. The size of the nodes indicates the amount of publications, and the thickness of the lines represents the intensity of cooperation.

Productive institutions

Over 12 years, 200 institutions have published on this topic, with the Chinese Academy of Sciences, University of Pennsylvania, and Capital Medical University emerging as the top three contributors. The German Cancer Research Center excelled in citation metrics, suggesting high impact (Supplementary Table 3). In addition, Stanford University, Chinese Academy of Sciences, and University of Utah showed the strongest institutional linkages, and the transnational cooperation among institutions of other countries was generally low (Supplementary Figure 1).

Journals

The research was disseminated across 197 journals, with Neuro-Oncology, Frontiers in Oncology, and Cancers being the most prolific (Table 1). The top 10 journals in this domain were ranked above the journal citation report (JCR) Q2 division. Scientific Reports, European Radiology, and Neuro-Oncology garnered the highest citations, with counts of 1483, 920, and 906, respectively. The co-occurrence network of these journals is depicted in Figure 4.

Figure 4 Co-occurrence network analysis of top journals. The lines represent partnerships, and different colors represent different years of cooperation. Larger circles mean more publications.

Table 1 Top 10 journals related to gliomas radiomics.

Rank	Journal (country)	Count	IF (2022)	JCR (2022)	Total citations
1	Neuro-Oncology (United States)	63	15.9	Q1	906
2	Frontiers in Oncology (Switzerland)	62	4.7	Q2	372
3	Cancers (United States)	46	5.2	Q2	190
4	European Radiology (Germany)	44	5.9	Q1	920
5	Scientific Reports (England)	41	4.6	Q2	1483
6	Journal of Magnetic Resonance Imaging (United States)	20	4.4	Q1	532
7	American Journal of Neuroradiology (United States)	19	3.5	Q2	483
8	Medical Physics (United States)	18	3.8	Q2	102
9	Journal of Neuro-Oncology (United States)	17	3.9	Q2	287
10	International Journal of Radiation Oncology Biology Physics (United States)	15	7	Q1	71

IF: Impact factor; JCR: Journal citation report.

Keywords

In the co-occurrence analysis of keywords, key emerging terms in recent years included "glioblastoma", "radiomics", "survival", "classification", and "MRI" (Figure 5). Disease type and clinical goal analyses reveal a focus on classification and survival, with less emphasis on prognosis and diagnosis (Figure 6A). "Glioblastoma" and "astrocytoma" dominate the study subjects (Figure 6B).

Figure 5  Co-occurrence network analysis of keywords regarding radiomics in glioma.

Figure 6 Analysis of keywords from clinical goals and disease types. A: Distribution of publications by goal; B: Distribution of publications by disease category.

RQS assessment characteristics

The RQS assessment included 260 articles, with 20.4% lacking verification and 27.3% being multicenter studies. Detailed characteristics and RQS scores are available in Supplementary Table 4. Research goals were categorized into four subgroups including differential diagnosis (15.7%), treatment response (12.3%), prognosis prediction (33.5%), and grading or molecular typing (38.5%) with the dataset sizes (median, interquartile range) being (120.20-935), (90.15-463), (162.22-704), and (130.36-894), respectively (P < 0.01). The overall basic adherence rate for RQS was 46.8%, with rates for domains 1 to 6 being 99.2%, 79.6%, 77.7%, 98.5%, 1.5%, and 58.1%, respectively (Table 2).

Table 2 Basic adherence rate according to the six key domains.

Radiomics quality score	Basic adherence rate (%)
Total 16 items	46.80
Domain 1: Protocol quality and stability in image and segmentation	99.20
Protocol quality	98.80
Test-retest	36.90
Phantom study	10.40
Multiple segmentation	33.10
Domain 2: Feature selection and validation	79.60
Feature reduction or adjustment of multiple testing	98.80
Validation	79.6
Domain 3: Biologic/clinical validation and utility	77.70
Multivariate analysis with non-radiomics features	67.30
Biologic correlates	27.70
Comparison to “gold standard”	5.80
Potential clinical utility 1	18.10
Domain 4: Model performance index	98.50
Discrimination statistics	97.30
Calibration statistics	60.80
Cut-off analysis	54.60
Domain 5: High level of evidence	1.50
Prospective study	0.40
Cost-effective analysis	1.10
Domain 6: Open science and data	58.10

Subgroups

The median and interquartile range of total RQS score was 11 (9-14). Subgroup analysis by journal type revealed significant differences in domain 1 (P = 0.039) and its subitems (P = 0.03). Pairwise comparisons between groups showed significant differences only in domain 1 (P = 0.036). Research goal subgroup analysis indicated significant differences across all domains except domain 3 (P = 0.18). Specific differences are detailed in Table 3 and Table 4, with pairwise comparisons provided in Supplementary Table 5.

Table 3 Subgroup analysis in journal types, median (interquartile range).

Radiomics quality score	Median score	Imaging journals	Clinical journals	Comprehensive journals	P value
Total 36 points	11 (9–14)	11 (8.25–14)	12 (9–16)	11.5 (8–15)	0.379
Domain 1: Protocol quality and stability in image and segmentation (0 to 5 points)	1 (1–2)	1 (1–2)	2 (1–2)	1 (1–2)	0.039
Image protocol quality (2)	1 (1–1)	1 (1–1)	1 (1–1)	1 (1–1)	0.201
Multiple segmentations (1)	0 (0–1)	0 (0–1)	0 (0–1)	0 (0–0.25)	0.03
Phantom study on all scanners (1)	0 (0–0)	0 (0–0)	0 (0–0)	0 (0–0)	0.205
Imaging at multiple time points (1)	0 (0–0)	0 (0–0)	0 (0–0)	0 (0–0)	0.383
Domain 2: Feature selection and validation (-8 to 8 points)	5 (5–6)	5 (5–6)	5 (5–6)	5 (1–2)	0.553
Feature reduction or adjustment for multiple testing (-3 or 3)	3 (3–3)	3 (3–3)	3 (3–3)	3 (3–3)	0.539
Validation (-5, 2, 3, 4, or 5)	2 (2–3)	2 (2–3)	2 (2–3)	2 (1.5–2.25)	0.585
Domain 3: Biologic/clinical validation and utility (0 to 6points)	1 (1–2)	1 (0.25–2)	1 (1–2)	1 (0–2)	0.613
Multivariable analysis with non-radiomics features (1)	1 (0–1)	1 (0–1)	1 (0–1)	1 (0–1)	0.059
Detect and discuss biological correlates (1)	0 (0–1)	0 (0–0)	0 (0–1)	0 (0–1)	0.075
Comparison to ‘gold standard’ (2)	0 (0–0)	0 (0–0)	0 (0–0)	0 (0–0)	0.669
Potential clinical utility (2)	0 (0–0)	0 (0–0)	0 (0–0)	0 (0–0)	0.628
Domain 4: Model performance index (0 to 5 points)	3 (2–4)	3 (2–4)	3 (2–4)	3 (2–5)	0.315
Cut-off analyses (1)	1 (0–1)	0 (0–1)	1 (0–1)	1 (0–1)	0.101
Discrimination statistics (2)	2 (1–2)	2 (1.5–2)	2 (1–2)	2 (1.75–2)	0.071
Calibration statistics (2)	1 (0–2)	1 (0–2)	1 (0–2)	0.5 (0–2)	0.175
Domain 5: High level of evidence (0 to 8 points)	0 (0–0)	0 (0–0)	0 (0–0)	0 (0–0)	0.588
Prospective study registered in a trial database (7)	0 (0–0)	0 (0–0)	0 (0–0)	0 (0–0)	0.435
Cost-effectiveness analysis (1)	0 (0–0)	0 (0–0)	0 (0–0)	0 (0–0)	0.236
Domain 6: Open science and data (0 to 4 points)	0.653 (0–1)	1 (0–1)	1 (0–1)	0.5 (0–2)	0.649

Table 4 Subgroup analysis in the goal of research, median (interquartile range).

Radiomics quality score	Differential diagnosis	Treatment response	Prognosis prediction	Grading or molecular typing	P value
Total 36 points	10 (8–13)	11 (8–14)	13 (11–15)	11 (9–15)	0
Domain 1: Protocol quality and stability in image and segmentation (0 to 5 points)	2 (1–2)	2 (1–2)	1 (1–2)	1 (1–2)	0.006
Image protocol quality (2)	1 (1–1)	1 (1–1)	1 (1–1)	1 (1–1)	< 0.001
Multiple segmentations (1)	1 (1–1)	0 (0–1)	0 (0–0.75)	1 (0–1)	< 0.001
Phantom study on all scanners (1)	0 (0–0)	0 (0–0)	0 (0–0)	0 (0–0)	0.002
Imaging at multiple time points (1)	0 (0–0)	0 (0–0)	0 (0–0)	0 (0–0)	0.36
Domain 2: Feature selection and validation (-8 to 8 points)	5 (–2–5)	5 (–2–6)	5 (5–6)	5 (5–5)	0.012
Feature reduction or adjustment for multiple testing (-3 or 3)	3 (3–3)	3 (3–3)	3 (3–3)	3 (3–3)	0.601
Validation (-5, 2, 3, 4, or 5)	2 (–5–2)	2 (2–2)	2 (2–3)	2 (–5–3)	0.007
Domain 3: Biologic/clinical validation and utility (0 to 6 points)	2 (0–2)	1 (1–2)	1 (1–2)	2 (0–2)	0.18
Multivariable analysis with non-radiomics features (1)	0 (0–0)	1 (0–1)	1 (1–1)	1 (0–1)	< 0.001
Detect and discuss biological correlates (1)	0 (0–0)	0 (0–0)	0 (–0–1)	0 (0–1)	0.001
Comparison to ‘gold standard’ (2)	0 (0–2)	0 (0–0)	0 (0–0)	0 (0–0)	< 0.001
Potential clinical utility (2)	0 (0–2)	0 (0–0)	0 (0–0)	0 (0–0)	0.032
Domain 4: Model performance index (0 to 5 points)	2 (2–2)	3 (2–4)	4 (3–5)	3 (2–4)	< 0.001
Cut-off analyses (1)	0 (0–1)	0 (0–1)	1 (1–1)	0 (0–1)	< 0.001
Discrimination statistics (2)	2 (1.5–2)	2 (2–2)	2 (1–2)	2 (1–2)	0.049
Calibration statistics (2)	0 (0–0)	0 (0–2)	0 (1–2)	0 (0–2)	< 0.001
Domain 5: High level of evidence (0 to 8 points)	0 (0–0)	0 (0–0)	0 (0–0)	0 (0–0)	0.001
Prospective study registered in a trial database (7)	0 (0–0)	0 (0–0)	0 (0–0)	0 (0–0)	0.148
Cost-effectiveness analysis (1)	0 (0–0)	0 (0–0)	0 (0–0)	0 (0–0)	< 0.001
Domain 6: Open science and data (0 to 4 points)	0 (0–1)	1 (0–1)	1 (0–1)	0 (0–1)	0.016

Correlation between RQS and bibliometrics

The Spearman correlation coefficients between the total RQS score and the JCR division and the 5-year impact factors were -0.69 and 0.318, respectively, indicating a stronger correlation with the JCR division. The total RQS score positively correlated with sample size (coefficient 0.336) but was not significantly associated with research goals, journal types, or publication year (P = 0.523, P = 0.349, and P = 0.435, respectively).

DISCUSSION

Our bibliometric analysis identified 2019 as a critical inflection point, marking a rapid increase in glioma radiomics research. Prominent contributions from the United States, China, and Germany, alongside a predominance of American publishers, highlight the global significance of this field. The presence of both medical and engineering journals among the top ten indicates a robust interdisciplinary approach, underscoring the importance of integrating medical and engineering perspectives in radiomics research. Furthermore, the analysis reveals a need to enhance transnational inter-agency cooperation, potentially offering more profound insights into glioma research globally.

The introduction of molecular biomarkers into the classification of gliomas since 2016[23], with increased emphasis in 2021[24,25], aligns with the observed focus on classification in most studies. Glioblastoma, the glioma subtype with the worst prognosis and highest postoperative recurrence rate, understandably commands significant research attention[26], and the trend of similar proportions is presented in keyword analysis (Figure 6B). Thus, our RQS analysis focused on adult-type diffuse gliomas.

Our study showed an increase in ROS score compared to a 2019 survey[12] (37.1% vs 46.8%), which may be attributed to enhancements in several key areas including Domain 1 (which assesses the quality of image acquisition and preprocessing) and Domains 3 and 4 (which focus on the validation and performance of radiomics models). Notably, the openness and sharing of scientific data in Domain 6 have seen the most substantial increase, reflecting a positive trend towards transparent and reproducible research practices. Despite these improvements, the overall score of Domain 2 remained stagnant, although its validation score did increase, indicating a need for continued focus on this area. However, several obstacles persist in clinical practice, warranting further attention. In the realm of test-retest reliability within Domain 1, an alarmingly low number of studies—specifically two[27,28]-have employed multi-timepoint imaging. The impact of varying imaging protocols on feature robustness is another area that requires greater scrutiny, as evidenced by the mere 10.4% of studies incorporating phantom-based validation. While standard image preprocessing pipelines are essential for harmonizing captured images[29], the adoption of tools like Combat presents a significant advancement. Combat has proven effective not only in reducing inconsistencies in radiomic feature values across magnetic resonance imaging scanners and centers but also in refining the precision of subsequent experimental analyses[30-32].

Although the validation score has improved compared to previous studies, there is a notable absence of generalizability. External validation is imperative, as algorithm performance is often compromised when assessed on external datasets[30,31]. Our analysis indicated a positive correlation between the RQS total score and sample size, suggesting that larger sample sizes may enhance the reliability of external validation. The disparity in RQS scores across different research goals could be attributed to the limited sample size in the treatment response subgroup, which adversely affects external validation scores. The lack of prospective studies, which carry the highest weight in scoring, is another critical area for future research. The absence of cost-benefit analyses in the studies surveyed may further impede the clinical translation and application of radiomics.

The significant improvement in the "Open science and data" domain is a step in the right direction; however, achieving complete re-presentation of research remains elusive. The reluctance to share open-access data may stem from restrictions on patient data release. Nonetheless, the full disclosure of scientific data, devoid of sensitive information and with accessible storage solutions, is essential for the clinical translation of radiomic findings[32,33].

These challenges are not confined to brain tumor research. A systematic review of prostate studies[34] revealed no public data sharing, and similar adherence issues were found in multiple myeloma studies[35] and rectal cancer research[36]. The radiomics field in stroke[37] also faces the dilemma of repeatability. While not all radiomics studies require phantom-based test-retest experiments or multi-timepoint scanning, a prospective design, multi-center collaboration, adequate sample size, comprehensive external verification, and a focus on retest research are pivotal for the future of glioma radiomics.

Our findings suggest a correlation between the reputation of journals and the publication of high-quality research, with a Spelman correlation coefficient of -0.69 between the RQS total score and the JCR division. This underscores the critical role of publishers and reviewers in enforcing stringent publication standards and addressing the aforementioned obstacles in clinical transformation. The recent Check List for Evaluation of Radiomics (CLEAR checklist) published by the European Society of Radiology and the European Society of Medical Imaging Informatics offers a valuable framework for enhancing the quality of radiomics research reporting[38].

Our study acknowledges several limitations. The quality assessment was primarily concentrated on adult-type diffuse gliomas, which, while prevalent, do not encompass the full spectrum of gliomas. Additionally, our evaluation did not extend to studies incorporating deep learning or its integration with radiomics, as the RQS is not optimally designed to assess such methodologies. Nevertheless, emerging AI modeling checklists, including the Minimum Information for Medical AI Reporting checklist[39], Checklist for AI in Medical Imaging[40], and extensions of the Transparent Reporting of a multivariable prediction model of Individual Prognosis Or Diagnosis statement and the Prediction model Risk Of Bias Assessment Tool[41], offer complementary criteria to enhance the RQS framework. Notably, a novel methodological scoring tool, Metrics[42], developed by an international consortium of experts, addresses this gap, providing a comprehensive 30-item evaluation spanning nine domains, suitable for both handcrafted radiomics and deep learning-based pipelines. Future comparisons between these tools and the exploration of Metrics' impact on enhancing imaging research and clinical translation will be pivotal.

CONCLUSION

In summary, enhancing the methodological rigor and scientific reporting of glioma radiomics is imperative. The integrative approach of bibliometrics with RQS offers significant reference value for the future, guiding research design, execution, reporting, and pre-publication peer review. This synergy is poised to elevate the quality and clinical relevance of glioma radiomics, fostering a more robust and clinically viable body of research.