Diagnostic value of artificial intelligence automatic detection systems for breast BI-RADS 4 nodules

Abstract

BACKGROUND

The incidence rate of breast cancer has exceeded that of lung cancer, and it has become the most malignant type of cancer in the world. BI-RADS 4 breast nodules have a wide range of malignant risks and are associated with challenging clinical decision-making.

AIM

To explore the diagnostic value of artificial intelligence (AI) automatic detection systems for BI-RADS 4 breast nodules and to assess whether conventional ultrasound BI-RADS classification with AI automatic detection systems can reduce the probability of BI-RADS 4 biopsy.

METHODS

A total of 107 BI-RADS breast nodules confirmed by pathology were selected between June 2019 and July 2020 at Hwa Mei Hospital, University of Chinese Academy of Sciences. These nodules were classified by ultrasound doctors and the AI-SONIC breast system. The diagnostic values of conventional ultrasound, the AI automatic detection system, conventional ultrasound combined with the AI automatic detection system and adjusted BI-RADS classification diagnosis were statistically analyzed.

RESULTS

Among the 107 breast nodules, 61 were benign (57.01%), and 46 were malignant (42.99%). The pathology results were considered the gold standard; furthermore, the sensitivity, specificity, accuracy, Youden index, and positive and negative predictive values were 84.78%, 67.21%, 74.77%, 0.5199, 66.10% and 85.42% for conventional ultrasound BI-RADS classification diagnosis, 86.96%, 75.41%, 80.37%, 0.6237, 72.73%, and 88.46% for automatic AI detection, 80.43%, 90.16%, 85.98%, 0.7059, 86.05%, and 85.94% for conventional ultrasound BI-RADS classification with automatic AI detection and 93.48%, 67.21%, 78.50%, 0.6069, 68.25%, and 93.18% for adjusted BI-RADS classification, respectively. The biopsy rate, cancer detection rate and malignancy risk were 100%, 42.99% and 0% and 67.29%, 61.11%, and 1.87% before and after BI-RADS adjustment, respectively.

CONCLUSION

Automatic AI detection has high accuracy in determining benign and malignant BI-RADS 4 breast nodules. Conventional ultrasound BI-RADS classification combined with AI automatic detection can reduce the biopsy rate of BI-RADS 4 breast nodules.

Key Words: BI-RADS classification; Artificial intelligence; Breast nodules; Breast tumor

Core Tip: The accuracy of the AI-SONIC breast system in diagnosing BI-RADS 4 nodules is very high, which can improve the diagnostic accuracy of young doctors. It can also be used to upgrade and downgrade BI-RADS 4 nodules, guide clinical decision-making, reduce the biopsy rate for BI-RADS 4 nodules and prevent the waste of medical resources.

INTRODUCTION

Breast cancer is the most common malignancy among women worldwide. It is also the leading cause of cancer death in women, seriously threatening health[1]. In January 2021, the American Cancer Society noted in their 2020 global cancer statistics report[2] that the incidence rate of breast cancer has exceeded that of lung cancer, and it has become the type of cancer with the greatest number of malignant tumors worldwide (accounting for 11.7% of the total number of new cases). In addition, the mortality rate of breast cancer (6.9%) ranks fifth among cancers. Ultrasound is an important imaging examination method for breast cancer screening. In 2013, the American Society of Radiology released the fifth edition of the BI-RADS, which added ultrasound content based on the fourth edition and promoted standardized examination of breast ultrasound[3]. However, the malignancy risk of BI-RADS class 4 nodules covers a wide range of 2%-95%, clinical decision-making is challenging, and further puncture biopsy or surgical treatment is often required[4]. Artificial intelligence (AI) automatic detection systems for ultrasound breast cancer screening have attracted the attention of scholars in recent years because of their advantages of rapidity, accuracy and objectivity, providing efficient and accurate support to determine the benign or malignant nature of breast nodules[5]. This study attempted to explore whether an AI automatic detection system helps distinguish benign and malignant BI-RADS 4 breast nodules to reduce the likelihood of biopsy.

MATERIALS AND METHODS

Study design

From June 2019 to July 2020, 107 breast nodules from 92 patients with BI-RADS class 4 nodules, which were detected by routine ultrasound examination in our hospital and confirmed by pathology through puncture biopsy or operation, were examined. The maximum diameter of the nodules was 0.5-3.7 cm. All the patients were women aged 22-83 (45.1 ± 13.2) years who had undergone routine ultrasound and AI automatic detection system examination before surgery. The exclusion criteria were as follows: patients under the age of 18; patients who were pregnant or lactating; patients with breast prosthesis implantation; or patients with a history of previous breast surgery. The study was approved by the ethics committee of our hospital (ethics approval No. pj-nbey-ky-2019-060-01). All the subjects signed informed consent before the examination.

Ultrasound scanners and AI software

The examination was performed by 2 ultrasound doctors with professional training and 2 years of breast examination experience using a commercially available unit, EPIQ7 (Philips), with a high-frequency linear array probe (5–12 MHz).

Demetics is an AI system based on the deep learning framework De-Light that utilizes ultrasound images for big data analysis of breast nodules. Two convolutional neural networks (CNNs) of large and small sizes were built into the system, and the nodule probability was calculated for each pixel. Then, the separated connected regions were cascaded into a new CNN for 2 classifications. The system has good learning ability and growth after in-depth learning of approximately 50000 breast nodule pathological results. This model automatically identifies two-dimensional grayscale ultrasound images of breast nodules. Radiologists do not need to outline breast nodules. The operator must only import ultrasound images into Demetics, and the system obtains the risk coefficient of the thyroid nodule. The range of the risk coefficient is 0-1, and the cutoff value is set to be 0.5 by the system. If the risk coefficient is ≥ 0.5, the nodule is diagnosed as malignant; if the risk coefficient is < 0.5, the nodule is diagnosed as benign.

Study pipeline

The patient was placed in the supine position, and the upper limbs were raised to fully expose the breast and armpit. Two ultrasound doctors who had received professional training and had 2 years of breast examination experience performed the procedure. After discovering the nodule, they carefully observed it, recorded the boundary, shape and internal echo of the nodule, and classified it according to the BI-RADS classification standard recommended by Zhou et al[6]. In cases of disagreement, the result was determined by negotiation. Irregular shape, vertical growth, boundary hyperechoic halo, irregular edge, microcalcification and posterior echo attenuation were considered malignant indices: if one index was satisfied, the lesion was classified as 4A; if two indices were satisfied, it was classified as 4B; if three indices were satisfied, it was classified as 4C; if four or more indices were satisfied, it was classified as 5. In this study, conventional ultrasound BI-RADS classification defined class 4A nodules as benign and class 4B and 4C nodules as malignant. BI-RADS 4 nodules were selected for detection by an AI automatic detection system.

After routine ultrasound examination, the two doctors used AI automatic detection systems (professional technicians performed AI pre-job training for operators in the early stage). Static ultrasound images that clearly showed breast nodules were transmitted in DICOM format in real time and stored in an automatic sonic breast detection system for automatic labeling, processing and analysis. The breast nodules were automatically quantified and identified through the AI algorithm. According to the malignant characteristics of BI-RADS of the American College of Radiology[4], breast nodules include edge features, structural features, and calcification. The five characteristics of echo type and growth direction can be used to automatically assess benign and malignant nodules, and the probability value of benign and malignant nodules is interpreted by the recording system.

For conventional ultrasound BI-RADS classification with the AI automatic detection system prediction model, malignancy was considered when both indicated malignancies; otherwise, the diagnosis was benign. The prediction model of the AI automatic detection system adjusts the classification of conventional ultrasonic BI-RADS: If the AI score is greater than 0.5, the classification is increased by one category; If the AI score is less than 0.5, the classification is decreased by one category[7].

Statistical analysis

SPSS 21.0 statistical analysis software was used to calculate the sensitivity, specificity, accuracy, Jordan index, positive predictive value and negative predictive value of the conventional ultrasound BI-RADS classification, AI automatic detection system, conventional ultrasound BI-RADS classification with AI automatic detection system and adjusted BI-RADS classification diagnosis; the pathological results were used as the gold standard. The biopsy rate, cancer detection rate and malignancy risk rate of post BI-RADS classification diagnosis were compared with the χ² test. In all analyses, a P value below 0.05 was considered significant.

RESULTS

Pathological results

Among the 107 breast nodules, 61 were benign (57.01%), namely, 32 cases of fibroadenoma, 18 of adenopathy, 3 of granulomatous mastitis, 2 of intraductal papilloma, 2 of galactocele, 1 of plasma cell mastitis, 1 of phyllodes tumor, 1 of sclerosing adenosis, and 1 of nodular fasciitis, and 46 were malignant (42.99%), including 31 invasive ductal carcinomas. There were 5 cases of intraductal papillary carcinoma, 4 of invasive lobular carcinoma, 2 of encapsulated papillary carcinoma, 1 of mucinous carcinoma, 1 of undifferentiated carcinoma, 1 of malignant phyllodes tumor and 1 of solid papillary carcinoma (Table 1).

Table 1 results of 107 breast cases.

Pathological results	Number of nodules
Benign
Fibroadenoma	32
Adenosis	18
Granulomatous mastitis	3
Intraductal papilloma	2
Galactocele	2
Plasma cell mastitis	1
Phyllodes tumor	1
Sclerosing adenosis	1
Nodular fasciitis	1
Malignant
Invasive ductal carcinoma	31
Intraductal papillary carcinoma	5
Invasive lobular carcinoma	4
Encapsulated papillary carcinoma	2
Mucinous carcinoma	1
Undifferentiated carcinoma	1
Malignant phyllodes tumor	1
Solid papillary carcinoma	1

Comparison of the four diagnostic models with pathological results

According to conventional ultrasound BI-RADS classification, 4A nodules were classified as benign, and 4B and 4C nodules were classified as malignant; therefore, 59 malignant and 48 benign nodules were diagnosed. The AI automatic detection system defined 0-0.5 as benign and 0.6-1 as malignant, and 55 malignant and 52 benign nodules were diagnosed (Figure 1 and Figure 2). For conventional ultrasound BI-RADS classification with an AI automatic detection system, the presence of malignancy indices was defined as malignant, and others were defined as benign; therefore, 43 malignant and 64 benign nodules were diagnosed. According to the adjusted BI-RADS classification, if the AI score was greater than 0.5, the classification was upgraded by one category, and if the AI score was less than 0.5, the classification was downgraded by one category; therefore, 63 malignant and 44 benign nodules were diagnosed. BI-RADS classification distribution and risk prediction before and after adjustment were also performed (Table 2).

Figure 1 AI-SONIC breast system automatically recognizes markers and quantifies breast nodule characteristics. BI-RADS 4C, breast invasive ductal carcinoma confirmed by pathological findings. A: Conventional ultrasound BI-RADS classification suggests BI-RADS 4C; B: Automatic measurement and display of the growth direction; C: Edge feature analysis: the color changes from blue, green, yellow and red in turn to clear to blur; D: The dotted red line represents a strong echo; E: Based on the longitudinal section of the right breast nodule, the benign and. malignancy probability of this lesion was 0.84, as detected by artificial intelligence; F: The pathological diagnosis was invasive ductal carcinoma of the breast.

Figure 2 AI-SONIC breast system automatically recognizes markers and quantifies breast nodule characteristics. BI-RADS 4A, breast fibroadenoma confirmed by pathological findings. A: Conventional ultrasound BI-RADS classification suggests BI-RADS 4A; B: Automatic measurement and display of the growth direction; C: Edge feature analysis: the color changes from blue, green, yellow and red in turn to clear to blur; D: The dotted red line represents a strong echo; E: Based on the longitudinal section of the right breast nodule, the benign and malignancy probability of this lesion was 0.39, as detected by artificial intelligence; F: The pathological diagnosis was fibroadenoma of the breast.

Table 2 Diagnostic efficiency of four diagnostic models.

Inspection method	Pathology		Susceptibility (%)	Specificity (%)	Accuracy (%)	Jordan index	Positive predictive value (%)	Negative predictive value (%)
	Benign (n = 61)	Malignant (n = 46)
Conventional ultrasound BI-RADS classification			84.78	57.21	74.44	0.5199	61.11	85.42
Benign (n = 48)	41	7
Malignant (n = 59)	20	39
AI-SONIC Breast system			86.96	75.41	80.37	0.6237	72.73	88.46
Benign (n = 52)	46	6
Malignant (n = 53)	15	40
AI-SONIC Breast system combined BI-RADS classification of conventional ultrasound			80.43	90.16	85.98	0.7059	86.05	85.94
Benign (n = 64)	55	9
Malignant (n = 43)	6	37
Adjusted BI-RADS classification			93.48	67.21	78.50	0.6069	68.25	93.18
Benign (n = 44)	41	3
Malignant (n = 63)	20	43

Diagnostic efficiency of the four diagnostic models

The pathology results were considered the gold standard; furthermore, the sensitivity, specificity, accuracy, Youden index, positive predictive value and negative predictive value of conventional ultrasound BI-RADS classification diagnosis, the AI automatic detection system, the conventional ultrasound BI-RADS classification combined with AI automatic detection system and adjusted BI-RADS classification diagnosis were 84.78%, 67.21%, 74.77%, 0.5199, 66.10% and 85.42%; 86.96%, 75.41%, 80.37%, 0.6237, 72.73%, and 88.46%; 80.43%, 90.16%, 85.98%, 0.7059, 86.05%, and 85.94%; and 93.48%, 67.21%, 78.50%, 0.6069, 68.25%, and 93.18%, respectively (Table 3).

Table 3 BI-RADS classification distribution and risk prediction before and after adjustment.

Inspection method	Biopsy rate (%)	Malignancy risk (%)	Cancer detection rate (%)
BI-RADS classification before adjustment	100	0	42.99
4A (n = 48)
4B (n = 20)
4C (n = 39)
Adjusted BI-RADS classification	67.29	1.87	61.11
3 (n = 35)
4A (n = 9)
4B (n = 20)
4C (n = 11)
5 (n = 32)

DISCUSSION

The incidence and mortality of breast cancer in China are increasing annually with a growing disease burden. The prevention and treatment of breast cancer are very important[8]. Indeed, early accurate, reliable diagnosis and treatment are crucial for patient prognosis[9,10]. With the screening of breast cancer and attention to health, an increasing number of asymptomatic breast nodules are being identified[11]. According to the National Comprehensive Cancer Network breast cancer clinical practice guidelines[12], BI-RADS 4-type breast nodules should be assessed by biopsy, but only 2% of all breast nodules are positive. Chaiwerawattana et al[13] have reported that 92.35% of patients with BI-RADS class 4 breast nodules screened by the guidelines underwent unnecessary biopsies. This issue creates a burden on patients and wastes many medical resources. Although breast ultrasound has the advantages of simple operation, no radiation and low cost, it has large operator dependence and poor repeatability. In fact, there are great differences in ultrasound execution and the interpretation of images, which results in different BI-RADS classifications. For example, Wang et al[14] reported that among 220 cases of breast nodules, BI-RADS 4A was the dividing point between benign and malignant lesions. After multiple ultrasound examinations, up to 21.8% of cases had two different diagnostic results, which creates confusion among clinicians. This study aimed to find an objective and noninvasive method to determine benign and malignant BI-RADS class 4 nodules by applying an AI automatic detection system.

AI has powerful image analysis and information processing capabilities[15,16] and can mine ultrasonic image information that cannot be captured by human eyes. It can quickly, accurately and objectively analyze images, reduce doctors' burden, alleviate the impact on medical resources, improve the accuracy of diagnosis and help clinicians in prognosis and risk stratification to benefit a majority of patients[17,18]. The AI-SONIC Breast classification technology uses BI-RADS classification as the diagnosis basis, integrates scanning, reading and reporting, and can provide a comprehensive and objective evaluation. The AI-SONIC Breast system has high diagnostic efficiency. In this study, the AI automatic detection systems had higher sensitivity, specificity and accuracy than young doctors but lower diagnostic efficiency than the ultrasonic s-detect classification technology for breast nodules reported by Zhou et al[19]. The reason may be that different AI systems have different degrees of machine training. The nodules selected in this study were BI-RADS 4 and above, excluding some simple and typical benign lesions and increasing the difficulty of diagnosis. The accuracy of the AI-SONIC Breast system with conventional ultrasound BI-RADS classification was 85.98%, which is significantly higher than that of the conventional ultrasound BI-RADS classification (74.77%) and indicates that AI automatic detection has high diagnostic efficiency for BI-RADS class 4 breast nodules, e.g., higher than that of young doctors. Its application in the clinic can improve the diagnostic accuracy of young doctors and increase diagnostic confidence. The system can also be used to upgrade and downgrade BI-RADS class 4 nodules and guide decision-making. The adjusted BI-RADS classification decreased the biopsy rate of breast nodules from 100% to 67.29%, which greatly reduced unnecessary puncture biopsy. The cancer detection rate of BI-RADS classification after adjustment was approximately 61.11%, which was significantly higher than that before adjustment (42.99%); this will help to effectively avoid the waste of medical resources. According to the adjusted BI-RADS classification, the risk of malignancy was approximately 1.87%, and only 2 cases of malignant nodules were downgraded to class 3. One case was mucinous carcinoma, and both conventional ultrasound BI-RADS classification and the AI detection system classified this nodule as benign, i.e., a missed diagnosis. The reason is that breast mucinous carcinoma is a special type of malignant breast tumor with a low incidence rate and is often neglected[20], and its growth is inflated. Sonograms of breast mucinous carcinoma mostly reveal hypoechoic nodules with clear borders and regular morphology, and the posterior echo is enhanced. In general, there is no calcification and no obvious blood flow signal. It has similar sonographic features to benign breast tumors, which are easily misdiagnosed as breast fibroadenoma or adenosis[21]. In our study, this nodule was diagnosed as a BI-RADS 4A nodule by conventional ultrasound. The score of the AI detection system was 0.44, which indicated benign. The other case involved intraductal papillary carcinoma; the nodule was small, and the maximum diameter was only 4 mm. Conventional ultrasound showed that the nodule grew nearly vertically, which was consistent with malignancy, and it was diagnosed as a BI-RADS 4A nodule. The AI detection system suggested 0.38, which was benign. In such cases, a missed diagnosis (or misdiagnosis) can be corrected, and a diagnosis and treatment plan can be decided through short-term follow-up or combined with other new technologies, such as breast contrast-enhanced ultrasound[22], ultrasonic elastography[23], automatic breast volume scanner[24] or puncture biopsy.

The limitations of this article are as follows: (1) The sample size was small, the pathological types were incomplete, and there were no special types of breast cancer, such as neuroendocrine carcinoma, medullary carcinoma and Paget’s disease; (2) Conventional ultrasound was performed by two young doctors, and the diagnostic efficacy of different seniority doctors and AI automatic detection systems was not compared; and (3) The AI-SONIC breast system has certain limitations and cannot recognize and determine dynamic ultrasound images. Its feature analysis does not include important information such as the blood flow signal, peripheral echo and elastic characteristics, and there is a certain error in the judgment of equal echo or small nodules.

CONCLUSION

AI automatic detection has high accuracy in determining benign and malignant BI-RADS 4 breast nodules. Conventional ultrasound BI-RADS classification with AI automatic detection can reduce the biopsy rate of BI-RADS 4 breast nodules.

ARTICLE HIGHLIGHTS

Research background

With the popularization of breast screening, an increasing number of BI-RADS 4 nodules have been detected. According to clinical guidelines, such nodules require biopsy. However, the vast majority of BI-RADS 4 nodules are benign, which results in a large number of unnecessary biopsies.

Research motivation

To reduce the biopsy rate for BI-RADS 4 nodules and prevent the waste of medical resources.

Research objectives

Our goal is to improve the preoperative diagnostic accuracy of breast nodules as much as possible, not only to reduce misdiagnosis and missed diagnosis, but also to avoid unnecessary biopsy.

Research methods

We used an artificial intelligence (AI) system to regrade BI-RADS 4 nodules and used pathology results as the gold standard.

Research results

The diagnostic value of AI detection system is higher than that of other methods. The BI-RADS classification results adjusted by AI detection system are closer to the pathological results.

Research conclusions

The AI system has very high diagnostic efficiency for BI-RADS 4 nodules and can effectively prevent many unnecessary puncture biopsies of such nodules.

Research perspectives

In the future, we will continue to study the application of AI in breast cancer and use AI to predict the prognosis of breast cancer.