Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 05.05.20 in Vol 8, No 5 (2020): May

Preprints (earlier versions) of this paper are available at, first published Feb 14, 2020.

This paper is in the following e-collection/theme issue:

    Original Paper

    Evaluation of the Quadri-Planes Method in Computer-Aided Diagnosis of Breast Lesions by Ultrasonography: Prospective Single-Center Study

    The Xiangya Medical School, Central South University, Changsha, Hunan, China

    *these authors contributed equally

    Corresponding Author:

    Ping Zhou, MD

    The Xiangya Medical School

    Central South University

    172 Laodong W Rd

    Tianxin District

    Changsha, Hunan, 410015


    Phone: 86 731 88618403

    Fax:86 731 88618403



    Background: Computer-aided diagnosis (CAD) is a tool that can help radiologists diagnose breast lesions by ultrasonography. Previous studies have demonstrated that CAD can help reduce the incidence of missed diagnoses by radiologists. However, the optimal method to apply CAD to breast lesions using diagnostic planes has not been assessed.

    Objective: The aim of this study was to compare the performance of radiologists with different levels of experience when using CAD with the quadri-planes method to detect breast tumors.

    Methods: From November 2018 to October 2019, we enrolled patients in the study who had a breast mass as their most prominent symptom. We assigned 2 ultrasound radiologists (with 1 and 5 years of experience, respectively) to read breast ultrasonography images without CAD and then to perform a second reading while applying CAD with the quadri-planes method. We then compared the diagnostic performance of the readers for the 2 readings (without and with CAD). The McNemar test for paired data was used for statistical analysis.

    Results: A total of 331 patients were included in this study (mean age 43.88 years, range 17-70, SD 12.10), including 512 lesions (mean diameter 1.85 centimeters, SD 1.19; range 0.26-9.5); 200/512 (39.1%) were malignant, and 312/512 (60.9%) were benign. For CAD, the area under the receiver operating characteristic curve (AUC) improved significantly from 0.76 (95% CI 0.71-0.79) with the cross-planes method to 0.84 (95% CI 0.80-0.88; P<.001) with the quadri-planes method. For the novice reader, the AUC significantly improved from 0.73 (95% CI 0.69-0.78) for the without-CAD mode to 0.83 (95% CI 0.80-0.87; P<.001) for the combined-CAD mode with the quadri-planes method. For the experienced reader, the AUC improved from 0.85 (95% CI 0.81-0.88) to 0.87 (95% CI 0.84-0.91; P=.15). The kappa indicating consistency between the experienced reader and the novice reader for the combined-CAD mode was 0.63. For the novice reader, the sensitivity significantly improved from 60.0% for the without-CAD mode to 79.0% for the combined-CAD mode (P=.004). The specificity, negative predictive value, positive predictive value, and accuracy improved from 84.9% to 87.8% (P=.53), 76.8% to 86.7% (P=.07), 71.9% to 80.6% (P=.13), and 75.2% to 84.4% (P=.12), respectively. For the experienced reader, the sensitivity improved significantly from 76.0% for the without-CAD mode to 87.0% for the combined-CAD mode (P=.045). The NPV and accuracy moderately improved from 85.8% and 86.3% to 91.0% (P=.27) and 87.0% (P=.84), respectively. The specificity and positive predictive value decreased from 87.4% to 81.3% (P=.25) and from 87.2% to 93.0% (P=.16), respectively.

    Conclusions: S-Detect is a feasible diagnostic tool that can improve the sensitivity, accuracy, and AUC of the quadri-planes method for both novice and experienced readers while also improving the specificity for the novice reader. It demonstrates important application value in the clinical diagnosis of breast cancer.

    Trial Registration: 1800019649;

    JMIR Med Inform 2020;8(5):e18251




    Breast cancer is one of the most common cancers in women and the second leading cause of cancer-related mortality worldwide [1,2]. Early diagnosis of breast cancer can increase the treatment options and survival rate of patients [3]; however, early diagnosis depends on accurate and reliable diagnosis using medical imageology. As a convenient modality, breast ultrasonography plays an important role in breast cancer screening. Despite the improvements in ultrasound diagnosis with the application of new technology, dependence on operator experience remains the main limitation of ultrasound-based diagnosis [4,5]. S-Detect is a recently developed computer-aided diagnosis (CAD) system for breast cancer that provides assistance in morphological analysis based on the Breast Imaging Reporting and Data System (BI-RADS) lexicon and classification [6]. Many studies have reported that the S-Detect system has potential to become a novel diagnostic tool for radiologists [7-10].

    In our previous study, the sensitivity was too high in the cross-planes method because it considered the lesion to be malignant if any image of 2 planes indicated malignancy, leading to a decrease in specificity. No study has evaluated the diagnostic performance of CAD in breast lesions with respect to diagnostic planes (cross-plane and quadri-plane methods). Therefore, the purpose of this study was to compare the performance of radiologists with different levels of experience in detecting breast cancer using CAD with the quadri-planes method.


    Patient Selection

    We prospectively enrolled patients in our study from November 2018 to October 2019. All patients underwent grayscale breast ultrasound examination before surgery. All lesions were examined after surgery to confirm their pathological type. This prospective single center study was approved by the Institutional Review Board of Third Xiangya Hospital. Informed consent was obtained from all patients.

    The inclusion criteria were as follows: patients aged 17-70 years with breast tumors requiring surgery. The exclusion criteria were a history of neoadjuvant chemotherapy or endocrine therapy before surgery, lesions punctured by core-needle biopsy or a Mammotome system, equipment of the breast with a prosthesis, unclear lesions as displayed by ultrasound images, and unwillingness to take part in the study.

    Ultrasound Image Acquisition

    All images were obtained with an RS80A ultrasound system (Samsung Medison Co Ltd) with a 5-13 megahertz bandwidth (8.4 MHz center frequency) linear transducer. All ultrasound examinations were performed by an independent radiologist with 5 years of experience. In the cross-planes method, 2 typical images of the tumor in the longitudinal and transverse planes were stored in the ultrasound system; in the quadri-planes method, 2 additional cross-plane images were acquired by rotating the probe 45 degrees around the center of the mass.

    Computer-Aided Diagnostic System

    Our CAD system, S-Detect, extracts features using an integration of artificial neural network classifiers internally installed in the RS80A ultrasound equipment. The sensitivity of the instrument was set to the default. To test the reproducibility of the CAD marks with the same image, we randomly selected 20/512 (3.9%) examinations and passed them through the CAD system 3 times; the results showed that the markings were consistent in all images.

    In S-Detect, the cursor was placed on the identified center of the lesion, and a region of interest was automatically drawn along the border of the mass by the ultrasound system. If the borderline was considered inaccurate in any area of the tumor, it was manually edited to achieve the optimum fitness. The ultrasound image features of the lesion were analyzed according to the BI-RADS lexicon, and the final classifications were automatically performed by the ultrasound system. In the S-Detect system, the final assessment classification was divided into dichotomous results of “possibly benign” or “possibly malignant.”

    Diagnostic Criteria

    According to the fifth version of BI-RADS, the radiologists classified the lesions from category 3 to category 5. BI-RADS category 4 was further subdivided into categories 4A, 4B, and 4C. Category 3 is considered probably benign (<2% likelihood of malignancy), and categories 4A, 4B, and 4C range from low to high suspicion (2%-10%, 10%-50%, and 50%-95% likelihood of malignancy, respectively). Category 5 indicates a high malignancy rate (>95% likelihood of malignancy). Malignant signs in breast ultrasound imaging include irregular shape, antiparallel orientation, noncircumscribed margin, microcalcification, acoustic halo, posterior shadowing, and abnormalities of the surrounding tissue. Lesions with no definitive malignant sign were assigned to category 3; lesions with 1, 2, and 3 malignant signs were assigned to categories 4A, 4B, and 4C, respectively; and lesions with more than 4 malignant signs were assigned to category 5. Accordingly, category 3 and 4A lesions were regarded as benign, and category 4B, 4C, and 5 lesions were regarded as malignant [11,12].

    To assess the combination of ultrasound and the CAD system, we acquired images of the longitudinal and transverse planes of the tumor for CAD with the cross-planes method. If 1 plane indicated “possibly malignant,” the outcome was considered positive, and the BI-RADS category diagnosis was increased by 1 level (ie, 3 to 4A, 4A to 4B, 4B to 4C, 4C to 5). If both planes indicated “possibly benign,” the outcome was considered negative, and the BI-RADS category diagnosis was decreased by 1 level (ie, 5 to 4C, 4C to 4B, 4B to 4A, 4A to 3) [13]. For the quadri-planes method, if any 2 planes indicated “possibly malignant,” the outcome was considered positive, and the BI-RADS category diagnosis was increased by 1 level. If all 4 planes indicated “possibly benign,” the outcome was considered negative, and the BI-RADS category diagnosis was decreased by 1 level.

    Readers, Reading Modes, and Training

    The study included 2 readers: a novice reader with 1 year of ultrasound experience and an experienced reader with 5 years of ultrasound experience. Both readers were trained in the reading procedures with 20 ultrasound images (from the 512 examinations) that were not part of the study set, 10 of which were read without using CAD (without-CAD mode). The readers assessed the other 10 images in combined-CAD mode with the cross-planes method and the quadri-planes method; the readers first read the ultrasound images without using CAD and then mechanically combined the indications of the CAD marks to make the final decision.

    Both readers performed every examination in each reading mode independently and were blinded to any information about the patients, including age, manifestation of symptoms, and previous radiology reports. The readers were asked to read for at least 2 hours per day to simulate the typical process of batch reading in such examinations (Figure 1).

    Figure 1. The study design and workflow. CAD: computer-aided diagnosis.
    View this figure

    Statistical Analysis

    Statistical evaluation was performed using SPSS software version 19.0 (IBM Corporation). Taking the pathology results as the gold standard, we analyzed the diagnostic sensitivity, specificity, and area under the receiver operating characteristic (ROC) curve (AUC) in without-CAD mode and combined-CAD mode (with the quadri-planes method). The confirmatory diagnosis was defined as the diagnosis made on the basis of pathology. The diagnostic parameters of the combined-CAD mode and without-CAD mode were compared using the McNemar test for sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy for match-paired data. We used the Hanley and McNeil method to analyze the differences between pairs of AUCs. The number of malignant planes of each tumor was recorded with the quadri-planes method, and the ROC curves were drawn based on the pathological results to determine the cutoff value based on the maximum Youden index. The degree of agreement between the experienced reader in without-CAD mode and the novice reader in combined-CAD mode was analyzed using kappa statistics. The criteria for the kappa values were poor ≤0.2, fair 0.21-0.4, moderate 0.41-0.6, good 0.61-0.8, and perfect 0.81-1 [14]. For all tests mentioned, P<.05 was considered to indicate statistical significance.


    Characteristics of the Patients and Lesions

    The patient demographics and lesion characteristics are summarized in Table 1. A total of 331 patients who presented with 512 lesions were included in this study. The mean age of the examined patients was 43.88 years, range 17-70 (SD 12.10). The diameters of the lesions ranged from 0.26-9.50 centimeters, mean 1.85 (SD 1.19). Among the 512 breast lesions, 200/512 (39.1%) were malignant and 312/512 (61.9%) were benign. The mean sizes of all lesions were similar and were close to 2 cm; benign lesions were the smallest (1.82 cm) and malignant lesions were the largest (2.28 cm).

    Table 1. Patient demographics (N=331) and lesion characteristics (N=512).
    View this table

    Reader Performance

    The diagnostic performance of CAD and of the novice and experienced readers in comparison with the pathological diagnoses is depicted in Table 2.

    The statistical evaluation of the performance of the CAD system and of the readers is shown in Table 2. For CAD, the AUC improved significantly between the cross-planes method and the quadri-planes method (Z=4.42, P<.001). The cutoff value of the positive planes in the quadri-planes method was 2.5 based on the Youden index of 0.68. Considering that breast cancer often demonstrates invasive characteristics and has relatively poor prognosis [15], we set the threshold to any 2 positive planes of 4 images.

    For the novice reader, the improvement in the AUCs was significant between the without-CAD mode and combined-CAD mode with the quadri-planes method (Z=5.55, P<.001). However, there was no significant difference in the AUCs for the without-CAD and combined-CAD modes for the experienced reader (Z=1.44, P=.15; Table 3, Figure 2). The kappa indicating consistency between the experienced reader in without-CAD mode and the novice reader in combined-CAD mode was 0.63.

    When a BI-RADS category 4A threshold was used, in contrast to CAD with the cross-planes method, significant improvements in specificity (P<.001), PPV (P=.01), and accuracy (P=.03) were observed for the quadri-planes method; however, there was no significant difference in NPV, and the sensitivity decreased. The sensitivity, NPV, and accuracy improved in the combined-CAD mode compared with the without-CAD mode for both readers (Table 3). Among these, the sensitivity improved significantly between the 2 reading modes for both the novice reader (P=.004) and the experienced reader (P=.045), whereas the accuracy improved significantly only for the novice reader. Moreover, there were no significant differences between modes for either reader with respect to specificity, PPV, or NPV.

    Table 2. Diagnostic performance of the computer-aided diagnosis system and the novice and experienced readers in the 2 reading modes with the Breast Imaging Reporting and Data System Category 4A threshold. The pathological diagnosis was considered to be the gold standard.
    View this table
    Table 3. Statistical evaluation of the performance of the computer-aided diagnosis system and the 2 readers with P values indicting differences between various groups.
    View this table
    Figure 2. Receiver operating characteristic curves for the CAD method and the readers. (A) CAD with the quadri-plane and cross-plane methods (if either plane was malignant, the result was regarded as positive); CAD with the quadri-planes method with 2 planes as the threshold (if ≥2 planes were malignant, the result was regarded as positive); the novice reader without CAD; the experienced reader without CAD. (B) The novice reader without CAD; the experienced reader without CAD; the novice reader combined with CAD with the quadri-planes method; the experienced reader combined with CAD with the quadri-planes method. AUC: area under the curve; CAD: computer-aided diagnosis.
    View this figure

    Management of Diagnostic Decision Changes

    In contrast to the mode without CAD, changes in the diagnostic decision with combined-CAD mode were moderately more common for the novice reader than for the experienced reader (115/512, 22.5% vs 70/512, 13.7%; P=.09). The proportions of malignancy lesions correctly upgraded from category 4A to 4B by the novice reader and experienced reader were similar (44/115, 38% vs 27/70, 39%; P=.88). However, the proportion of benign lesions correctly downgraded from category 4B to 4A by the novice reader was higher than that by experienced reader, with a very close to significant statistical difference (37/115, 32% vs 10/70, 14%; P=.05). The proportions of malignancy lesions incorrectly downgraded from category 4B to 4A by both readers were similar (6/115, 5% vs 5/70, 7%). In addition, the proportion of benign lesions incorrectly upgraded by the novice reader was lower than that by the experienced reader (28/115, 24% vs 28/70, 40%). The management decision changes of the 2 readers are provided in Table 4.

    The kappa indicating consistency between the experienced reader and the novice reader with combined-CAD mode was slightly higher than that for without-CAD mode (0.63 vs 0.57). These results are provided in Table 5.

    Table 4. Management decision changes by the 2 readers using CAD with the quadri-planes method.
    View this table
    Table 5. Comparisons of consistency between the experienced reader in without-CAD mode and the novice reader in without-CAD mode and combined-CAD mode with the quadri-planes method.
    View this table


    Principal Findings

    In our study, the AUCs of CAD with the quadri-planes method were significantly higher than those of CAD with the cross-planes method (P<.001); even when we chose any 2 malignant planes as the threshold, the AUC of the quadri-planes method was still higher than that of the cross-planes method (P<.001). The sensitivity, accuracy, and AUC improved for both the novice and experienced readers using combined-CAD mode with the quadri-planes method. Additionally, compared to without-CAD mode, the consistency level improved from fair to good between the novice reader in combined-CAD mode and the experienced reader in without-CAD mode.

    Choi et al [10] recently reported that the specificity and AUC of both experienced and inexperienced readers improved using a CAD system combined with S-Detect; moreover, the sensitivity of the inexperienced reader improved significantly. Although the diagnosis performance of the readers improved in Choi’s study, the sensitivity of the readers combined with CAD was not satisfactory for detecting breast cancer (66.7% and 75.0%, respectively). These results may have been obtained because the proportion of malignant lesions was too low (6%); moreover, the data in that study were derived from a small number of patients. All these factors can lead to increased false negative results. According to our previous study [16], high sensitivity is a remarkable characteristic of S-Detect in the cross-planes method; this is similar to some previously published studies, where the sensitivity of the ultrasound CAD system was reported to be high (between 88.9% and 100%) [17,18].

    It is known that high sensitivity in diagnostic performance can lead to unnecessary breast biopsies and increased medical costs borne by patients; therefore, we developed the quadri-planes method to address this problem. CAD in the quadri-planes method resulted in both improved sensitivity (60.0% to 79.0%) and specificity (84.9% to 87.8%) for the novice reader; in addition, there was no statistically significant change in specificity for the experienced reader (93.0% to 87.2%), while the sensitivity improved significantly (76.0% to 87.0%). This indicates that CAD with the quadri-planes method can improve the sensitivity and specificity of the results reported by readers, especially less experienced readers.

    In our investigation, the specificity, accuracy, and AUC of CAD with the quadri-planes method were all higher than those of CAD with the cross-planes method, although the sensitivity of the quadri-plane method was slightly lower. This is likely because the quadri-planes method is based on the cross-planes method; therefore, 2 of the 4 planes in the quadri-planes method were the same as those in the cross-planes method. In addition, the threshold in the quadri-planes method for dichotomization of the final CAD assessment was set at any 2 of 4 positive planes; however, the threshold in the cross-planes method was set as any 1 of 2 positive planes. This may have led to the increase in specificity and the decrease in sensitivity of the quadri-planes method compared to the cross-planes method.

    When the assessments differed in category 4A and 4B between the readers and the CAD, the proportion of correct adjustments using CAD for the inexperienced reader was higher than that for the experienced reader (81/115, 70% vs 37/70, 53%). This indicates that the less experienced reader obtained more benefit from the CAD system; this is related to the fact that combining CAD with the quadri-planes method led to improvements in sensitivity, specificity, and accuracy for the inexperienced reader, while the accuracy between the quadri-planes method and the experienced reader was similar (83.8% vs 86.3%). For CAD with the quadri-planes method, the consistency between the experienced reader and novice reader was good. As such, CAD assistance with the quadri-planes method can not only improve diagnostic performance but can also be expected to play a more weighted role in providing a second opinion, especially for less experienced readers. Consequently, this system can reduce misdiagnosis by less experienced readers in addition to reducing variability in readers’ interpretations and overcoming the effects of inexperience. These improvements in diagnostic performance by combining CAD and ultrasound may reduce both the misdiagnosis and missed diagnosis ratios of breast cancer by readers with different experience levels.

    Several reports have described applying different types of CAD to breast ultrasound [6,19-21]. These studies all reported that the CAD systems enhanced the diagnostic performance of breast ultrasound, especially specificity and accuracy. Shen et al [20] argued that CAD systems can be helpful in evaluating fuzzy category 4 lesions. Wang et al [21] suggested that combining CAD is more helpful for inexperienced readers than experienced ones, with greater improvement in the diagnostic performance in the inexperienced group. Kim’s study involved 2 staff radiologists with 7 and 19 years of experience, respectively; both these readers can be described as experienced, so their false positive rates were low and their false negative rates were relatively high. In addition, the surgery proportion was only 27.6% and the core needle biopsy proportion was 61.5%, which may also have affected the results. The retrospective analysis was performed by only 1 radiologist with 7 years of ultrasound experience. In Wang’s study, the CAD system was relatively old, and the experience between the 8 readers varied, which may have led to increases in false negative and false positive rates. Therefore, the results of the above studies show that traditional CAD methods are not sufficient to balance the sensitivity and specificity to effectively reduce false negative or false positive results. In our study, the sensitivity, NPV, and accuracy of both readers improved; this supports the idea that S-Detect can reliably provide a second view that can be referred to by readers. Although the CAD methods were not exactly the same as in previous studies [17,18], high sensitivity balanced with specificity is a remarkable superiority of the quadri-planes method. Instead, the proportion of the benign lesions in our study was lower (Table 1), and the mean size of the lesions was larger; also, all the patients had breast masses as their prominent symptom, which may lead to differences between the results as in the study by Wu et al [22]. According to the BI-RADS criteria [23], we subdivided category 4 into categories 4A, 4B, and 4C, and the threshold was set to category 4A in grayscale ultrasound; consequently, the specificity was high and the sensitivity was relatively low. Moreover, S-Detect provides the final assessment in a dichotomized form of possibly benign and possibly malignant; we consider that these factors may also affect the diagnostic capability of readers combined with CAD.

    Strengths and Prospects

    The results of our study are encouraging for daily clinical breast cancer screening practiced by readers, although some pathology subtypes of breast cancer had better outcomes in situ [24]. However, breast cancer is still a relatively aggressive disease that possesses higher rates of metastases and poorer survival rates [25]. Thus, it is important to detect breast cancer accurately in early stages to reduce its mortality rate [26]. Additionally, S-Detect is a concise and user-friendly program that is integrated in the ultrasound machine; the quadri-planes method enables the reader to immediately achieve a more precise result during real-time ultrasonography, which can easily be applied to routine work (Figure 3). However, it is not recommended to apply CAD alone or as a replacement for a human reader in the diagnosis of breast lesions at present, as shown in Kim’s study [6] (Figure 4). However, there is reason to believe that this will be possible in the near future. Further investigation with technical advances can be anticipated to develop a more sophisticated algorithm using the multiple plane assessment BI-RADS ultrasonographic categories.

    Ultrasound scanning is a real-time and multi-angle inspection process; a lesion can be observed from different planes to collect imaging features such as the internal situation, the relationship of the lesion with its surroundings, the blood supply model, and patient histories. Obviously, ultrasound can obtain more image data and clinical information than CAD. The quadri-planes method with CAD can extract more features from a tumor with maximum objectivity; combined with the expertise of a reader, the weaknesses of each method can be counteracted by the strengths of the others, which can assist readers in making more accurate diagnoses regardless of their experience.

    Figure 3. A breast lesion assessed by CAD with the cross-planes method (A, B) and the quadri-planes method (A-D).
    View this figure
    Figure 4. Example of a ductal carcinoma in situ lesion with a size of 1.90×1.10 centimeters showing a clear margin, regular shape, and microcalcification that was incorrectly diagnosed as benign by S-Detect with the cross-planes (A, B) and quadri-planes (A-D) methods. The lesion was classified as BI-RADS category 4B by the readers.
    View this figure


    There are several limitations of this study. First, the number of cases in the single center study was relatively small. Second, the presentation of calcifications is still not included in the current version of S-Detect due to its limited analysis ability for microcalcifications [27]. Third, some small nodules classified as BI-RADS category 2 or 3 with sizes of around 1 cm without surgical operation were not included in this study, which may have affected the results. Fourth, the number of planes of the lesions for CAD was set to 4. It can be argued that it would have been better to study additional planes. Fifth, both of the readers were relatively inexperienced breast scan readers. In China, the specialty of breast imaging is new, and its staff are young compared with those in other imaging specialties. These factors may have affected the results.


    S-Detect is a feasible diagnostic tool that can improve the sensitivity, accuracy, and AUC of both novice and experienced readers in the quadri-planes method while also improving the specificity for the novice reader; thus, it demonstrates important application value in the clinical diagnosis of breast cancer.


    This study was supported by the National Natural Science Foundation of China (81871367).

    Conflicts of Interest

    None declared.


    1. Siegel RL, Miller KD, Jemal A. Cancer statistics, 2019. CA Cancer J Clin 2019 Jan;69(1):7-34 [FREE Full text] [CrossRef] [Medline]
    2. Chen W, Zheng R, Baade PD, Zhang S, Zeng H, Bray F, et al. Cancer statistics in China, 2015. CA Cancer J Clin 2016;66(2):115-132 [FREE Full text] [CrossRef] [Medline]
    3. Brinkley D, Haybrittle JL. The curability of breast cancer. Lancet 1975 Jul 19;2(7925):95-97 [FREE Full text] [CrossRef] [Medline]
    4. Takahashi R, Kajikawa Y. Computer-aided diagnosis: A survey with bibliometric analysis. Int J Med Inform 2017 May;101:58-67 [FREE Full text] [CrossRef] [Medline]
    5. Cho E, Kim E, Song MK, Yoon JH. Application of Computer-Aided Diagnosis on Breast Ultrasonography: Evaluation of Diagnostic Performances and Agreement of Radiologists According to Different Levels of Experience. J Ultrasound Med 2018 Jan;37(1):209-216 [FREE Full text] [CrossRef] [Medline]
    6. Kim K, Song MK, Kim E, Yoon JH. Clinical application of S-Detect to breast masses on ultrasonography: a study evaluating the diagnostic performance and agreement with a dedicated breast radiologist. Ultrasonography 2017 Jan;36(1):3-9 [FREE Full text] [CrossRef] [Medline]
    7. Horsch K, Giger ML, Vyborny CJ, Venta LA. Performance of computer-aided diagnosis in the interpretation of lesions on breast sonography. Acad Radiol 2004 Mar;11(3):272-280. [CrossRef] [Medline]
    8. Komeda Y, Handa H, Watanabe T, Nomura T, Kitahashi M, Sakurai T, et al. Computer-Aided Diagnosis Based on Convolutional Neural Network System for Colorectal Polyp Classification: Preliminary Experience. Oncology 2017;93 Suppl 1:30-34 [FREE Full text] [CrossRef] [Medline]
    9. Zhao C, Xiao M, Jiang Y, Liu H, Wang M, Wang H, et al. Feasibility of computer-assisted diagnosis for breast ultrasound: the results of the diagnostic performance of S-detect from a single center in China. Cancer Manag Res 2019;11:921-930 [FREE Full text] [CrossRef] [Medline]
    10. Choi J, Kang BJ, Baek JE, Lee HS, Kim SH. Application of computer-aided diagnosis in breast ultrasound interpretation: improvements in diagnostic performance according to reader experience. Ultrasonography 2018 Jul;37(3):217-225 [FREE Full text] [CrossRef] [Medline]
    11. Migowski A. [Early detection of breast cancer and the interpretation of results of survival studies]. Cien Saude Colet 2015 Apr;20(4):1309 [FREE Full text] [CrossRef] [Medline]
    12. Guo R, Lu G, Qin B, Fei B. Ultrasound Imaging Technologies for Breast Cancer Detection and Management: A Review. Ultrasound Med Biol 2018 Jan;44(1):37-70 [FREE Full text] [CrossRef] [Medline]
    13. Wang M, Yang Z, Liu C, Yan J, Zhang W, Sun J, et al. Differential Diagnosis of Breast Category 3 and 4 Nodules Through BI-RADS Classification in Conjunction with Shear Wave Elastography. Ultrasound Med Biol 2017 Mar;43(3):601-606 [FREE Full text] [CrossRef] [Medline]
    14. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977 Mar;33(1):159-174. [Medline]
    15. Waks AG, Winer EP. Breast Cancer Treatment. JAMA 2019 Jan 22;321(3):316. [CrossRef] [Medline]
    16. Yongping L, Zhou P, Juan Z, Yongfeng Z, Liu W, Shi Y. Performance of Computer-Aided Diagnosis in Ultrasonography for Detection of Breast Lesions Less and More Than 2 cm: Prospective Comparative Study. JMIR Med Inform 2020 Mar 02;8(3):e16334 [FREE Full text] [CrossRef] [Medline]
    17. Chabi M, Borget I, Ardiles R, Aboud G, Boussouar S, Vilar V, et al. Evaluation of the accuracy of a computer-aided diagnosis (CAD) system in breast ultrasound according to the radiologist's experience. Acad Radiol 2012 Mar;19(3):311-319. [CrossRef] [Medline]
    18. Morra L, Sacchetto D, Durando M, Agliozzo S, Carbonaro LA, Delsanto S, et al. Breast Cancer: Computer-aided Detection with Digital Breast Tomosynthesis. Radiology 2015 Oct;277(1):56-63. [CrossRef] [Medline]
    19. Xiao M, Zhao C, Zhu Q, Zhang J, Liu H, Li J, et al. An investigation of the classification accuracy of a deep learning framework-based computer-aided diagnosis system in different pathological types of breast lesions. J Thorac Dis 2019 Dec;11(12):5023-5031 [FREE Full text] [CrossRef] [Medline]
    20. Shen W, Chang R, Moon WK. Computer aided classification system for breast ultrasound based on Breast Imaging Reporting and Data System (BI-RADS). Ultrasound Med Biol 2007 Nov;33(11):1688-1698. [CrossRef] [Medline]
    21. Wang Y, Jiang S, Wang H, Guo YH, Liu B, Hou Y, et al. CAD algorithms for solid breast masses discrimination: evaluation of the accuracy and interobserver variability. Ultrasound Med Biol 2010 Aug;36(8):1273-1281. [CrossRef] [Medline]
    22. Wu J, Zhao Z, Zhang W, Liang M, Ou B, Yang H, et al. Computer-Aided Diagnosis of Solid Breast Lesions With Ultrasound: Factors Associated With False-negative and False-positive Results. J Ultrasound Med 2019 Dec;38(12):3193-3202 [FREE Full text] [CrossRef] [Medline]
    23. Rao AA, Feneis J, Lalonde C, Ojeda-Fournier H. A Pictorial Review of Changes in the BI-RADS Fifth Edition. Radiographics 2016;36(3):623-639 [FREE Full text] [CrossRef] [Medline]
    24. Ward EM, DeSantis CE, Lin CC, Kramer JL, Jemal A, Kohler B, et al. Cancer statistics: Breast cancer in situ. CA Cancer J Clin 2015;65(6):481-495 [FREE Full text] [CrossRef] [Medline]
    25. Harbeck N, Gnant M. Breast cancer. Lancet 2017 Mar 18;389(10074):1134-1150. [CrossRef] [Medline]
    26. Wang Y, Fan W, Zhao S, Zhang K, Zhang L, Zhang P, et al. Qualitative, quantitative and combination score systems in differential diagnosis of breast lesions by contrast-enhanced ultrasound. Eur J Radiol 2016 Jan;85(1):48-54. [CrossRef] [Medline]
    27. Sickles EA. Breast calcifications: mammographic evaluation. Radiology 1986 Aug;160(2):289-293. [CrossRef] [Medline]


    AUC: area under the receiving operating characteristic curve
    BI-RADS: Breast Imaging Reporting and Data System
    CAD: computer-aided diagnosis
    NPV: negative predictive value
    PPV: positive predictive value
    ROC: receiver operating characteristic

    Edited by G Eysenbach; submitted 14.02.20; peer-reviewed by J Zhang, K Karanam, D Di Stasio, T Muto; comments to author 10.03.20; revised version received 16.03.20; accepted 10.04.20; published 05.05.20

    ©Liang Yongping, Zhang Juan, Ping Zhou, Zhao Yongfeng, Wengang Liu, Yifan Shi. Originally published in JMIR Medical Informatics (, 05.05.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.