BREAST COMPANION SOFTWARE SYSTEM
Device Facts
| Record ID | P100007 |
|---|---|
| Device Name | BREAST COMPANION SOFTWARE SYSTEM |
| Applicant | Almen Laboratories, Inc. |
| Product Code | MYN · Radiology |
| Decision Date | Feb 10, 2012 |
| Decision | APWD |
| Regulation | 21 CFR 892.2070 |
| Device Class | Class 2 |
| Attributes | Software as a Medical Device, Real-World Evidence |
Real-World Evidence
| Submission | Device | Sponsor | RWD Sources | RWE Use Summary | Key Tags |
|---|---|---|---|---|---|
| P100007 · Feb 10, 2012 | BREAST COMPANION SOFTWARE SYSTEM | Almen Laboratories, Inc. | PACS/RIS clinical database; Retrospective medical records; Pathology reports; Clinical follow-up records | Retrospective clinical cases were used to validate the performance of the Breast Companion® software. The study compared radiologist performance 'with' and 'without' the device using a cohort of 596 archived, de-identified clinical ultrasound studies with confirmed pathology or two-year negative follow-up. | Retrospective cohort; PACS-based database; Clinical validation; Diagnostic ultrasound |
Clinical Evidence
| Study Design | Population | Comparator | Key Endpoints |
|---|---|---|---|
| Clinical Validation Study; Retrospective MRMC (Multiple Reader/Multiple Cases) study; Follow-up/Duration: 3 years (2006-2009); Study Period: 2006-2009 | Women referred for diagnostic breast ultrasound workup or biopsy; Sample Size: 596 cases; Number of Sites: 2 (Thomas Jefferson University Hospital and UCSD School of Medicine) | Radiologist performance without the Breast Companion® software | ROC AUC, Sensitivity, Specificity |
AI Performance
| Output | Algorithm | Acceptance | Observed | Dev DS | Dev Readers | Test DS | Test Readers |
|---|---|---|---|---|---|---|---|
| Breast Ultrasound Lesion Assessment (standalone) | Case-based reasoning approach | — | ROC AUC 98.2% (empirical), 98.6% (fitted) | — | — | Validation cohort: 596 archived breast ultrasound image cases. | — |
| Breast Ultrasound Lesion Assessment (reader improvement) | Case-based reasoning approach | — | Mean ROC AUC increased from 83.12% (without) to 87.26% (with) | — | — | Clinical Validation Study: 596 confirmed breast ultrasound cases. | 4 (sub-specialty breast imaging radiologists) |
Indications for Use
Breast Companion® is a computer-aided system intended for improving the ACR BI-RADS® assessment of ultrasound images of lesions of the female breast as part of the diagnostic workup.
Device Story
Breast Companion (BC) is an adjunctive software tool for radiologists performing breast ultrasound diagnostic workups. It integrates with the BI-RADS Companion system to assist in ACR BI-RADS assessment. The device takes DICOM ultrasound images as input; the radiologist segments the lesion of interest. The system uses a case-based reasoning algorithm to search a reference library for the seven most similar lesions; it computes a Computer-aided Lesion Assessment (CLA) score based on a weighted average of the BI-RADS scores of these similar cases. The output is displayed to the radiologist, who may incorporate the score into their final BI-RADS report. Used in clinical settings, the device aims to improve diagnostic accuracy and specificity by providing comparative visual and quantitative decision support, potentially reducing unnecessary biopsies for nonmalignant lesions.
Clinical Evidence
Clinical validation involved a retrospective MRMC study of 596 breast ultrasound cases (22.3% malignant) read by four sub-specialty radiologists. Primary endpoints were ROC-AUC, sensitivity, and specificity. Results showed a statistically significant improvement in mean ROC-AUC (83.12% without vs. 87.26% with BC, p=0.0112) and specificity (41.47% without vs. 56.37% with BC) when using the device. Sensitivity remained statistically equivalent. Standalone performance testing on the same cohort yielded an AUC of 98.6%.
Technological Characteristics
Software-based diagnostic tool; case-based reasoning algorithm; operates on DICOM ultrasound images; integrates with existing PACS/RIS workflows; standalone or networked deployment within hospital firewall; utilizes BI-RADS reporting standards.
Indications for Use
Indicated for use by radiologists as an adjunctive tool to support the diagnostic workup of breast lesions in female patients, following detection or identification via screening mammography or physical examination and subsequent ultrasound imaging.
Regulatory Classification
Identification
Medical image analyzers, including computer-assisted/aided detection (CADe) devices for mammography breast cancer, ultrasound breast lesions, radiograph lung nodules, and radiograph dental caries detection, is a prescription device that is intended to identify, mark, highlight, or in any other manner direct the clinicians' attention to portions of a radiology image that may reveal abnormalities during interpretation of patient radiology images by the clinicians. This device incorporates pattern recognition and data analysis capabilities and operates on previously acquired medical images. This device is not intended to replace the review by a qualified radiologist, and is not intended to be used for triage, or to recommend diagnosis.
Special Controls
*Classification.* Class II (special controls). The special controls for this device are:(1) Design verification and validation must include: (i) A detailed description of the image analysis algorithms including a description of the algorithm inputs and outputs, each major component or block, and algorithm limitations. (ii) A detailed description of pre-specified performance testing methods and dataset(s) used to assess whether the device will improve reader performance as intended and to characterize the standalone device performance. Performance testing includes one or more standalone tests, side-by-side comparisons, or a reader study, as applicable. (iii) Results from performance testing that demonstrate that the device improves reader performance in the intended use population when used in accordance with the instructions for use. The performance assessment must be based on appropriate diagnostic accuracy measures ( *e.g.,* receiver operator characteristic plot, sensitivity, specificity, predictive value, and diagnostic likelihood ratio). The test dataset must contain a sufficient number of cases from important cohorts (*e.g.,* subsets defined by clinically relevant confounders, effect modifiers, concomitant diseases, and subsets defined by image acquisition characteristics) such that the performance estimates and confidence intervals of the device for these individual subsets can be characterized for the intended use population and imaging equipment.(iv) Appropriate software documentation ( *e.g.,* device hazard analysis; software requirements specification document; software design specification document; traceability analysis; description of verification and validation activities including system level test protocol, pass/fail criteria, and results; and cybersecurity).(2) Labeling must include the following: (i) A detailed description of the patient population for which the device is indicated for use. (ii) A detailed description of the intended reading protocol. (iii) A detailed description of the intended user and user training that addresses appropriate reading protocols for the device. (iv) A detailed description of the device inputs and outputs. (v) A detailed description of compatible imaging hardware and imaging protocols. (vi) Discussion of warnings, precautions, and limitations must include situations in which the device may fail or may not operate at its expected performance level ( *e.g.,* poor image quality or for certain subpopulations), as applicable.(vii) Device operating instructions. (viii) A detailed summary of the performance testing, including: test methods, dataset characteristics, results, and a summary of sub-analyses on case distributions stratified by relevant confounders, such as lesion and organ characteristics, disease stages, and imaging equipment.
Predicate Devices
- BI-RADS Companion (K072258)
Submission Summary (Full Text)
{0}
SUMMARY OF SAFETY AND EFFECTIVENESS DATA (SSED)
I. GENERAL INFORMATION
Device Generic Name: Systems, Image Processing, Radiological
Device Trade Name: Breast Companion® Software System
Applicant's Name and Address: Almen Laboratories, Inc.
1672 Gil Way
Vista, CA 92084
Date(s) of Panel Recommendation: None
Premarket Approval Application (PMA) Number: P100007
Product Code: MYN
Date of FDA Notice of Approval: February 10, 2012
Expedited: Not Applicable
II. INDICATIONS FOR USE
Breast Companion® is a computer-aided system intended for improving the ACR BI-RADS® assessment of ultrasound images of lesions of the female breast as part of the diagnostic workup.
III. CONTRAINDICATIONS
There are no contraindications for the use of this device.
IV. WARNINGS AND PRECAUTIONS
Warnings and Precautions for users of the device are stated in the product labeling.
V. DEVICE DESCRIPTION
Breast Companion® (BC) is a BI-RADS Companion™ (K072258) with an added optional scoring function (Computer-aided Lesion Assessment, or CLA). It is an adjunctive tool to be used in support of radiologist readings and is intended to be used in the diagnostic breast work-up process.
BC includes a report-generating function that is fully compliant with the ACR BI-RADS® reporting system (American College of Radiology, Breast Imaging Reporting
PMA P100007: FDA Summary of Safety and Effectiveness Data
{1}
and Data System). BC CLA is optionally used by a Radiologist who is completing an ACR BI-RADS® ultrasound assessment report of a lesion that was detected or identified in a previous and independent procedure using a different modality such as screening mammography or physical examination and then imaged with ultrasound. This ACR BI-RADS® report is a record of the sonographic assessment of the previously detected lesion and incorporates recommendations by the radiologist for further action within the overall diagnostic process. BC CLA is a computer-aided function that was developed using a case-based reasoning approach for breast ultrasound image understanding operating with the physician in the loop.
The two components are related as indicated in Figure 1 below.
Figure 1. Breast Companion® upgrade to a BI-RADS Companion™ (K072258) installation in a typical user scenario
The BC CLA function is utilized by the radiologist who opens a breast ultrasound study using the BI-RADS Companion™ and reviews the breast ultrasound lesion images in the study. The implemented flow of BC functionality, as shown in Figure 2, working within BI-RADS Companion™ can be summarized as follows:
1. User inputs DICOM images
2. User selects one or more lesion(s) for further assessment.
3. User adjusts measurements and/or lesion border.
4. User Completes BI-RADS Classification Form
PMA P100007: FDA Summary of Safety and Effectiveness Data
{2}
5. User Optionally selects BC CLA to compute CLA Score or display similar cases from the BC Reference Library
6. User (optional) annotates his/her opinion on the lesion assessment.
7. User generates BI-RADS Report
8. User stores the assessed case in his/her personal Teaching File (optional)
Figure 2. Overall flow diagram of the BC Computer-aided Lesion Assessment Functionality
At radiologist’s discretion, the user can review all image (DICOM) views in the study, select specific image view(s) for review, select one or more lesion views to use in the BI-RADS Assessment, and select or segment the area or areas on which to make the diagnostic decision. The radiologist is able to modify and review the computer-generated lesion border or re-define a lesion with manual segmentation by cursor outline. The radiologist proceeds with the ACR BI-RADS® Classification Form and selects the appropriate descriptors for the assessment of the lesion. Before finalizing the assessment Category, the radiologist may optionally use the BC CLA Scoring Process, as illustrated
PMA P100007: FDA Summary of Safety and Effectiveness Data
{3}
in Figure 3, which retrieves the most similar lesion images from the Reference Library and compute the CLA score. The radiologist completes lesion assessment and may or may not include the CLA Score in the generated BI-RADS compliant Report.
The BC CLA scoring method as illustrated in Figure 3 is outlined as follows:
1. Input DICOM image with User outlined lesion for further assessment
2. Search RL and retrieve the seven most similar lesions, based on “Relative Similarity,” and display them next to the lesion of interest
3. Compute BC CLA Score, where CLA Score is a weighted average of the numerical BI-RADS scores assigned to the seven most similar cases.
Figure 3. Low level Computer-aided Lesion Assessment Score Process
## VI. ALTERNATIVE PRACTICES AND PROCEDURES
Currently, there is no alternative to this computer-aided diagnostic tool to assist radiologists in the BI-RADS Assessment for ultrasound breast diagnostic work-up.
## VII. MARKETING HISTORY
The Breast Companion® Software System has not been marketed in the United States or any foreign country.
PMA P100007: FDA Summary of Safety and Effectiveness Data
{4}
PMA P100007: FDA Summary of Safety and Effectiveness Data
page 5
# VIII. POTENTIAL ADVERSE EFFECTS OF THE DEVICE ON HEALTH
There are no known direct risks to safety or health caused by, or related to, the use of the device. Indirect risks are that the device may mark some nonmalignant lesions (false positive readings). If a physician determines that a false positive mark indicate an area that is suspicious enough for follow-up, then the patient may be subjected to unnecessary concern and/or biopsy. There are no false negative where BI-RADS 3, 4, and 5 lesions were downgraded to BI-RADS2.
# IX. SUMMARY OF PRECLINICAL STUDIES
There are no non-clinical studies to report for this device.
# X. SUMMARY OF PRIMARY CLINICAL STUDIES
## A. Study Design
### Study History
There were multiple research and subsequent clinical validation studies conducted to evaluate the safety and effectiveness of the Breast Companion® Software System. The studies are published in detail in various scientific publications (including multiple peer reviewed) and results were reported and were presented at SPIE 2001-2004, RSNA 2003-2011, AAPM 2003, 2009-2011, AIUM 2008-2011, AI 26-30.¹⁻⁸ R&D stages of the studies were supported in part by multiple NIH/NCI, SBIR and 3 private research grants.
### Contributors & Collaborators
Two selected institution participated in the clinical validation of the software included Thomas Jefferson University Hospital (including the Breast Imaging Center of TJU) in Philadelphia, PA and the UCSD School of Medicine (lead site) in San Diego, CA. All clinical data, clinical validation and clinical analyses of the results were developed, collected, processed, and analyzed in those institutions under their respective IRB protocols. The readers represented a variety of experience levels in sub-specialty from minimal to extensive. The readers assigned to the Validation Study were randomly selected by the validation sites.
### 1. Clinical Inclusion and Exclusion Criteria
#### Inclusion Criteria
Study inclusion criteria were:
- Chronologically consecutive files, with the search starting 5/4/2003
- Known (confirmed) findings from PACS and/or RIS
- At least 2 acceptable views of each mass
{5}
- Images free of graphic overlays, calipers or other markers
- Minimal artifacts, no foreign objects
- Conclusive pathology results from biopsy or two-year negative follow-up
## Exclusion Criteria
Study exclusion criteria were:
- “No findings” for a selected file, incomplete data, or “known cancer” (BI-RADS Categories 0, 1, 6)
- Patient studies had been used in previous research work
- Fewer than 2 views of the mass available
- Graphic overlays or markers present
- Excessive artifacts or only color Doppler images present
- No or inconclusive pathology results
## 2. Clinical Endpoints
The following classical methods and endpoints were used:
- MRMC study design
- DBM method in estimation of validation Sample Size
- ROC AUC as primary endpoint,
- Specificity and Sensitivity as secondary endpoints.
In the BC study, ROC curves for BI-RADS scores assigned to a cohort of breast ultrasound imaging studies by multiple sub-specialty radiologists were compared “without” and “with” the use of the BC CLA.
## Validation Case File Selection and clinical data HIPAA compliance and security
Potential validation case files were initially identified by the clinical validation group from the catchment population by searching chronologically through their RIS database. This PACS-based digital database consists of files from studies (Joint Commission inspections for “standard of good practice” and HIPAA Regulations applied) performed on patients referred to their respective breast imaging clinics for diagnostic workup or biopsy. All case files were obtained retrospectively from PACS. Only those cases with corresponding confirmed findings (biopsy result or a documented two-year negative follow up) were “enrolled” in the study and the cases were retrieved in consecutive chronological order. No patients or cases were produced specifically for this study or per request from Almen Laboratories. Therefore the catchment represents a population pool that includes representative ethnic and racial groups, age groups, and tests completed on diagnostic ultrasound mammography stations in clinical use across all participating localities.
## B. Accountability of PMA Cohort
PMA P100007: FDA Summary of Safety and Effectiveness Data
{6}
The Validation Study period was for approximately 3 years, from late 2006 – late 2009. The initial IRB approval for the validation dataset was received in late 2006 and the study began shortly thereafter. Annual IRB reports were made to document study progress, and the IRB granted continued approval through the study period. The study was divided into the following periods:
| Date | Data | # Cases Read | Use | Endpoints |
| --- | --- | --- | --- | --- |
| 2006 | BC Standalone performance validation | 596 | Breast Companion® standalone performance validation and evaluation (no radiologists’ reading) | Statistical estimate on the software accuracy on 596 cases; Full ROC including AUC, Sensitivity, Specificity |
| 2006-2008 | Clinical validation - baseline reading “without” BC | 596 | Clinical Validation baseline “without” Breast Companion® | Statistical estimate on accuracy of the radiologists reading performance without using Breast Companion® software - 596 cases; Full ROC including AUC, Sensitivity, Specificity |
| 2008-2009 | BC Clinical validation - reading “with” BC CLA | 596, | Clinical Validation “with” Breast Companion® | Statistical estimate on accuracy of reading performance while using Breast Companion® software -596 cases; Full ROC including AUC, Sensitivity, Specificity |
Table 3. Clinical Validation Study Periods
## C. Study Population Demographics and Baseline Parameters
The clinical validation group assigned a unique anonymous study identifier to each enrolled case and removed all personally identifiable health information from each study case. Access to study files was restricted to approved medical staff with a research need to know. Access to the electronic data in the computer workstation with the BC installed was restricted to the Lead Investigator (non-clinical role), the Co-Investigators (clinical reading group) and their research staff via login/password and a network firewall. The Research Coordinators maintained the key to this code number and any other patient medical information in hardcopy form in a secure double-locked file cabinet in a secure room. Each image was stored in digital format with patient information deleted in a computer database by a sequential code number. The entire package of the image database and the associated pathological information was utilized only in computers located within the hospital firewall constructed to eliminate outside or intruder intervention in patient data, although patient data was de-identified of patient health identifiers. There were no identifiers associated with any electronic file which could be traceable to an individual patient other than this secure code.
All clinical data used in validation studies was IRB approved and had verified confirmed findings.
PMA P100007: FDA Summary of Safety and Effectiveness Data
{7}
A minimum of two images for each mass were required for the case to be ruled eligible for enrollment in the study. Minimal ultrasound image artifacts were allowed, but each view had to be free of graphic overlays, calipers and other markers, including color Doppler. Studies with no lesion present or known cancers were excluded. The eligible cases with all eligible available images and views were then entered into the Validation Database for analysis "without" and "with" the BC CLA function.
The final validation database consisted of 596 archived image cases. The final case mix was as follows:
| Lesion type | N |
| --- | --- |
| Simple Cysts | 165 (27.7%) |
| Complicated Cysts | 58 (9.7%) |
| Solid Benign | 240 (40.3%) |
| Malignant | 133 (22.3%) |
| Total | 596 (100%) |
Table 1. Case Mix, All Cases
## Size Distribution
The analysis included a histogram distribution of lesion size as well as complete descriptive statistics. The range was 2.0 to 98 mm, mean 12.8, median 10.0, standard deviation 9.3. The 95% reference limit was 3.0 mm (95% CI 2.0-4.0 mm) for the lower limit and 34.1 mm (95% CI 31.0-52.0 mm) for the upper limit.
Fig. 2. Lesion size distribution in Validation dataset
## U/S Scanner Distribution
PMA P100007: FDA Summary of Safety and Effectiveness Data
{8}
The ultrasound imaging studies used in the Validation Study were performed on a variety of makes and models of scanners. The studies used in all clinical studies are based on “standards of good practice” in breast ultrasonography, ACR BIRADS Guidance, and standards of practice in an MQSA accredited facility²⁰.
## Truth
Truth was established for the 596 cases retrieved for the final validation dataset by confirming biopsies and pathology reports in 390 cases (65.4%), while the remaining 206 cases (34.6%) that did not have biopsies had benign clinical follow-up of at least 2 years. Of the 206 cases that were recommended for follow-up and did not require biopsy, the characterizations by the radiologist were as follows:
| Lesion type | N |
| --- | --- |
| Simple Cysts | 165 (80.1%) |
| Complicated Cysts | 21 (10.1%) |
| Solid Benign | 20 (9.7%) |
| Total | 206 |
Table 2. Case Mix, Follow-up Cases
## Patient assessment and demographic data
No human subjects were recruited for the study because only previously archived pre-existing breast ultrasound files were eligible for enrollment. Potential cases were identified by the clinical validation group as previously described. While subject cases were not specifically screened for race, ethnic background or health status, the class of subjects included only women who had diagnostic sonograms and other related procedures at the study institution (i.e. biopsy, traditional breast ultrasound, MRI, etc.). The study did not specifically include or exclude any ethnic groups. No case files were excluded from the study after enrollment. Because the study used pre-existing cases with patient identifiers removed as a condition of the study (HIPAA Regulations, 2005), a complete demographic profile of the study population was not analyzed.
## D. Device Standalone Performance Data Analysis
### Device Standalone Performance Data Analysis
Standalone performance testing was designed to demonstrate how the BC CLA scoring algorithm performed on the lesions that are outlined (segmented) by the radiologists compared to the confirmed truth of the cases. ROC-AUC, Sensitivity and Specificity were the endpoints of this evaluation.
PMA P100007: FDA Summary of Safety and Effectiveness Data
{9}
The "standalone" computing performance evaluation provided safety data for the BC CLA function. Specifically the question to be answered was, "Does the BC CLA function, which is subject of this PMA, produce computed-lesion assessments score corresponding to the appropriate BI-RADS Category for a lesion in the absence of physician interaction at the score computing phase?"
The diagnostic Sensitivity of the BC is defined as the conditional probability that a person having a disease will be correctly identified: TP/(TP+FN). The diagnostic Specificity of the BC is defined as the conditional probability that a person not having a disease will be correctly identified: TN/(TN+FP). True Positive, True Negative, False Positive, and False Negative definitions are used in accordance with traditional references.
The Validation Case Files cohort was used to produce BC CLA scores that were consequently analyzed with ROC analysis. The 596 lesions assessed by the radiologist were scored using the Reference Library cases in batch processing using the radiologist's segmentation but without other radiologist interference in the CLA scoring. As a result, 596 BC CLA scores were produced and compared to the confirmed truth of each case.
This "standalone" computing performance evaluation was performed six to nine months prior to the "without" reading under the supervision of a single lead expert radiologist using the 596 validation cases and the Reference Library, as in the proposed device. The radiologist did not participate in the BC CLA scoring that was done and recorded automatically as a direct export from the BC software. The radiologist's supervision was limited to border definition of the lesion-in-question that was confirmed or corrected by the radiologist. Since BC CLA is intended to be used with the radiologist in the loop and under radiologist supervision this was the correct approach to evaluating "standalone" computing performance. When the lesion border was confirmed by the radiologist, the BC CLA scoring function was then used in automatic batch mode (no presence of any clinical personnel) and the BC CLA score produced was automatically recorded for each case.
For empirical estimations Analyse-It® Software by Analyse-It, Ltd., version 2.2, that has been accepted by FDA in multiple studies, was used. For comparison of the statistical results the lead of the validation group used also FDA recommended JROCFIT software as follows for the data analysis: Maximum likelihood estimation of ROC curve from categorical rating data (JAVA translation by John Eng, M.D. The Russell H. Morgan Department of Radiology and Radiological Science Johns Hopkins University, Baltimore, Maryland, USA, Version 1.0.2, March 2004). No statistically significant differences in the ROC results for both software packages were found. Therefore we present only one set of the outcomes for ROC analyses.
Table 4 below presents the ROC AUC analysis of the standalone performance of the BC CLA scoring function.
PMA P100007: FDA Summary of Safety and Effectiveness Data
{10}
| Test | Area (A_{z}) (fitted) | Area (A_{z}) (empirical) | 95% CI | SE | Z | p |
| --- | --- | --- | --- | --- | --- | --- |
| BC CLA | 98.6% | 98.2% | 0.97 to 0.99 | 0.006 | 82.81 | <0.0001 |
H₀: Area ≤ 0.5. H₁: Area > 0.5.
Table 4. ROC AUC (A<sub>z</sub>) analysis of standalone performance (BC CLA)
Table 5 below presents the Sensitivity and Specificity data for the BC CLA.
| BC CLA Score (Positive test > cutoff) | TP rate (Sensitivity) | 95% CI | | TN rate (Specificity) | 95% CI | |
| --- | --- | --- | --- | --- | --- | --- |
| Cutoff 2.5 | 0.985 | 0.947 | to 0.998 | 0.890 | 0.858 | to 0.917 |
| Cutoff 3.0 | 0.955 | 0.904 | to 0.983 | 0.946 | 0.921 | to 0.965 |
| Cutoff 3.5 | 0.932 | 0.875 | to 0.969 | 0.961 | 0.939 | to 0.977 |
| Cutoff 4.0 | 0.865 | 0.795 | to 0.918 | 0.978 | 0.961 | to 0.990 |
Table 5. Sensitivity and Specificity analysis for standalone BC CLA for different cut-off thresholds – BI-RADS 3 and 4 (for p < 0.0001)
Figure 3 below is an ROC AUC (empirical) curve plot for “standalone” BC CLA computing performance on the validation cohort.
PMA P100007: FDA Summary of Safety and Effectiveness Data
{11}
Fig. 3. ROC curve and AUC for "standalone" BC CLA computing performance.
For AUC ROC, BC CLA reached 98.2% for empirical and 98.6% for fitted method. At traditional BI-RADS Category 3, which corresponds to BC CLA 3 score, Sensitivity reached 95.5% with Specificity is 94.6% while the most effective BC CLA performance was found at 2.5 threshold level.
BC CLA does not produce the numeric estimation on the same scale as radiologists do when they go through the BI-RADS Assessment process and summarize their assessment in selection of one of the BI-RADS Categories. Therefore to illustrate the comparative performance of BC CLA, Table 6 below summarizes the fitted ROC computations (JROCKFIT software, method developed by professor Metz of Chicago University and used in their clinical trial at John Hopkins) based on the "without" reading data of the validation group of radiologists.
| 596 cases | RADs Average | BC CLA |
| --- | --- | --- |
| ROC AUC (A_{z}, fitted) | 83.1% | 98.7% |
| Sensitivity | 98.7% | 98.5% |
| Specificity | 41.5% | 89.0% |
Table 6. Area (A<sub>z</sub>), Sensitivity and Specificity comparison of BC CLA with FROC results of 4 radiologists reading validation data set of 596 cases "without" the device
All the differences with the exception of differences in Sensitivity exceed the statistical variability about the average values. Therefore we can conclude that we have evidence that the "standalone" safety of the device exceeds that of the comparable radiologists' results.
## E. Reproducibility Study
The reproducibility of BC CLA was compared for two views of the same mass, radial and anti-radial, acquired during the same examination for 125 masses. An example of such an image pair is shown in Figure 4 for a malignancy. These images show some differences in shape, echo pattern, boundary, and posterior shadowing. One might expect greater variability in malignancies and solid benign masses with perhaps less variability in complicated cysts and simple cysts. The measured CLA values for the image pairs are listed in Table 7 by type of mass. No significant differences were found. The areas under the ROC curves for the two views were likewise not significantly different (Table 7): 0.95 ± 0.02 for radial and 0.97 ± 0.02 for anti-radial.
PMA P100007: FDA Summary of Safety and Effectiveness Data
{12}
Table 8 also shows ROC AUC results when the CLA values for each of the image pairs are used separately or when they are combined in various ways.
Fig. 4. Radial (left) and anti-radial views of the same mass (malignant).
| Lesion Type | Radial Mean ± SD | Anti-Radial Mean ± SD | 2-tailed p value |
| --- | --- | --- | --- |
| Simple Cyst | 2.00 ± 0.00 | 2.03 ± 0.18 | 0.324* |
| Complicated Cyst | 2.51 ± 0.85 | 2.36 ± 0.75 | 0.531* |
| Solid Benign | 2.58 ± 0.92 | 2.66 ± 0.96 | 0.599* |
| Malignant | 4.59 ± 0.54 | 4.67 ± 0.51 | 0.540* |
* These p-values are not significant
Table 7. CLA Values for Two Views of Same Mass (125 cases, 30% malignant)
| | A_{z} Fit | ± SE | A_{z} Trapezoidal | ± SE |
| --- | --- | --- | --- | --- |
| Radial | 0.9513 | 0.0207 | 0.9577 | 0.0275 |
| Anti-Radial | 0.9691 | 0.0229 | 0.9677 | 0.0242 |
Table 8. ROC AUC for Two Views of the Same Mass (125 cases, 30% malignant)
An additional 36 cases of the same mass scanned on more than one scanner were analyzed. All the differences did not exceed the statistical variability above the average values.
Therefore we can confidently conclude that we have clear evidence of proven reproducibility for the device tested performance.
F. Clinical Data
PMA P100007: FDA Summary of Safety and Effectiveness Data
{13}
# Clinical Validation Data Collection Method
Four sub-specialty breast imaging radiologists independently, in 6 random sessions, assessed the total of 596 US cases (100 randomly selected cases for Sample Size estimation and the balance of 496 cases in validation reading) using a standard BI-RADS hardcopy Classification Form (ACR) that includes final assessment category, descriptors and recommendations for follow-up interval or biopsy. After at least 6 months' interval from the "without" reading ("washout period"), the same group of radiologists again in 6 random sessions, completed a BI-RADS assessment of the same validation set of randomly mixed cases while using the BC device. In addition to BI-RADS Assessment, the software recorded the radiologist selected Probability of Malignancy. The readers were not aware of the confirmed findings for any case for both reads – "without" and "with".
# Clinical Validation Data Analyses Methods
The Clinical Validation Study "with" and "without" the BC device provided effectiveness data via reader performance testing using a Multiple Reader/Multiple Cases (MRMC) design.
The performances of the radiologists "with" and "without" Breast Companion® were compared, to determine if the radiologist improved his/her performance with the use of the BC software.
Primary endpoints were defined as:
- ROC-AUC ("A₂" or Area Under ROC Curve)
- Sensitivity of Breast Companion at BI-RADS cut-point 3, for a 5-scale
- Specificity of Breast Companion at BI-RADS cut-point 3, for a 5-scale
The primary analysis was based on the comparison of mean performance measure using the two modalities ("without" and "with" BC). Formally:
$$
\mathrm{H}_0: \text{Perf(with)} = \text{Perf(without)};
$$
$$
\mathrm{H}_1: \text{Perf (with)} \neq \text{Perf(without)}.
$$
Here Perf (with/without) denotes the mean performance measure value for the given modality (treatment) for the reference populations of Cases and Readers. The performance measure is ROC-AUC, Sensitivity at cut-point 3, or Specificity at cut-point 3. A two-sided 5% CI significance level was used to test for each performance measure.
The study compared radiologist performance (BI-RADS Assessment) without use of the BC CLA software to radiologist performance (BI-RADS Assessment) while using the BC CLA scoring software.
PMA P100007: FDA Summary of Safety and Effectiveness Data
{14}
# Clinical Validation Reading Results
Summarized ROC analysis results for the validation cohort of 596 cases are presented in Table 9. The AUC differences between the radiologists for "without" reading are not significantly different and are similar to those achieved by both clinical groups in prior studies, and are similar to those reported elsewhere for breast sonography.
| | A_{z} | Sensitivity Cutoff 3 | Specificity Cutoff 3 | Sensitivity Cutoff 4 | Specificity Cutoff 4 |
| --- | --- | --- | --- | --- | --- |
| Rad1 without BC | 81.88% | 97.74% | 54.00% | 28.57% | 98.49% |
| Rad1 with BC | 85.22% | 97.74% | 57.02% | 39.85% | 98.27% |
| Rad2 without BC | 81.81% | 98.50% | 41.90% | 45.11% | 96.54% |
| Rad2 with BC | 86.76% | 96.99% | 61.99% | 43.61% | 96.98% |
| Rad3 without BC | 83.99% | 100.00% | 25.92% | 87.97% | 76.89% |
| Rad3 with BC | 86.29% | 99.25% | 34.77% | 73.68% | 90.28% |
| Rad4 without BC | 84.80% | 98.50% | 44.06% | 54.89% | 96.11% |
| Rad4 with BC | 90.77% | 93.23% | 71.71% | 66.92% | 96.11% |
| Average "without" | 83.12% | 98.68% | 41.47% | 54.14% | 92.01% |
| Average "with" | 87.26% | 96.80% | 56.37% | 56.02% | 95.41% |
| Average Difference | 4.14% | -1.88% | 14.90% | 1.88% | 3.40% |
| Parameters of statistical difference | p=0.05 | CI 95% interval size "without" 4.75% | CI 95% interval size "without" 8.97% | CI 95% interval size "without" 15.68% | CI 95% interval size "without" 4.42% |
Table 9. Summary of "without" and "with" Validation Study on the set of 596 confirmed cases, cancer prevalence 22.3%.
In addition to effectiveness estimation based on ROC AUC, a traditional accuracy index based on TP, TN, FP and FN differences between "without" and "with" was also computed. Traditional estimation could be computed by using Accuracy index $A_c = (TP + TN) / (TP + TN + FP + FN)$. Using the statistically significant input from the readings ROC results the $A_c$ increase "with" compared to radiologist performance "without" were computed as +6.51% (for 596 case cohort) on average and +6.53 (for 496 case cohort) on average.
## Statistical significance of the endpoints
PMA P100007: FDA Summary of Safety and Effectiveness Data
{15}
As part of the statistical analyses, the hypothesis $H_0$ that “without” and “with” treatments are on average equal was tested. Readers were treated as “random” effects in the model because there was no pre-selection or filtration of the readers. With $p = 0.0024$ for the 496 case cohort and $p = 0.0112$ for the 596 case cohort, the software confirmed that for ROC AUC, “treatments are not equal” and therefore there is a statistically significant difference between the two “treatments” or “modalities”, with the “with” modality being more accurate than the “without” modality.
Statistical Significance of Specificity Improvement "with" BCis an important co-primary endpoint result. It was found that Specificity improvement of the "with" reading is statistically significant compared to the Specificity of the "without" reading. DBM software output confirms that Specificity of "without" and "with" treatments "are not equal" while for Sensitivity "The treatment SENSITIVITIES were not significantly different" ($p > 0.5$).
## XI. PANEL MEETING RECOMMENDATION AND FDA'S POST-PANEL ACTION
In accordance with the provisions of section 515(c)(2) of the act as amended by the Safe Medical Devices Act of 1990, this PMA was not referred to the Radiological Advisory Panel, an FDA advisory committee, for review and recommendation because the information in the PMA substantially duplicates information previously reviewed by this panel.
## XII. CONCLUSION.S DRAWN FROM PRECLINICAL AND CLINICAL STUDIES
### A. Safety Conclusions
There are no known direct risks to safety or health caused by, or related to, the use of the device. Indirect risks are that the device may mark some nonmalignant lesions (false positive readings). If a physician determines that a false positive mark indicate an area that is suspicious enough for follow-up, then the patient may be subjected to unnecessary concern and/or biopsy. There are no false negative where BI-RADS 3, 4, and 5 lesions were downgraded to BI-RADS 2.
### B. Effectiveness Conclusions
The summary results of Breast Companion® standalone performance and reproducibility evaluation described above and ROC data of BC performance compared to that of the radiologists (Tables 6-8; Fig. 3) support the conclusion that the device's performance is safe. In addition, the clinical Validation Study provided statistically significant data (Table 9) for the effectiveness of the Breast Companion®.
### C. Overall Conclusions
The data in this application support the reasonable assurance of safety and effectiveness of this device when used in accordance with the indications for use.
PMA P100007: FDA Summary of Safety and Effectiveness Data
{16}
The effectiveness of Breast Companion® evaluation was analyzed based on measurements of the physical performance of imaging systems, as well as other bench tests and standalone measurements. In addition, the diagnostic accuracy analysis includes measurement of sensitivity, specificity, the ROC curve and its summary measures.
## XIII. CDRH DECISION
CDRH issued an approval order on February 10, 2012. The final conditions of approval cited in the approval order.
The applicant’s manufacturing facility was inspected on August 30, 2010 and found to be in compliance with the device Quality System (QS) regulation (21 CFR 820).
## XIV. APPROVAL SPECIFICATIONS
Directions for use: See device labeling.
Hazards to Health from Use of the Device: See Indications, Contraindications, Warnings, Precautions, and Adverse Events in the device labeling.
Post-approval Requirements and Restrictions: See approval order.
## XV. REFERENCES
1. Galperin, M: “Statistical Analysis to Assess Automated Level of Suspicion Scoring Methods in Breast Ultrasound.” Proc SPIE, Vol 5034:432-439, 2002
2. MP Andre, M Galperin, L. Olson, C. Comstock, K. Richman, M. O’Boyle: “Performance of a computer-aided tool for diagnostic breast ultrasound,” RSNA:235, Chicago, 2005
3. M Galperin, BB Goldberg, CJ D’Orsi, H Ojeda-Fournier, CRB Merritt, MP Andre: “Computer-aided teaching file tool for BIRADS training and education,” RSNA: 709, Chicago IL, 2007
4. M Galperin, DJ D’Orsi, BB Goldberg, CRB Merritt, MP Andre, H Ojeda-Fournier, et al.: “BIRADS Tutor®: CAD-based Advanced Electronic Teaching File for BI-RADS Training in Assessment and Breast US Reporting,” RSNA: 518, Chicago IL, 2009
5. M. Galperin, M.P. Andre, H. Ojeda-Fournier, L.K. Olson, M.K. O’Boyle, M. Ledgerwood, et al.: “Radiologists Performance and Reproducibility Comparative Clinical Study: ‘With’ and ‘Without’ Computerized Decision-making Support System in Breast Ultrasound BI-RADS Image Interpretation, Classification, and Assessment,” RSNA, Chicago IL, 2011.
PMA P100007: FDA Summary of Safety and Effectiveness Data
page 17
20
{17}
6. M Galperin, L K Olson, A N Wilkes, M P Andre, F Forsberg, B B Goldberg: "Practical Improvement of BI-RADS Interpretation and Assessment Reading while Using Validated Breast Ultrasound Diagnostic Scoring Tool as Decision-Making Support System", RSNA, Chicago IL, 2011.
7. L Brogoch, M Andre, M Ledgerwood, H Ojeda-Fournier, L Olson, M Galperin, M O'Boyle, C Comstock, M Aiken: "Reader variability and predictive value of BIRADS descriptors for lesions in breast sonography." Presented at Amer Inst Ultrasound in Medicine, San Diego CA, 2010.
8. M Ledgerwood, M Andre, M Galperin, H Ojeda-Fournier, L Olson, M O'Boyle, C Comstock: "A computer-aided diagnostic method for assessment of findings on breast sonograms." Presented at Amer Inst Ultrasound in Medicine, San Diego CA, 2010.
PMA P100007: FDA Summary of Safety and Effectiveness Data
page 18
21
