- Open Access
Dummy run quality assurance study in the Korean Radiation Oncology Group 19 − 09 multi-institutional prospective cohort study of breast cancer
Radiation Oncology volume 17, Article number: 186 (2022)
The Korean Radiation Oncology Group (KROG) 19 − 09 prospective cohort study aims to determine the effect of regional nodal irradiation on regional recurrence rates in ypN0 breast cancer patients. Dosimetric variations between radiotherapy (RT) plans of participating institutions may affect the clinical outcome of the study. We performed this study to assess inter-institutional dosimetric variations by dummy run.
Twelve participating institutions created RT plans for four clinical scenarios using computed tomography images of two dummy cases. Based on a reference structure set, we analyzed dose-volume histograms after collecting the RT plans.
We found variations in dose distribution between institutions, especially in the regional nodal areas. Whole breast and regional nodal irradiation (WBI + RNI) plans had lower inter-institutional agreement and similarity for 95% isodose lines than WBI plans. Fleiss’s kappa values, which were used to measure inter-institutional agreement for the 95% isodose lines, were 0.830 and 0.767 for the large and medium breast WBI plans, respectively, and 0.731 and 0.679 for the large and medium breast WBI + RNI plans, respectively. There were outliers in minimum dose delivered to 95% of the structure (D95%) of axillary level 1 among WBI plans and in D95% of the interpectoral region and axillary level 4 among WBI + RNI plans.
We found inter-institutional and inter-case variations in radiation dose delivered to target volumes and organs at risk. As KROG 19 − 09 is a prospective cohort study, we accepted the dosimetric variation among the different institutions. Actual patient RT plan data should be collected to achieve reliable KROG 19 − 09 study results.
Regional recurrence is rare, even without regional nodal irradiation (RNI), in T1–3N1 breast cancer patients with ypN0 after neoadjuvant chemotherapy (NAC) and breast-conserving surgery (BCS) . In a retrospective study in Korea of patients with ypN0 after NAC and BCS from 2005 to 2011, only 39.1% received RNI, and it did not improve locoregional control or survival . However, there is no definitive data that support the omission of RNI in patients with ypN0 after NAC and BCS yet, especially in patients who undergo sentinel lymph node biopsy (SLNB).
To determine the effect of RNI on regional recurrence rates in those patients, the Korean Radiation Oncology Group (KROG) initiated a study titled ‘Prospective cohort study to evaluate the effect of regional nodal irradiation omission on the regional recurrence rate after neoadjuvant chemotherapy followed by breast-conserving surgery and sentinel lymph node biopsy in clinical T1-3 with lymph node metastasis in axillary level I and ypN0 breast cancer patient (the KROG 19 − 09 study, CRIS no. KCT0004567)’ in October 2019.
The treatment policy for RNI in ypN0 patients is diverse in Korea. In this prospective cohort study, the institutional policies are respected, and radiation oncologists choose either whole breast irradiation (WBI) or WBI + RNI after discussion with their patients. Radiotherapy (RT) is delivered according to institutional policy.
The KROG 19 − 09 protocol for radiotherapy is as follows: 4–6 megavolt x-ray is recommended; computed tomography (CT) simulation is required; supine position is recommended; either conventional fractionation (daily dose of 1.8-2.0 Gy, 23–28 fractions, total dose of 45-50.4 Gy) or hypofractionation (daily dose of 2.5-3.0 Gy, 13–16 fractions, total dose of 39-43.2 Gy) is allowed; 3-D conformal radiotherapy is recommended; intensity-modulated radiotherapy is allowed; adherence to the ESTRO (European Society for Radiotherapy and Oncology) consensus guideline for clinical target volume (CTV) for whole breast and regional lymph node is recommended; V5Gy, V10Gy, V20Gy, V30Gy, and the mean dose for ipsilateral lung and heart on the dose-volume histogram (DVH) must be recorded. At the kick-off meeting for KROG 19 − 09, we found that the participating institutions use various RT techniques and field designs. It is well known that the actual dose coverage of the regional nodal area varies according to RT technique and field design [3,4,5]. Variations in dose distribution in the regional nodal area and the organs at risk (OARs) between the participating institutions may affect the clinical outcome of the study. The purpose of this study was to assess inter-institutional dosimetric variations by dummy run.
Dummy run procedure
Four clinical scenarios, (1) whole breast irradiation (WBI) in a large-breast case, (2) whole breast and regional nodal irradiation (WBI + RNI) in the large-breast case, (3) WBI in a medium-breast case and (4) WBI + RNI in the medium-breast case, were prepared upon which institution-specific radiation treatment plans were created based on each institution’s protocols (Fig. 1). This study does not include information about the patient-specific quality assurance protocols of the treatment plans.
All institutions were also asked to fill out a prepared questionnaire regarding the treatment environment and plan information such as treatment planning system, photon beam energy, dose calculation algorithm, treatment technique, total dose, and fraction number. Anonymized CT images of a large- and a medium-breast case with cT1–3N1 breast cancer on the left side with ypN0 were provided for the treatment planning. In terms of physical characteristics, the large-breast case had a height of 1.57 m, weight of 75 kg, and body mass index (BMI) of 30.4 (kg/m2), and the medium-breast case had a height of a 1.61 m, weight of 63 kg and BMI of 24.3 (kg/m2).
The host institution received dose distribution data for the WBI and WBI + RNI plans for the large- and medium-breast cases created by the 12 participating institutions in the format of each institution’s RT Dose Digital Imaging and Communications in Medicine (DICOM) files. Reference CTVs and OARs were delineated according to the ESTRO consensus guideline and a cardiac contouring atlas by a radiation oncologist (Y-J. Kim) [6, 7] to evaluate the DVHs. These reference contours were analyzed by radiation oncology panel reviewers MY. Kim and WK. Cho to ensure they conformed to guidelines. The reference contours were not supplied to the participating institutions but only used for the DVH analysis. The reference structure set included a left breast, interpectoral node CTV (CTVn_intpect), axillae and supraclavicular node CTVs (CTVn_L1-L4), internal mammary node CTV (CTVn_IMN), left lung, heart, shoulder joint with 1 cm margin (sh joint + 1 cm) and left anterior descending coronary artery (LAD_coronary a.).
Based on the reference structure set, we assessed inter-institutional variations in dose to the CTVs and OARs. DVH analysis software developed using ESAPI (eclipse script application pro-gram interface, from Varian’s treatment planning system) was used. For DVH comparisons, we analyzed mean doses and standard deviations for each of the clinical scenarios for all institutions. For CTVs, we calculated the minimum dose delivered to 95% of the structure (D95%). Because of inter-institutional differences in dose fractionation schedules, the results were expressed as D95% as a percentage of the prescribed dose. For the OARs, we calculated the volume receiving 20 Gy (V20Gy) for the left lung; the mean dose and the minimum dose delivered to 5% of the structure (D5%) for the heart and the right breast, respectively; and the volume receiving 20 Gy (V20Gy) and 5 Gy (V5Gy), for the LAD_coronary a. and sh joint + 1 cm, respectively. Boost dose to the tumor bed was not considered when analyzing DVHs. To analyze agreement and similarity of 95% isodose lines between institutions, we utilized two analysis tools; (1) Fleiss’s kappa  was calculated using the computational environment for radiotherapy research (CERR)  that is MATLAB-based radiotherapy research platform, and (2) Jaccard and Dice similarity coefficients were calculated using MIM (OH, USA) [10, 11]. Fleiss’s kappa value assesses the reliability of agreement between the participating institutions’ isodose lines, which evaluates the degree of inter-variation .
Distributions of variables were expressed as medians and ranges (minimum-maximum), and differences between large- and medium-breast cases were assessed using the Wilcoxon rank-sum test. To graphically display the data, we present box plots of five variables: minimum value, maximum value, median, and the first (Q1) and third (Q3) quartiles (i.e., values of the 25th and 75th percentiles of the data set). The interquartile range (IQR) is the distance between the first and third quartiles. If a data value was less than Q1-1.5IQR or greater than Q3 + 1.5IQR, the value was considered an outlier. Outliers were plotted as individual points, and each individual point is labelled with the participating institution and value. A p-value less than 0.05 was considered statistically significant. All statistical analysis was performed using SAS software, version 9.4 (SAS Institute Inc., Cary, NC, USA.), and R software, version 4.1.0 (R Project for Statistical Computing).
Treatment plan information
As shown in Tables 1, 11 institutions used the Eclipse RTP system (Varian Medical System, Palo Alto, CA), and one used Pinnacle (Philips Healthcare, Andover, MA). Ten institutions used 6MV photon beams for breast and supraclavicular node fields, and two institutions used 6MV beams for breast and 10MV beams for supraclavicular node fields. All institutions calculated dose distributions using an adaptive convolution (AC) (n = 1) or analytical anisotropic algorithm (AAA) (n = 11) to account for inhomogeneity.
For the WBI plans, nine institutions used standard tangential field three-dimensional conformal radiation therapy (3D-CRT) for the large- and medium-breast cases. Three employed intensity-modulated radiation therapy (IMRT) for both cases. Notably, volumetric-modulated arc therapy (VMAT) was used by one institution in the WBI plan of both cases. For the WBI + RNI plans, for the large-breast case, seven institutions used 3D-CRT and five used IMRT, and for the medium-breast case, five used 3D-CRT and seven used IMRT.
The prescription dose (Gy) for all treatment plans of participating institutions ranged from 40.05 to 50.4 Gy, and the fraction numbers ranged from 15 to 28. The prescribed dose per fraction was 1.8 to 2.7 Gy. All the dose fractionation schedules are accepted as biologically equivalent for breast cancer.
Radiation dose variation between institutions
Figure 2 presents the target contours and isodose lines from the participating institutions for their WBI + RNI plans for the large-breast case. The red and orange lines represent the breast and regional node targets, and the yellow lines are the 95% isodose lines. One institution (L) did not delineate the breast or regional node targets, and three (A, B, I) delineated only breast targets. In these institutions, the RT fields were designed according to anatomical landmarks. We found variations in dose distributions between institutions, especially in the regional nodal areas. As shown in Table 2, Fleiss’s kappa values, which were used to measure inter-institutional agreement for the 95% isodose lines, were 0.830 and 0.767 for the large and medium breast WBI plans, respectively, and 0.731 and 0.679 for the large and medium breast WBI + RNI plans, respectively. The WBI + RNI plans had lower inter-institutional agreement and similarity for 95% isodose lines than the WBI plans.
There were three outliers among WBI plans with significant differences in D95% of axillary level 1 (Fig. 3). This means that at these institutions, cases received significant doses to the axillary level 1, even with a WBI plan. There were several outliers among WBI + RNI plans with significant differences in D95% of the interpectoral region and axillary level 4 (Fig. 4). This means that cases did not receive a sufficient dose to these areas at these institutions, even with a WBI + RNI plan.
Radiation dose variation between cases
The differences in dose distributions between the large- and medium-breast treatment plans are shown in Table 3. In the WBI plans, the median D95% (%) of CTVn-L1 for the large- and medium-breast cases were 6.0% (range, 2.6-78.4%) and 2.9% (1.2-22.9%), respectively. This difference was statistically significant (p-value = 0.0304). There was a statistically significant difference in LAD_coronary a. V20Gy (%) (p-value = 0.0024, median; 39.4% (range, 6.1-42.6%) vs. 45.1% (range, 38.1-52.7%) for the large- and medium-breast cases, respectively). The was also a statistically significant difference in LAD_coronary a. V20Gy (%) in the WBI + RNI plans (p-value = 0.0007), median; 41.6% (range, 7.8-43.1%) vs. 48.2% (40.5-53.7%) for large- and medium-breast cases, respectively).
In this dummy run study, we found variations in radiation dose delivered to target volumes and organs at risk between institutions. As KROG 19 − 09 is a prospective cohort study, we accepted the dosimetric variation among the different institutions. Although IMRT is available at all the participating institutions, a majority use 3D plans for breast cancer because of limitations in resources. Some institutions still use anatomical landmarks without CTV contouring to reduce workload. The idea that irradiation of the entire lymphatic system may not be necessary for oncologic benefit and the low incidence of severe toxicity with breast RT plans are cited as reasons for using anatomical landmarks without CTV contouring. With a 3D plan, it is inevitable that unintentional doses are delivered to unintended areas. Another limitation of 3D plans is an unavoidable low dose at the field junction. Differences in RT techniques are the main reason for the dose variations between institutions.
As shown in previous international trial dummy runs, it is essential to implement quality assurance to allow the quality of trial data to be optimized and quantified [12,13,14]. In 2017, a phase III randomized trial was initiated by the Korean Radiation Oncology Group (the KROG 17 − 01 study, NCT03269981) to analyze the impact of RNI in pN1 breast cancer patients receiving effective systemic therapy. The primary objective of the KROG 17 − 01 study was to compare disease-free survival between WBI and WBI + RNI in pN1 breast cancer patients who received BCS and taxane-based chemotherapy. For adequate interpretation of the KROG 17 − 01 study results, an in-silico planning study comparing radiation dose distributions to the regional lymph nodes between the WBI and WBI + RNI plans of institutions participating in the KROG 17 − 01 study was performed . The study found that the relative nodal dose was significantly lower with WBI than WBI + RNI (p-value < 0.01) in all nodal regions. It also found moderate-to-strong agreement in radiotherapy treatment volumes between the participants. Significant proportions of radiation were unintentionally delivered to the axillary lymph node level 1 and IMN regions in the WBI plans. Our findings agree with the KROG 17 − 01 in-silico study.
Another dummy run study for quality assurance of a randomized trial on IMN irradiation (the KROG 08 − 06 study) reported that the mean radiation dose to the IMN region was 40–74% of the prescribed dose in their WBI arm . In our study, the median D95% (%) of CTVn-IMN for the large- and medium-breast cases were 4.1% (range, 1.6-17.3%) and 3.6% (range, 1.4-15.5%), respectively, in the WBI plans. Compared to the KROG 08 − 06 dummy run study, the radiation dose to the IMN region in the WBI alone plan was lower in our study.
Recently an insightful dose evaluation study was published . The study reconstructed the treatment plans of the landmark Z0011 , AMAROS , EORTC 22,922 − 10,925 , and MA.20  randomized lymph node irradiation trials to assess the dose distribution to actual lymph node metastases and the ESTRO-CTVs. They searched the study protocols of the Z0011, AMAROS, EORTC, and MA.20 trials for specifications regarding the treatment planning procedure. The field arrangements described in the protocols were used to imitate the 2D treatment plans on 3D computed tomography datasets of (1) a standard patient, (2) an obese patient with large breasts, and (3) a slender patient with small breasts. In these landmark trials, dose distributions at the axillary level 1, 2, 3 and the supraclavicular and IMN regions varied. These variations resulted from differences in RT techniques and field designs in each trial. They also found that the extent of incidental irradiation to the axillary nodes depended clearly on the patient’s body shape. In line with this previous study, we found inter-institutional and inter-case dose variations.
Variations in dose distribution at the regional nodal areas and OARs between participating institutions may affect the KROG 19 − 09 study’s clinical outcomes. To reduce the dosimetric variation among institutions, we plan to provide feedback from periodic audits. Furthermore, actual patient RT plan data should be collected to analyze the effect of these variations on regional recurrence rates and toxicity. To ensure reliable results, participants of KROG 19 − 09 agreed to the collection of actual patient RT plan data. We plan to collect DVH data by using artificial intelligence (AI)-based auto contouring software to delineate CTV and OAR structures. We will be able to analyze the relationship between the radiation dose and recurrence or toxicity rates based on the segmented structures with these data.
There are several limitations to this study. First, we performed the dummy run study in only two different breast-sized cases because of resource limitations. The inclusion of more cases with more diverse breast sizes could have provided more information about dose variation. Second, some institutions dropped out of and others joined KROG 19 − 09 after this dummy run study. Therefore, the results of this dummy run study are not able to reflect all the participants of KROG 19 − 09. However, this study is worthwhile as it allowed the participants to agree on collecting actual patient RT planning data in order to obtain reliable results for KROG 19 − 09.
In conclusion, in this dummy run study, we found inter-institutional and inter-case variations in radiation dose delivered to target volumes and organs at risk. As KROG 19 − 09 is a prospective cohort study, we accepted the dosimetric variation among the different institutions. Actual patient RT plan data should be collected to achieve reliable KROG 19 − 09 study results.
Availability of supporting data
The supporting data is available.
Mamounas EP, Anderson SJ, Dignam JJ, Bear HD, Julian TB, Geyer CE Jr, et al. Predictors of locoregional recurrence after neoadjuvant chemotherapy: results from combined analysis of National Surgical Adjuvant Breast and Bowel Project B-18 and B-27. J Clin Oncol. 2012;30:3960–6.
Cho WK, Park W, Choi DH, Kim YB, Kim JH, Kim SS, et al. Role of Elective Nodal Irradiation in Patients With ypN0 After Neoadjuvant Chemotherapy Followed by Breast-Conserving Surgery (KROG 16–16). Clin Breast Cancer. 2019;19:78–86.
Hurkmans CW, Borger JH, Rutgers EJ, van Tienhoven G, Group EBCC, Radiotherapy Cooperative G. Quality assurance of axillary radiotherapy in the EORTC AMAROS trial 10981/22023: the dummy run. Radiother Oncol. 2003;68:233–40.
Jagsi R, Chadha M, Moni J, Ballman K, Laurie F, Buchholz TA, et al. Radiation field design in the ACOSOG Z0011 (Alliance) Trial. J Clin Oncol. 2014;32:3600–6.
Borm KJ, Oechsner M, Dusberg M, Buschner G, Weber W, Combs SE, et al. Irradiation of regional lymph node areas in breast cancer - Dose evaluation according to the Z0011, AMAROS, EORTC 10981–22023 and MA-20 field design. Radiother Oncol. 2020;142:195–201.
Offersen BV, Boersma LJ, Kirkove C, Hol S, Aznar MC, Biete Sola A, et al. ESTRO consensus guideline on target volume delineation for elective radiation therapy of early stage breast cancer. Radiother Oncol. 2015;114:3–10.
Duane F, Aznar MC, Bartlett F, Cutter DJ, Darby SC, Jagsi R, et al. A cardiac contouring atlas for radiotherapy. Radiother Oncol. 2017;122:416–22.
Rucker G, Schimek-Jasch T, Nestle U. Measuring inter-observer agreement in contour delineation of medical imaging in a dummy run using Fleiss’ kappa. Methods Inf Med. 2012;51:489–94.
Deasy JO, Blanco AI, Clark VH. CERR: a computational environment for radiotherapy research. Med Phys. 2003;30:979–85.
Allozi R, Li XA, White J, Apte A, Tai A, Michalski JM, et al. Tools for consensus analysis of experts’ contours for radiotherapy structure definitions. Radiother Oncol. 2010;97:572–8.
Fedorov A, Beichel R, Kalpathy-Cramer J, Finet J, Fillion-Robin J-C, Pujol S, et al. 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magn Reson Imaging. 2012;30:1323–41.
Fairchild A, Collette L, Hurkmans CW, Baumert B, Weber DC, Gulyban A, et al. Do results of the EORTC dummy run predict quality of radiotherapy delivered within multicentre clinical trials? Eur J Cancer. 2012;48(17):3232–9.
Kearvell R, Haworth A, Ebert MA, Murray J, Hooton B, Richardson S, et al. Quality improvements in prostate radiotherapy: outcomes and impact of comprehensive quality assurance during the TROG 03.04 ‘RADAR’ trial. J Med Imaging Radiat Oncol. 2013;57(2):247–57.
Khaw P, Do V, Lim K, Cunninghame J, Dixon J, Vassie J, et al Radiotherapy Quality Assurance in the PORTEC-3 (TROG 08.04) Trial. Clin Oncol (R Coll Radiol). 2022;34(3):198-204.
Kim H, Kim H, Park W, Baek JY, Ahn SJ, Kim MY, et al. Comparison of Dose Distribution in Regional Lymph Nodes in Whole-Breast Radiotherapy vs. Whole-Breast Plus Regional Lymph Node Irradiation: An In Silico Planning Study in Participating Institutions of the Phase III Randomized Trial (KROG 1701). Cancers (Basel) 2020;12.
Chung Y, Kim JW, Shin KH, Kim SS, Ahn SJ, Park W, et al. Dummy run of quality assurance program in a phase 3 randomized trial investigating the role of internal mammary lymph node irradiation in breast cancer patients: Korean Radiation Oncology Group 08 – 06 study. Int J Radiat Oncol Biol Phys. 2015;91:419–26.
Giuliano AE, Ballman KV, McCall L, Beitsch PD, Brennan MB, Kelemen PR, et al. Effect of Axillary Dissection vs No Axillary Dissection on 10-Year Overall Survival Among Women With Invasive Breast Cancer and Sentinel Node Metastasis: The ACOSOG Z0011 (Alliance) Randomized Clinical Trial. JAMA. 2017;318:918–26.
Donker M, van Tienhoven G, Straver ME, Meijnen P, van de Velde CJH, Mansel RE, et al. Radiotherapy or surgery of the axilla after a positive sentinel node in breast cancer (EORTC 10981–22023 AMAROS): a randomised, multicentre, open-label, phase 3 non-inferiority trial. Lancet Oncol. 2014;15:1303–10.
Poortmans PM, Collette S, Kirkove C, Van Limbergen E, Budach V, Struikmans H, et al. Internal Mammary and Medial Supraclavicular Irradiation in Breast Cancer. N Engl J Med. 2015;373:317–27.
Whelan TJ, Olivotto IA, Parulekar WR, Ackerman I, Chua BH, Nabid A, et al. Regional Nodal Irradiation in Early-Stage Breast Cancer. N Engl J Med. 2015;373:307–16.
This work was supported by the National Cancer Center Korea [grant numbers 2110210].
Ethical approval and consent to participate
The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Review Board of the National Cancer Center and Cancer Research Institute in Korea (IRB no. NCC2021-0006, 01-13-2021). Informed consent was obtained from all subjects involved in the study.
Consent for publication
We accept the conditions of submission and the BMC Copyright and License Agreement and consent to publication.
We have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Kim, M., Park, B., Kim, H. et al. Dummy run quality assurance study in the Korean Radiation Oncology Group 19 − 09 multi-institutional prospective cohort study of breast cancer. Radiat Oncol 17, 186 (2022). https://doi.org/10.1186/s13014-022-02140-0
- Breast neoplasms
- Quality assurance
- Dummy run
- Dosimetric variations