Evaluation of inter-observer variability of bladder boundary delineation on cone-beam CT
Radiation Oncologyvolume 8, Article number: 185 (2013)
In-room cone-beam computerized tomography (CBCT) imaging is a promising method to reduce setup errors, especially in organs such as the bladder that often have large intrafractional variations due to organ movement. CBCT image quality is limited by low contrast and imaging artifacts, but few data have been reported about inter-observer variability of bladder boundary delineation on CBCT. The aim of this work was to analyze and evaluate the inter-observer contouring uncertainties of bladder boundary delineation on CBCT images in a prospective fashion.
Five radiation oncologists contoured 10 bladders using the CBCT datasets of consecutive 10 patients (including 4 females) who were irradiated to the pelvic region. Prostates were also contoured in male patients. Patients who had had prostatectomy were excluded. The coefficient of variation (COV), conformity index (CIgen), and coordinates of center-of-mass (COM) of the bladder and prostate were calculated for each patient.
The mean COV for the bladder and prostate was 0.08 and 0.20, respectively. The mean CIgen of the bladder and prostate was 0.81 and 0.66, respectively. The root mean square (RMS) of the inter-observer standard deviation (σ) of the COM displacement in the left-right (LR) and anterior-posterior (AP) direction was 0.79, 0.87 and 0.54 for the bladder and 0.63, 0.99 and 1.72 for the prostate. Regarding the mean COV and CIgen for the bladder, the differences between males and females were not significant.
Inter-observer variability for bladder delineation on CBCT images was substantially small regardless of gender. We believe that our results support the applicability of CBCT in adaptive radiotherapy for bladder cancer.
The bladder continually changes volume and position on a daily basis, and as a result, treating a bladder typically requires at least a 1.5- to 2-cm isotropic setup margin in radiotherapy [1, 2]. Such a large margin and treatment field may result in late bladder and bowel toxicity [3, 4]. Conformal irradiation of the bladder may reduce these complication risks.
Recently, various kinds of image-guidance technology, such as implanted fiducial markers, on-board kilovoltage cone-beam computed tomography (CBCT), and ultrasonograpy, are widely used [5, 6]. We had previously reported the efficacy of implanted fiducial markers in reducing uncertainty due to setup error and internal organ motion [7, 8], but implantation is an invasive procedure, and fiducial markers are themselves surrogates for implanted organ position and provide no information on organ deformation or volume.
Of the other image-guidance technologies, CBCT is less invasive and the most common image-guided radiation therapy (IGRT) method, providing the volumetric-anatomic information and the opportunity to localize target volumes in a few minutes before each treatment fraction. Daily online adaptive radiotherapy using pre-planned treatment plans and CBCT has received much attention for its ability to reduce setup error and the required margins, thereby reducing the dose to the bowel in external beam radiotherapy for bladder cancer [9–13]. However, CBCT images have been qualitatively described as inferior to those of diagnostic CT, which may account for the uncertainty in delineating organ boundaries described in previous studies [14, 15].
Regarding delineating bladder boundaries on planning CT images, it was reported that the inter-observer variation was relatively small [16, 17], but few data are available about inter-observer variation on CBCT images. Most of the available data were reported in prostate cancer patients in a retrospective fashion, and the bladder was contoured as an organ at risk. These data could contain patient selection bias and gender bias, because some preparation protocols were applied to most of the prostate cancer patients and these patients were inevitably male. The bowel and bladder preparation protocol, such as voiding and collecting urine, defecating before treatment and endorectal balloon, is used to reduce factors of influence in interfraction motion, but these procedure may affect the delineation of the bladder on CBCT images. Moreover, the effect of organs peculiar to women (e.g., uterus and ovaries) in detecting organ boundaries with CBCT images was not considered.
To study image-guided radiotherapy for bladder cancer using CBCT, we conducted a prospective contouring protocol to analyze and evaluate the inter-observer contouring uncertainties of bladder boundary delineation on CBCT images with minimal preparation. We also analyzed the inter-observer contouring uncertainties of the prostate as the benchmark to link with previously published studies.
Patients’ and observers’ characteristics
Since April 2011, ten consecutive patients who were irradiated to the pelvic region were enrolled in this multiple-observer contouring study. The ethical committee of Hokkaido University Hospital approved this study (number 010-0305). Patients who had had prostatectomy were excluded. The individual patients’ characteristics are listed in Table 1. Of the five patients with bladder tumors, two patients received ureteral stents prior to radiotherapy. Fiducial markers were not placed in any of the patients.
Five physicians (four experienced radiation oncologists and one senior resident of the Department of Radiation Oncology who had worked in genito-urinary service) were recruited for the study (KN, RK, TI, SO, KY, and KH). The clinical experience of radiotherapy of all observers was ranged from 3 to 8 years with an average experience of 5.6 years.
CBCT image acquisition
Patients with bladder cancer were asked to void just before their treatment during the treatment course, and no other bowel or bladder preparation protocol including diet-related instruction was offered to any of the 10 patients. All CBCT datasets were acquired weekly in the supine position, immediately after initial setup to skin marks. CBCT images were not used to adjust the patient’s position in this study period.
All patients were imaged and treated on a Varian Clinac iX Linear Accelerator (Varian Medical Systems, Palo Alto, CA, USA) using the kV imaging system. The CBCT images were acquired using standard factory settings of 125 kVp, 80 mA, and 20 ms per projection with a half bow-tie filter. Images were reconstructed at an axial slice thickness of 0.25 cm.
For delineation of the organ boundaries, we used the first CBCT dataset of each patient that contained the entire bladder and prostate during the treatment course.
All observers were asked to delineate the outer contour of the whole bladder and prostate without margin for microscopic extension and seminal vesicles. In all cases the bladder was contoured as a solid organ. Contouring was performed in a blinded fashion, i.e., each observer could use only one image dataset of the patient at the time of delineation. Access to the structures drawn by other participants or the other imaging modalities (e.g., treatment planning CT, diagnostic CT, or MRI) as well as the help of a radiologist was not permitted. Contouring was carried out in the treatment planning system (Eclipse ver. 8.9, Varian Medical Systems, Inc.) using the standard tools available. Observers were free to modify window range and level of the images as preferred, and interpolation of the contours between slices was allowed. Intra-observer error was not investigated as part of this study.
Inter-observer variation analysis
The total encompassing delineated volume and the overlapping volume between the observers’ contours were calculated using the Eclipse planning system Boolean function.
To assess inter-observer variations in organ volumes, we calculated coefficients of variation (COV = standard deviation/mean volume) for the bladder and prostate. The COVs of all observers’ contours per patient were calculated and averaged over all patients.
To evaluate the inter-observer concordance, the generalized conformity index (CIgen), defined as the ratio of the sum of all overlapping volumes between pairs of observers and the sum of all overlapping and all non-overlapping volumes between the same pairs , was used, as follows:
A CIgen of 1 indicates 100% concordance for the volume segmentation, a CIgen of 0.5 indicates 50% agreement between observers for the encompassing volume, a CIgen of 0 indicates no concordance in delineation. The CIgens were calculated per patient and averaged over all patients.
Coordinates of the center-of-mass (COM) of each structure in 3D were also extracted. COM displacement values along the left-right (LR), anterior-posterior (AP), and cranial-caudal (CC) direction were analyzed. As the overall mean of standard deviation, the root mean square (RMS) of the total COM standard deviation (σ) on CBCT was calculated, as follows:
where σ i indicates the standard deviation of the COM displacement value of the structure in patient i drawn by the respective observer in a given direction.
To evaluate the reliability of this study, we calculated the intra-class correlation coefficients (ICC(2,k)), where k represents the number of observers. The ICC is a tool for reliability analysis, which is defined from the variance components as
where the subscripts ws and bs denote within-subject and between-subjects variance, respectively. As the true value of the variance is unknown, we use estimates from analysis of variance (ANOVA) analysis, which provides the variance components with respective mean squares between patient cases (MSbpat), within one patient case (MSwpat), between observers (MSobs), and between error terms (MSerr). As different forms of ICC are described in the literature, we selected ICC(2,k) for the situation in which some physicians (observers) of the department delineated organ boundaries in multiple patients, once for each patient. The ICC can be used to assess the overall reliability of k observers in contouring all n given cases (ICC(2,k)), as follows:
ICC values < 0.4 indicate poor reliability, ICC values between 0.4 and 0.6 indicate moderate reliability, and ICC values > 0.6 or 0.8 denote substantial or excellent reliability, respectively .
Statistical analysis was performed with JMP 9.0.3 (SAS Institute, Cary, NC, USA) and SPSS 11.5 (SPSS Inc., Chicago IL). Statistical significance of the outcome was assumed for p<0.05.
All observers were able to contour both the bladder and the prostate using the CBCT images. Figure 1 shows the variation between observers for a male patient and a female patient. The mean contoured volume (range of standard deviation of the volume) of the bladder for all patients was 32.4-204.0 cm3 (2.1-17.2 cm3). For the male patient, the mean volume of the prostate was 19.6-111.9 cm3 (4.0-7.9 cm3).
The average ICC(2,k) values of observers for the bladder was 0.9954. When separated by gender, the average ICC(2,k) values for male and female bladder was 0.9980 and 0.9873, respectively. This suggests correlation between the observers in both gender. The average ICC(2,k) values for the prostate was 0.9950.
The mean COV (± standard error of the mean) of the bladder and prostate was 0.08 (± 0.01) and 0.20 (± 0.04), respectively. Data of individual patients are shown in Table 2. The difference of COV between the bladder and prostate was statistically significant (p=0.0442). Regarding the mean bladder COV between the male patient and the female patient, the difference was not significant (0.07 for the male, 0.08 for the female, p=0.7745).
The mean CIgen (± standard error of the mean) of the bladder and prostate was 0.81 (± 0.02) and 0.66 (± 0.03), respectively (Figure 2). The difference of mean CIgen between the bladder and prostate was statistically significant (p=0.0038). The difference of mean bladder CIgen between the male patient and the female patient was not significant (0.80 for the male, 0.82 for the female, p=0.7099).
The RMS of the standard deviation (σ) of the inter-observer COM displacement was 0.79, 0.87, and 0.54 for the bladder and 0.63, 0.99, and 1.72 for the prostate in the LR, AP, and CC direction, respectively (Figure 3). Regarding the COM location for the bladder in terms of gender, σ was 0.89, 1.00, and 0.41 for males and 0.60, 0.64, and 0.68 for females in the LR, AP, and CC direction, respectively.
CBCT is an established strategy for 3D image guidance during treatment. It provides reasonable soft-tissue contrast and enables the verification of both target volume and organ at risk displacements. Prostate cancer is one of the most frequently targeted tumors using CBCT, and many studies have reported its efficacy [20, 21]. The authors of some of these studies reported that the accuracy of kilovoltage CBCT was similar to that of kV fiducial imaging for prostate patients with implanted gold fiducial markers [22, 23], but the subjective CBCT image quality was worse compared with that of diagnostic CT or MRI  and large inter-observer variability in organ boundary delineation was expected.
CBCT has been found to be useful, especially in organs expected to have large intrafractional error due to organ movement, such as the bladder, but up to now few data have been available about the accuracy of bladder delineation by CBCT. Foroudi et al. reported 4 patients with bladder cancer in whom the conformity index for CBCT was not significantly inferior to that of conventional planning CT in the contouring of the whole bladder as the clinical target volume (CTV) . However, most of the available data were reported in prostate cancer patients in whom the bladder was contoured as an OAR, and thus there could be some biases, such as patient selection, gender, and preparation protocol before each treatment. The aim of the present study was to analyze and evaluate the inter-observer contouring uncertainties of bladder boundary delineation on CBCT images in a prospective fashion.
There is no general consensus in the literature regarding the analysis of inter-observer variability in delineation. Recently, Fotina et al. reported common relationships between the different parameters reported and discussed the minimal set of parameters needed for “full description” of variability in delineation. They concluded that a combination of descriptive statistics, overlapping measurements, and statistical measures of agreement was required for a full reporting . We selected the COV and ICC(2,k) as parameters of descriptive statistics and statistical measures of agreement, and the CIgen as an indication of overlapping measurements as appropriate tool independent from the number of observers, following the suggestion of Kouwenhoven et al. .
The results of this study were in accordance with those of previous reports. Lütgendorf-Caucig et al. reported that the mean COV and CIgen for the bladder on CBCT imaging was 0.06 ± 0.02 and 0.82 ± 0.05, and RMS (σ) of the COM displacement for the bladder was smaller than 1mm in all directions. While for the prostate, the mean COV and CIgen was 0.24 ± 0.07 and 0.57 ± 0.09 and σ of the COM displacement was 0.4 mm (LR), 1.1 mm (AP), and 1.7 mm (CC), respectively . Weiss et al. reported the patient-averaged COV was 0.08 for the bladder and 0.19 for the prostate . White et al. reported the average standard deviation for COM displacements of the prostate was 0.7 mm (LR), 1.8 mm (AP), and 2.8 mm (CC) .
The limitation of this study is that the number of patients and observers was small especially when we separated them by gender. We could not find an apparent difference between males and females in either the mean bladder COV or the mean CIgen in our analysis but it is not conclusive. Regarding the σ of COM displacement, the significance of difference between males and females could not be statistically analyzed, but σ along each direction was quite small (equal to or less than 1 mm).
Inter-observer variability for bladder delineation on CBCT images was substantially small regardless of gender. We believe that our results support the applicability of CBCT in adaptive radiotherapy for bladder cancer.
Cone-beam computed tomography
Coefficients of variation
Generalized conformity index
Image-guided radiation therapy
Magnetic resonance imaging
Root mean square
Intra-class correlation coefficients.
Turner SL, Swindell R, Bowl N, Marrs J, Brookes B, Read G, Cowan RA: Bladder movement during radiation therapy for bladder cancer: implications for treatment planning. Int J Radiat Oncol Biol Phys 1997, 39: 355-360.
Harris SJ, Buchanan RB: An audit and evaluation of bladder movements during radical radiotherapy. Clin Oncol 1998, 10: 262-264. 10.1016/S0936-6555(98)80014-9
Zietman A, Skinner E: Quality of life after radical treatment for invasive bladder cancer. Semin Radiat Oncol 2005, 15: 55-59. 10.1016/j.semradonc.2004.07.013
Efstathiou J a, Bae K, Shipley WU, Kaufman DS, Hagan MP, Heney NM, Sandler HM: Late pelvic toxicity after bladder-sparing therapy in patients with invasive bladder cancer: RTOG 89–03, 95–06, 97–06, 99–06. J Clin Oncol: official J Am Soc of Clin Oncol 2009, 27: 4055-4061. 10.1200/JCO.2008.19.5776
Thariat J, Aluwini S, Pan Q, Caullery M, Marcy P-Y, Housset M, Lagrange J-L: Image-guided radiation therapy for muscle-invasive bladder cancer. Nat Rev Urol 2012, 9: 23-29.
Pos F, Remeijer P: Adaptive management of bladder cancer radiotherapy. Semin Radiat Oncol 2010, 20: 116-120. 10.1016/j.semradonc.2009.11.005
Shirato H, Shimizu S, Kitamura K, Nishioka T, Kagei K, Hashimoto S, Aoyama H, Kunieda T, Shinohara N, Dosaka-Akita H, Miyasaka K: Four-dimensional treatment planning and fluoroscopic real-time tumor tracking radiotherapy for moving tumor. Int J Radiat Oncol Biol Phys 2000, 48: 435-442. 10.1016/S0360-3016(00)00625-8
Shimizu S, Shirato H, Kitamura K, Shinohara N, Harabayashi T, Tsukamoto T, Koyanagi T, Miyasaka K: Use of an implanted marker and real-time tracking of the marker for the positioning of prostate and bladder cancers. Int J Radiat Oncol Biol Phys 2000, 48: 1591-1597. 10.1016/S0360-3016(00)00809-9
Tuomikoski L, Collan J, Keyriläinen J, Visapää H, Saarilahti K, Tenhunen M: Adaptive radiotherapy in muscle invasive urinary bladder cancer–an effective method to reduce the irradiated bowel volume. Radio Oncol: J Eur Society for Therapeutic Rad Oncol 2011, 99: 61-66. 10.1016/j.radonc.2011.02.011
Lalondrelle S, Huddart R, Warren-Oseni K, Hansen VN, McNair H, Thomas K, Dearnaley D, Horwich A, Khoo V: Adaptive-predictive organ localization using cone-beam computed tomography for improved accuracy in external beam radiotherapy for bladder cancer. Int J Radiat Oncol Biol Phys 2011, 79: 705-712. 10.1016/j.ijrobp.2009.12.003
Murthy V, Master Z, Adurkar P, Mallick I, Mahantshetty U, Bakshi G, Tongaonkar H, Shrivastava S: “Plan of the day” adaptive radiotherapy for bladder cancer using helical tomotherapy. Rad oncol: J Eur Soc for Therapeutic Rad Oncol 2011, 99: 55-60. 10.1016/j.radonc.2011.01.027
Foroudi F, Wong J, Kron T, Rolfo A, Haworth A, Roxby P, Thomas J, Herschtal A, Pham D, Williams S, Tai KH, Duchesne G: Online adaptive radiotherapy for muscle-invasive bladder cancer: Results of a pilot study. Int J Radiat Oncol Biol Phys 2011, 81: 765-771. 10.1016/j.ijrobp.2010.06.061
Button MR, Staffurth JN: Clinical application of image-guided radiotherapy in bladder and prostate cancer. Clin Oncol 2010, 22: 698-706. 10.1016/j.clon.2010.06.020
Lütgendorf-Caucig C, Fotina I, Stock M, Pötter R, Goldner G, Georg D: Feasibility of CBCT-based target and normal structure delineation in prostate cancer radiotherapy: multi-observer and image multi-modality study. Rad Oncol: J Eur Society for Therapeutic Rad Oncol 2011, 98: 154-161. 10.1016/j.radonc.2010.11.016
Weiss E, Wu J, Sleeman W, Bryant J, Mitra P, Myers M, Ivanova T, Mukhopadhyay N, Ramakrishnan V, Murphy M, Williamson J: Clinical evaluation of soft tissue organ boundary visualization on cone-beam computed tomographic imaging. Int J Radiat Oncol Biol Phys 2010, 78: 929-936. 10.1016/j.ijrobp.2010.02.007
Logue J, Sharrock C, Cowan R: Clinical variability of target volume description in conformal radiotherapy planning. Int J Radiat Oncol Biol Phys 1998, 41: 929-931. 10.1016/S0360-3016(98)00148-5
Meijer G, Rasch C, Remeijer P, Lebesque J: Three-dimensional analysis of delineation errors, setup errors, and organ motion during radiotherapy of bladder cancer. Int J Radiat Oncol Biol Phys 2003, 55: 1277-1287. 10.1016/S0360-3016(02)04162-7
Kouwenhoven E, Giezen M, Struikmans H: Measuring the similarity of target volume delineations independent of the number of observers. Phys Med Biol 2009, 54: 2863-2873. 10.1088/0031-9155/54/9/018
Fotina I, Lütgendorf-Caucig C, Stock M, Pötter R, Georg D: Critical discussion of evaluation parameters for inter-observer variability in target definition for radiation therapy. Strahlentherapie und Onkologie: Organ der Deutschen Röntgengesellschaft…[et al.] 2012, 188: 160-167. 10.1007/s00066-011-0027-6
Boda-Heggemann J, Lohr F, Wenz F, Flentje M, Guckenberger M: kV cone-beam CT-based IGRT: a clinical review. Strahlentherapie und Onkologie: Organ der Deutschen Röntgengesellschaft…[et al.] 2011, 187: 284-291. 10.1007/s00066-011-2236-4
Palombarini M, Mengoli S, Fantazzini P, Cadioli C, Degli Esposti C, Frezza GP: Analysis of inter-fraction setup errors and organ motion by daily kilovoltage cone beam computed tomography in intensity modulated radiotherapy of prostate cancer. Radiat Oncol (London, England) 2012, 7: 56. 10.1186/1748-717X-7-56
Moseley DJ, White E a, Wiltshire KL, Rosewall T, Sharpe MB, Siewerdsen JH, Bissonnette J-P, Gospodarowicz M, Warde P, Catton CN, Jaffray D a: Comparison of localization performance with implanted fiducial markers and cone-beam computed tomography for on-line image-guided radiotherapy of the prostate. Int J Radiat Oncol Biol Phys 2007, 67: 942-953. 10.1016/j.ijrobp.2006.10.039
Barney BM, Lee RJ, Handrahan D, Welsh KT, Cook JT, Sause WT: Image-guided radiotherapy (IGRT) for prostate cancer comparing kV imaging of fiducial markers with cone beam computed tomography (CBCT). Int J Radiat Oncol Biol Phys 2011, 80: 301-305. 10.1016/j.ijrobp.2010.06.007
Foroudi F, Haworth a, Pangehel a, Wong J, Roxby P, Duchesne G, Williams S, Tai KH: Inter-observer variability of clinical target volume delineation for bladder cancer using CT and cone beam CT. J Med Imaging Radiat Oncol 2009, 53: 100-106. 10.1111/j.1754-9485.2009.02044.x
White E a, Brock KK, Jaffray D a, Catton CN: Inter-observer variability of prostate delineation on cone beam computerised tomography images. Clinical oncology (Royal College of Radiologists (Great Britain)) 2009, 21: 32-38. 10.1016/j.clon.2008.11.007
This research was supported by a grant from the Ministry of Education, Science, Sports and Culture, Japan (No. 24591829) and the Japan Society for the Promotion of Science (JSPS) through the “Funding Program for World-Leading Innovative R&D on Science and Technology” (FIRST Program).
The authors declare that they have no competing interests.
KN conceived of the study, participated in the design of the study, carried out the treatment planning, participated in data collection and interpretation and in drafting and final revising of the manuscript, and performed the statistical analysis. TI, SO, KY, and KH carried out the treatment planning, participated in data interpretation and in drafting and final revising of the manuscript. RK and YN participated in data collection and interpretation and in drafting the manuscript. SS and RO participated in the design of the study and in drafting and final revising of the manuscript. HS participated in study design and coordination and in drafting and final revising of the manuscript. All authors read and approved the final manuscript.