Quantitative assessment of inter-observer variability in target volume delineation on stereotactic radiotherapy treatment for pituitary adenoma and meningioma near optic tract
© Yamazaki et al; licensee BioMed Central Ltd. 2011
Received: 2 December 2010
Accepted: 27 January 2011
Published: 27 January 2011
To assess inter-observer variability in delineating target volume and organs at risk in benign tumor adjacent to optic tract as a quality assurance exercise.
We quantitatively analyzed 21 plans made by 11 clinicians in seven CyberKnife centers. The clinicians were provided with a raw data set (pituitary adenoma and meningioma) including clinical information, and were asked to delineate the lesions and create a treatment plan. Their contouring and plans (10 adenoma and 11 meningioma plans), were then compared. In addition, we estimated the influence of differences in contouring by superimposing the respective contours onto a default plan.
The median planning target volume (PTV) and the ratio of the largest to the smallest contoured volume were 9.22 cm3 (range, 7.17 - 14.3 cm3) and 1.99 for pituitary adenoma, and 6.86 cm3 (range 6.05 - 14.6 cm3) and 2.41 for meningioma. PTV volume was 10.1 ± 1.74 cm3 for group 1 with a margin of 1 -2 mm around the CTV (n = 3) and 9.28 ± 1.8 cm3(p = 0.51) for group 2 with no margin (n = 7) in pituitary adenoma. In meningioma, group 1 showed larger PTV volume (10.1 ± 3.26 cm3) than group 2 (6.91 ± 0.7 cm3, p = 0.03). All submitted plan keep the irradiated dose to optic tract within the range of 50 Gy (equivalent total doses in 2 Gy fractionation). However, contours superimposed onto the dose distribution of the default plan indicated that an excessive dose 23.64 Gy (up to 268% of the default plan) in pituitary adenoma and 24.84 Gy (131% of the default plan) in meningioma to the optic nerve in the contours from different contouring.
Quality assurance revealed inter-observer variability in contour delineation and their influences on planning for pituitary adenoma and meningioma near optic tract.
Target delineation is an important issue in radiation oncology, especially for image-guided, high-precision radiotherapy . With increasing conformity of dose delivery, inter-observer variability in tumor identification and delineation plays an ever more critical role, even for uncomplicated lesions [2–6]. Although inter-observer variability in contouring is a well-known fact, we could not find any data on variability in the contouring of benign tumors near the optic tract.
Pituitary adenoma and meningioma are regarded as benign tumors and are rarely treated by radiotherapy if surgery can be performed. However, in cases that are ineligible for surgery due to a risk of excessive surgical complications, radiotherapy can play an important role in treatment for these benign tumors. Radiotherapy was initially performed using conventional technologies (Co-60 or Linac-based units) [7, 8]. Stereotactic, single-fraction radiosurgery (SRS) using the Gamma knife was then begun, followed shortly thereafter by stereotactic radiotherapy (SRT) at a number of institutes . The SRT experience for such tumors has been insufficient to develop a consensus on optimal treatment parameters, including prescribed dose and fractionation, especially for hypofractionated SRT. This lack of consensus applies as well to contouring of the planning target volume (PTV).
Therefore, we conducted a multi-institutional study in which participating radiation oncologists delineated tumors and organs at risk (OARs) and created treatment plans using inverse planning software for the CyberKnife System (Accuray, Sunnyvale CA, USA). Participants created treatment plans for two patients, one with pituitary adenoma and second with meningioma. Variability in contouring, planning target volumes, prescribed doses, and doses to OARs was assessed. In addition, we examined the influence of different contouring especially optic tract by superimposing each contour onto the default plan, and we visualized dose distribution using prescribed dose 3-D rendering.
Enhanced CT images for Case 1 (pituitary adenoma) and CT and MRI images for Case 2 (meningioma) were obtained at Soseikai General Hospital and sent via internet to seven CyberKnife institutes. For Case 1, CT images were acquired with a SOMATOM Volume Access scanner (Siemens AG, Munchen, Germany) at a 2-mm slice thickness. For Case 2, Contrast enhanced CT images were obtained with a Brilliance CT 64 scanner (Royal Philips Electronics, Euronext: PHIA, Eindhoven, Holland) at a 1.25-mm slice thickness (Default; CT level 35, window 75). MRI images were obtained by an Achieve 1.5 scanner (Royal Philips Electronics, Euronext: PHIA, Eindhoven, Holland) using a 4-mm slice gapless scan (TE 10 ms, TR 450 ms, FA 70°, SPIR). At each CyberKnife institute, the images were transferred to a treatment planning system (TPS; MultiPlan or OnTarget, Accuray) to create a radiotherapy plan for CyberKnife stereotactic radiotherapy (SRT). Participating physicians were required to submit both the printed materials used in their routine clinical work and raw data.
From seven CyberKnife centers, 11 radiation oncologists submitted plans and raw data for the meningioma and 10 for the pituitary adenoma. The collected data contained target volume contours, organs at risk (OARs), and minimum and maximum irradiated dose for those structures. Maximum and minimum doses for the PTV and the maximum dose for OARs were analyzed. Uniformity of dose distribution was assessed in terms of the minimum and maximum prescribed dose for the PTV.
The raw treatment plan data in TPS format were also submitted and analyzed using ShioRIS and ShioRIS-2 (software developed in-house by author H. S.). We superimposed those contours on our default plan (created by author T. T. in Soseikai General Hospital and confirmed by other two physicians), and examined differences in the dose-volume histogram (DVH) for each contour to estimate a prescribed dose for each contoured PTV and organs at risk. The equation; equivalent total doses, EQD2 = n × d × (α/β + d)/(α/β + 2); the dose that would be equivalent to a 2 Gy fractionation was used for the calculation, with EQD210; α/β = 10 for PTV and EQD22; α/β = 2 for OARs. Next, comparison of dose distribution and dose volume renderings for the prescribed dose were analyzed for the pituitary adenoma using ShioRIS-2 in 9 contours.
Reference for planning of pituitary adenoma and meningioma
PTV = CTV + 1 mm (CTV = GTV),
PTV = CTV + 1 mm (CTV = GTV)
PTV = CTV + 2 - 4 mm (CTV = GTV)
PTV = CTV + 2 - 3 mm (CTV = GTV)
15 - 20 Gy marginal dose, 25 Gy or more for secreting pituitary adenoma
11 - 18 Gy marginal dose (recommended for 14 Gy or more)
Conventional fractionated SRT
isocenter 45 - 68 Gy/daily 1.8 Gy/fr., D95 50 - 56 Gy/daily 2 Gy/fr.
45 - 50 Gy/25 - 28 fr.
Constrains for organs at risk 
Optic tract < 8 - 10 Gy,
Optic tract 50 Gy/25 fr
Spinal cord < 50 Gy (10 cm or less in length)
Retina < 45 Gy
Lens < 10 Gy
Brain stem < 60 Gy (1/3 volume)
Case 1. Pituitary adenoma
No of plan
Volume of PTV
8.06 ± 2.45
9.53 ± 1.75
30 Gy/5 fr.
25 Gy/5 fr.
16 Gy/1 fr.
21 Gy/3 fr.
21 Gy/3 fr.
22.5 Gy/3 fr.
24 Gy/5 fr.
24 Gy/5 fr.
23 Gy/3 fr.
24 Gy/3 fr.
24 Gy/3 fr.
Minimal dose/prescribed dose
83.7 ± 9 range 72 - 90
80 ± 12 range 60 - 99
Maximal dose/prescribed dose
122 ± 15 range 110 - 157
129 ± 17 range 105 - 157
0.12 - 2.25
0.02 - 4.48
3.07 - 18.6
0.05 - 4.52
4.97 - 15.4
4.5 - 19.5
17.3 - 24.74
30.0 - 42.9
4.61 - 15.4
4.1 - 19.9
16.6 - 26.4
22 - 48
0.01 - 2.00
0.03 - 4.08
0.88 - 6.68
0.13 - 3.81
Left optic nerve
0.79 - 8.96
0.4 - 8.5
12.4 - 23.4
23 - 50
Right optic nerve
7.93 - 26.4
14.6 - 47.9
0.37 - 23.4
17.5 - 57.5
Case 2. Meningioma
This patient is a 50-year-old female with sphenoid ridge meningioma. She experienced back pain while performing nursing care for her mother three years before, and was diagnosed at the time as having a meningioma. During several years of follow-up the tumor grew, eventually requiring surgical intervention. Thereafter, a residual tumor grew slowly and she was recommended for further treatment with the CyberKnife. She received SRT using 30 Gy in 5 fractions (6 Gy × 5 times in consecutive 5 days) for D90. Default plan used CTV = GTV and PTV = CTV + 1 mm. Conformity index was 1.12. Those plans were verified by other two physicians, and used as a control references.
All statistical analyses were carried out with the Statview-v5.0 software program. Student's t-test was used for normally distributed data and the Mann Whitney U-test for skewed data. Percentages were analyzed with the Chi-square test. A value of p < 0.05 was considered to be statistically significant.
Case 1. pituitary adenoma
Case 2. meningioma
Inter-observer variation is a well-known problem in medical practice. Gardenia et al. first reported on this issue in the 1950 s , and it became a subject for discussion in the radiotherapeutic community in the 1970 s. In the 1990 s, many articles were published about inter-observer variation for a variety of cancers: prostate cancer , brain tumors , breast cancer  head and neck cancer [12, 13], and lung cancer [14, 15]. However, we were unable to find any papers that examined inter-observer variation for pituitary adenoma and meningioma; to the best of our knowledge, this is the first such report.
DVHs analysis by superimposing different contours from multiple clinicians onto the default treatment plan showed higher maximal dose for optic tract (Figure 3). It was increased to 23.64 Gy (268% higher dose than default plan) for the pituitary adenoma and 19.39 Gy (131%) for the meningioma. These results imply that contour deviations across plans could easily cause unexpectedly higher doses to OARs. On the other hand, some comparison plans prescribed 19% lower does than the default 24.24 Gy in the meningioma. Although the dose to PTV is not a matter of this study because it will be changed by physician's decision (PTV definition etc.), we can suggest that there are such a variety of different SRT plans using same CT images.
Several limitations should be considered in our study. At first, BED assessment is not validated in hypofractionated SRT, however it is an only method to compare different fractionation quantitatively at present. Next, although we used default plan as a control references after confirmed by other two physicians, there is neither consensus in contouring nor planning in these area, so that in fact it is only simulation examination. Thirdly, although we confirmed precision of fusion software by visual inspection at least by other two physicians, accuracy of fusion is still qualified by subjective methods.
To obtain reproducible outcomes using an inverse plan, consensus among the participants should be reached in advance to avoid uncertainty; for example, definitions of major violations should be provided and training sessions made available for participants to improve the conformity of their plans to an agreed upon benchmark. These results underline the importance of QA assessment for reproducible outcomes, not only in contouring and the setting of dose constraints, but also for planned dose distributions especially in a multi-clinician study. We should keep in mind the risk of such new techniques as cyberknife if the QA is not followed.
In conclusion, quality assurance revealed inter-observer variability in contour delineation of pituitary adenoma and meningioma near optic tract.
The authors wish to thank Ms. Hitomi Fuse, Mr. Yoshiichi Murashima, Mr. Naokazu Higashinaka, and Mr. Yoshiaki Furutani for their dedicated contributions to this manuscript.
- Riboldi M, Sharp GC, Baroni G, Chen GT: Four-dimensional targeting error analysis in image-guided radiotherapy. Phys Med Biol 2009, 54: 5995-6008. 10.1088/0031-9155/54/19/022View ArticlePubMedGoogle Scholar
- Yamazaki H, Nishiyama K, Tanaka E, Koiwai K, Shikama N, Ito Y, Arahira S, Tamamoto T, Shibata T, Tamaki Y, Kodaira T, Oguchi M: Dummy run for a phase II multi-institute trial of chemoradiotherapy for unresectable pancreatic cancer: inter-observer variance in contour delineation. Anticancer Res 2007, 27: 2965-71.PubMedGoogle Scholar
- Gandevia B, Stradling P: Observer variation in the tomographic diagnosis of tuberuculous cavitation. Tubercle 1957, 38: 113-22. 10.1016/S0041-3879(57)80005-1View ArticlePubMedGoogle Scholar
- Villeirs GM, Van Vaerenbergh K, Vakaet L, Bral S, Claus F, De Neve WJ, Verstraete KL, De Meerleer GO: Interobserver delineation variation using CT versus combined CT + MRI in intensity-modulated radiotherapy for prostate cancer. Strahlenther Onkol 2005, 181: 424-30. 10.1007/s00066-005-1383-xView ArticlePubMedGoogle Scholar
- Weltens C, Menten J, Feron M, Weltens C, Menten J, Feron M, Bellon E, Demaerel P, Maes F, Van den Bogaert W, van der Schueren E, et al.: Interobserver variations in gross tumor volume delineation of brain tumors on computed tomography and impact of magnetic resonance imaging. Radiother Oncol 2001, 60: 49-59. 10.1016/S0167-8140(01)00371-1View ArticlePubMedGoogle Scholar
- Poortmans PM, Venselaar JL, Struikmans H, Hurkmans CW, Davis JB, Huyskens D, van Tienhoven G, Vlaun V, Lagendijk JJ, Mijnheer BJ, De Winter KA, Van der Hulst MH, Van den Bogaert WF: The potential impact of treatment variations on the results of radiotherapy of the internal mammary lymph node chain: a quality-assurance report on the dummy run of EORTC Phase III randomized trial 22922/10925 in Stage I--III breast cancer. Int J Radiat Oncol Biol Phys 2001, 49: 1399-408. 10.1016/S0360-3016(00)01549-2View ArticlePubMedGoogle Scholar
- Stafford SL, Pollock BE, Foote RL, Link MJ, Gorman DA, Schomberg PJ, Leavitt JA: Meningioma radiosurgery: tumor control, outcomes, and complications among 190 consecutive patients. Neurosurgery 2001, 49: 1029-1037. 10.1097/00006123-200111000-00001PubMedGoogle Scholar
- Erridge SC, Conkey DS, Stockton D, Strachan MW, Statham PF, Whittle IR, Grant R, Kerr GR, Gregor A: Radiotherapy for pituitary adenomas: long-term efficacy and toxicity. Radiother Oncol 2009, 93: 597-601. 10.1016/j.radonc.2009.09.011View ArticlePubMedGoogle Scholar
- Milker-Zabel S, Debus J, Thilmann C, Schlegel W, Wannenmacher M: Fractionated stereotactically guided radiotherapy and radiosurgery in the treatment of functional and nonfunctional adenomas of the pituitary gland. Int J Radiat Oncol Biol Phys 2001, 50: 1279-1286. 10.1016/S0360-3016(01)01535-8View ArticlePubMedGoogle Scholar
- Guideline for radiotherapy planningJapanese society for therapeutic radiology and oncology Japanese College of Radiology, Japan radiological society; [http://www.kkr-smc.com/rad/guideline/2008/]
- Emami B, Lyman J, Brown A, Coia L, Goitein M, Munzenrider JE, Shank B, Solin LJ, Wesson M: Tolerance of normal tissue to therapeutic irradiation. Int J Radiat Oncol Biol Phys 1991, 21: 109-22.View ArticlePubMedGoogle Scholar
- Jeanneret-Sozzi W, Moeckli R, Valley JF, Zouhair A, Ozsahin EM, Mirimanoff RO, SASRO: The reasons for discrepancies in target volume delineation: a SASRO study on head-and-neck and prostate cancers. Strahlenther Onkol 2006, 182: 450-7. 10.1007/s00066-006-1463-6View ArticlePubMedGoogle Scholar
- Geets X, Daisne JF, Arcangeli S, Coche E, De Poel M, Duprez T, Nardella G, Grégoire V: Inter-observer variability in the delineation of pharyngo-laryngeal tumor, parotid glands and cervical spinal cord: comparison between CT-scan and MRI. Radiother Oncol 2005, 77: 25-31. 10.1016/j.radonc.2005.04.010View ArticlePubMedGoogle Scholar
- Steenbakkers RJ, Duppen JC, Fitton I, Deurloo KE, Zijp L, Uitterhoeve AL, Rodrigus PT, Kramer GW, Bussink J, De Jaeger K, Belderbos JS, Hart AA, Nowak PJ, van Herk M, Rasch CR: Observer variation in target volume delineation of lung cancer related to radiation oncologist-computer interaction: a 'Big Brother' evaluation. Radiother Oncol 2005, 77: 182-90. 10.1016/j.radonc.2005.09.017View ArticlePubMedGoogle Scholar
- van Sornsen JR, Senan S, Underberg RW, Oei SS, Elshove D, Slotman BJ, Lagerwaard FJ: Use of CD-ROM-based tool for analyzing contouring variations in involved-field radiotherapy for stage III NSCLC. Int J Radiat Oncol Biol Phys 2005, 63: 334-339. 10.1016/j.ijrobp.2005.02.016View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.