Modeling of artificial intelligence-based respiratory motion prediction in MRI-guided radiotherapy: a review

Abstract

The advancement of precision radiotherapy techniques, such as volumetric modulated arc therapy (VMAT), stereotactic body radiotherapy (SBRT), and particle therapy, highlights the importance of radiotherapy in the treatment of cancer, while also posing challenges for respiratory motion management in thoracic and abdominal tumors. MRI-guided radiotherapy (MRIgRT) stands out as state-of-art real-time respiratory motion management approach owing to the non-ionizing radiation nature and superior soft-tissue contrast characteristic of MR imaging. In clinical practice, MR imaging often operates at a frequency of 4 Hz, resulting in approximately a 300 ms system latency of MRIgRT. This system latency decreases the accuracy of respiratory motion management in MRIgRT. Artificial intelligence (AI)-based respiratory motion prediction has recently emerged as a promising solution to address the system latency issues in MRIgRT, particularly for advanced contour prediction and volumetric prediction. However, implementing AI-based respiratory motion prediction faces several challenges including the collection of training datasets, the selection of prediction methods, and the formulation of complex contour and volumetric prediction problems. This review presents modeling approaches of AI-based respiratory motion prediction in MRIgRT, and provides recommendations for achieving consistent and generalizable results in this field.

Background

The advancement of precision radiotherapy techniques, such as volumetric modulated arc therapy (VMAT), stereotactic body radiotherapy (SBRT), and particle therapy, allows for the delivery of highly conformal doses to targets. However, the delivery of highly conformal doses to targets in the abdomen and thorax is challenging due to the respiratory motion affecting the treatment [1, 2]. To guide the implementation of respiratory motion management, the American Association of Physicists in Medicine (AAPM) has released several relevant guidelines, including TG-76 report [3] for photon therapy, TG-101 report [4] for SBRT, and TG-290 report [5] for particle therapy. These guidelines provide comprehensive summaries of intra-fractional respiratory motion management approaches, such as abdominal compression [6], breath hold [7], respiratory beam gating [8], and tumor tracking, which comprises both robotic, gimbaled and multi-leaf-collimator (MLC)-based tracking [9, 10]. Among these respiratory motion management approaches, real-time tumor tracking captured the most widespread attention for its precision and improved efficiency compared to, for instance, gating [11].

MRI-guided radiotherapy (MRIgRT), integrating MR imaging with a medical linear accelerator, stands out as state-of-art real-time motion management approach owing to the non-ionizing radiation nature and superior soft-tissue contrast characteristic of MR imaging [12, 13]. While clinical MRIgRT currently relies on gating [14, 15], several research studies have focused on the implementation of MLC-tracking [16,17,18]. However, the implementation of real-time tumor tracking in MRIgRT is associated with approximately a 300–350 ms system latency [17, 19], while the system latencies of real-time tumor tracking systems were 115 ms for a robotic linac system [20] and 48 ms for a gimbaled linac system [21]. Therefore, to avoid a decrease in dosimetry accuracy, it is important to implement respiratory motion prediction for the compensation of system latency in MRIgRT. Furthermore, the superior soft-tissue contrast characteristic provided by MR imaging facilitates target localization, thereby allowing for the extension of respiratory motion prediction from prediction of rigid shifts to two-dimensional (2D) contour or three-dimensional (3D) volume [19]. Notably, the implementation of contour and volumetric prediction mainly relies on artificial intelligence (AI)-based methods, which highlights the potential of AI-based respiratory motion prediction for MRIgRT [22].

However, previous studies of AI-based respiratory motion prediction for MRIgRT exhibited significant inconsistencies in the collection of training dataset, the selection of prediction methods, and the formulation of complex contour and volumetric prediction problems. For example, the difficulty of data collection limits the size of patient samples, resulting in a lack of diversity in respiratory motion patterns within the training datasets, which detrimentally affects the generalizability of prediction methods [23]. Theoretically, compared to linear predictors, complex recurrent neural networks (RNNs) require larger training datasets to capture temporal dependencies in training datasets [24]. In other words, the performance of prediction methods is associated with the characteristics of training datasets, which might be a potential explanation for the inconsistent results observed across different studies. This indicates the importance of data homogenization in AI-based respiratory motion prediction for MRIgRT. Therefore, this paper aims to provide a comprehensive review on the modeling approaches of AI-based respiratory motion prediction for MRIgRT, and to discuss potential solutions for achieving consistent results in this domain.

Literature search

Following the Preferred Reporting Items for systematic reviews and meta-analyses (PRISMA) guidelines [25], this study was conducted as a systematic review focusing on AI-based respiratory motion prediction (Fig. 1). The search terms used in the PubMed database were (magnetic resonance imaging) AND (motion tracking) AND ((radiotherapy) OR (radiation oncology)), resulting in a total of 203 articles until June 2024. The relevant studies were selected by screening the title and abstract of these articles according to these criteria: (1) design for MRIgRT; (2) utilization of AI algorithms; and (3) involvement of respiratory motion prediction. Subsequently, we searched the references and citations of these relevant studies in the Google Scholar search engine, finding a total of 12 studies within the scope of AI-based respiratory motion prediction.

Fig. 1

Flowchart of article identification, screening, and inclusion criteria for studies on AI-based respiratory motion prediction in MRIgRT

Problem definition

The aim of respiratory motion prediction is to obtain future information on respiratory motion from current data, thereby compensating for the system latency in real-time. Formally, respiratory motion prediction tasks can be formulated as

$${\hat y_{t + \Delta t}} = f({x_t})$$

(1)

where $\Delta t$ is a prediction method, $\:{x}_{t}$ is the current respiratory motion data, ${\hat y_{t + \Delta t}}$ is the future motion, and $\Delta t$ is the prediction window of the motion prediction task, typically corresponding to the system latency of real-time beam adaptation systems.

In MRIgRT, respiratory motion prediction can be classified into shift (i.e. centroid position) prediction, contour prediction, and volumetric prediction. The aims of shift prediction, contour prediction, and volumetric prediction are to obtain the future centroid position, 2D contour, and 3D volume of tumors, respectively. For contour and volume prediction, deformation vector fields (DVFs) or directly contours/volumes can be output by the model. Notably, the ultimate goal of MRIgRT is to achieve real-time 3D motion management, requiring 3D motion prediction [19].

Data characteristics

Input and output data

The input data $\:{\varvec{x}}_{t}$ for AI-based respiratory motion prediction in MRIgRT is the current centroid shift/the 2D contour of tumors extracted from 2D cine MR images [26, 27] or the 2D cine MR images themselves. Accordingly, the AAPM TG-264 report [10] recommended a minimum imaging frequency of 3 Hz for real-time tumor tracking systems to uphold end-to-end system latencies below 500 ms. In clinical practice, 2D cine MR imaging often operates at a frequency of 4–8 Hz, typically in a sagittal plane [28] or across interleaved orthogonal planes [29]. For the purpose of training and prediction, the input data $\:X=\left\{{x}_{1},\:{x}_{2},\:\dots\:,{x}_{t},\dots\:,{x}_{T}\right\}$ is reconstructed into subsequences of length $\:w$ at each timestep. For example, the subsequence at timestep $\:t$ is $\:S\:=\left\{{x}_{t\:-\:w},{x}_{t\:-\:w\:+\:1},{\dots\:,x}_{t}\right\}$. Notably, the length $\:w$ of subsequences often serves as a hyper-parameter during model optimization.

The output data ${\hat y_{t + \Delta t}}$ for respiratory motion prediction in MRIgRT is the future centroid position (from which the shift to the current centroid can be derived) or the future 2D tumor contour/cine MR image. The prediction window $\Delta t$ is chosen to match the system latencies of real-time tumor tracking systems. As shown in Table 1, reported end-to-end system latencies for MRIgRT were on average about 300 ms for 4 Hz imaging [16,17,18, 30, 31] and 200 ms for 8 Hz imaging [16, 17, 32]. Furthermore, Liu et al. [33] developed a real-time distortion correction method for correcting geometric distortions in MR images, and reported an end-to-end system latency of 319 ± 12 ms without distortion correction and 335 ± 34 ms with distortion correction. Therefore, the prediction window $\Delta t$ for AI-based respiratory motion prediction in MRIgRT is often set to 250 ms and 500 ms for a typical 4 Hz imaging frequency and can be adapted to the specific latency of the system via linear interpolation [28, 34].

Table 1 The reported tracking or beam gating end-to-end system latency for different MRIgRT systems. (*) asterisk denotes research prototypes

Diversity in respiratory motion patterns

The generalizability of AI-based systems heavily relies on the diversity of training datasets [35]. However, the diversity of training datasets is typically achieved by collecting a large amount of data, which is challenging in MRIgRT as most radiotherapy patients receive treatment on conventional linacs. Therefore, a quantitative description of respiratory motion patterns is needed in AI-based respiratory motion prediction for MRIgRT. For shift prediction, variability in the training data can be inferred by calculating the mean amplitude, period, and speed of respiratory motions [36,37,38,39,40,41]. For contour prediction, in addition to the previous values, multiple tumor sites, such as lung, pancreas, heart, liver, and mediastinum should be used [28], aligning with the fact that respiratory motion varies among tumor sites and can lead to different deformations/rotations of the irradiation target [42].

The regularity of respiratory motion also has significant impact on the performance of AI-based respiratory motion prediction [43]. For traditional real-time tumor tracking systems, Ernst et al. [44] analyzed 304 respiratory motion traces, extracted a total of 21 features for each trace to represent the regularity of motion and confirmed the correlations between these extracted features and the prediction performance of 6 prediction methods. In MRIgRT, the performance of AI-based respiratory motion prediction during irregular motion was investigated through case analyses with limited patient sample size [45].

Evaluation metrics

For shift prediction, the evaluation metrics mainly comprise mean absolute error (MAE) and root mean square error (RMSE). For contour prediction, the evaluation metrics include the Dice Similarity Coefficient (DSC) [46] and the Hausdorff distance [47]. For volumetric prediction, the reported evaluation metric is target registration error (TRE) [48].

The MAE quantifies the mean absolute difference between predicted and ground truth values:

$$MAE = \frac{1}{N}\sum\limits_{i = 1}^N {\left| {{y_i} - {{\hat y}_i}} \right|}$$

(2)

where $\:{\widehat{y}}_{i}$ and $\:{y}_{i}$ are the predicted and ground truth centroid data, respective. $\:N$ is the total number of points in the motion trace.

The RMSE is a metric used to quantify the difference between predicted and ground truth values:

$$RMSE = \sqrt {\frac{1}{N}} {\sum\limits_{i = 1}^N {({y_i} - {{\hat y}_i})} ^2}$$

(3)

The DSC is a metric used to measure the spatial overlap between the predicted contour and ground truth contour of tumors:

$$DSC(A,B) = \frac{{2(A \cap B)}}{{A + B}}$$

(4)

where A and B refer to the predicted contour and ground truth contour of tumors, respectively. The DSC ranges from 0 to 1, with 1 indicating perfect overlap and 0 indicating no overlap between the two sets.

The Hausdorff distance measures the similarity between the points in the predicted contour and ground truth contour:

$$H(A,B) = \max (h(A,B),h(B,A))$$

(5)

$$h(A,B) = \mathop {\max }\limits_{a \in A} \,\mathop {\min }\limits_{b \in B} \left\| {a - b} \right\|$$

(6)

where $\:A=\{{a}_{1},\dots\:,{a}_{p}\}$, $\:B=\{{b}_{1},\dots\:,{b}_{p}\}$, and $\:∥\cdot\:∥$∥\cdot\:∥\) is some underlying norm on the points of A and B (e.g., the L2 or Euclidean norm) [47]. The Hausdorff distance ranges from 0 to positive infinity, with 0 indicating perfect overlap between the two sets of points.

The TRE is the Euclidean distance between corresponding points in the predicted DVFs and ground truth (the motion estimated by a pre-trained registration network):

$$TRE = \frac{1}{n}\sum\limits_{i = 1}^n {\left\| {{p_i} - G{T_i}} \right\|}$$

(7)

where $\:P=\{{p}_{1},{p}_{2},\dots\:,{p}_{n}\}$ and $\:GT=\{{g}_{1},{g}_{2},\dots\:,{g}_{n}\}$ represent the points in the predicted DVFs and ground truth.

Prediction methods

Shift prediction

The shift prediction methods for MRIgRT are similar with those used in traditional real-time tumor tracking systems. As shown in Table 2, Yun et al. [49] utilized artificial neural networks (ANN) for 1D superior-inferior (SI) shift prediction among 29 lung cancer patients, and obtained mean RMSEs ranging from 0.5 to 0.9 mm across system latencies of 120 to 520 ms. Seregni et al. [29] compared linear extrapolation, autoregressive linear prediction (AR), and support vector machines (SVM) for 3D shift prediction in 6 lung cancer patients, and found that linear prediction method outperformed non-linear prediction method. Bourque et al. [50, 51] proposed a particle filter combined with an autoregressive model for 3D shift prediction among 5 healthy volunteers and 8 cancer patients, and verified that their prediction method was highly accurate and robust against varying imaging quality. Lombardo et al. [28, 34] compared ridge regression and long short-term memory (LSTM) networks for 1D SI shift prediction in-silico and experimentally confirmed the superiority of the LSTM. Their results also demonstrated that continuous online re-optimization can enhance the performance of prediction methods. Conversely, Li et al. [24] recently reported that linear prediction methods outperformed recurrent neural networks (RNNs) including LSTMs, bidirectional LSTMs (Bi-LSTMs), and gated recurrent unit networks (GRUs), among 21 liver cancer patients and 10 lung cancer patients.

Table 2 Methods for future shift prediction in MRIgRT

Recent literature [24, 28, 29] therefore reported inconsistencies in the superiority between linear and non-linear prediction methods for MRIgRT. This phenomenon can also be observed for motion prediction methods in the context of traditional real-time tumor tracking systems. Sharp et al. [37] reported that the linear prediction methods outperformed more complex prediction methods such as ANN and Kalman filter (KF). Jöhl et al. [41] compared 18 prediction methods for shift prediction among 93 respiratory motions, concluding that linear prediction methods were sufficient and non-linear prediction methods were not necessarily needed. On the contrary, Murphy et al. [43] demonstrated that non-linear prediction methods were more robust than linear prediction methods for irregular respiratory motions. Wang et al. [52] found that Bi-LSTM outperformed linear prediction methods using 103 respiratory motions, especially in cases with relatively long system latency. Wang et al. [53] also reported that an LSTM outperformed a SVM using a publicly available dataset. A possible explanation of these contradictory results might be that non-linear prediction methods have been relying more and more on RNNs compared to the ANNs in early studies. For example, Lin et al. [54] trained RNNs for shift prediction among 1703 respiratory motions, and showed the obtained LSTM to outperform an ANN. Furthermore, the performance of complex RNNs is more dependent on the optimization of model hyperparameters. As reported in Samadi et al. [23], tuning the hyperparameters of RNNs resulted in a 25–30% improvement for all models compared to previous studies.

Contour prediction and volumetric prediction

Contour prediction and volumetric prediction are relatively new research interests in motion management for radiotherapy. Theoretically, contour prediction and volumetric prediction can provide more accurate real-time tumor tracking over shift prediction. Contour prediction can be achieved through either directly predicting the 2D contour or by 2D image prediction. Contour prediction directly predicts future tumor contours from a sequence of observed contours, while image prediction predicts, from a series of past images, future images or DVFs which can be used to warp the last observed tumor contour. For real-time tumor tracking, contour prediction is the clinically more relevant method. However, image prediction eliminates the need for manual target delineation, allowing unsupervised training on larger datasets. Table 3 summarized the methods for contour prediction and volumetric prediction in MRIgRT.

Table 3 Methods for future contour and volumetric prediction in MRIgRT

For contour prediction, Noorda et al. [45] combined a subject-specific motion model with respiratory motion surrogate prediction on 4 healthy volunteers. However, this method could not extrapolate respiratory motion states not included in the constructed subject-specific motion model. Ginn et al. [22] proposed an image regression (IR) algorithm for image prediction using 8 healthy volunteers and 13 cancer patients. This method outperformed linear prediction methods in predicting the centroid position of 2D contour by utilizing a weighted combination of previously observed respiratory motion states, with weights determined via sum of squared differences (SSD) between current and past images. Nevertheless, the calculation of SSD is susceptible to image noise and IR may not provide accurate predictions for irregular respiratory motion not captured in the selected most similar images. Romaguera et al. [55] utilized a convolutional LSTM combined with spatial transformer layers (ConvLSTM-STL) for image prediction on 12 healthy volunteers, achieving median vessel misalignments of 0.45 mm and 0.57 mm for prediction windows of 320 ms and 640 ms, respectively. In a follow-up study, Romaguera et al. [48] demonstrated the superiority of a transformer network compared to ConvLSTMs and ConvGRUs for contour prediction. Interestingly, Lombardo et al. [56] found that an LSTM-shift prediction method, in which the last available contour is shifted by the difference between the predicted and last centroid position, outperformed both ConvLSTM contour and image prediction methods when looking at the accuracy of the predicted contours. More specifically, patients with larger respiratory motion are more likely to benefit from using the LSTM-shift prediction method.

In addition to spatiotemporal prediction, the implementation of volumetric prediction involves inferring 3D volumetric information from 2D cine MR images. This can be achieved through 2D-3D deformable image registration, which deforms a 3D pre-treatment reference volumetric image to align with orthogonal 2D cine MR images [57, 58]. However, the 2D-3D deformable image registration is time-consuming and infeasible for real-time applications. Alternatively, several studies constructed subject-specific or population-based motion models to overcome this limitation. Subject-specific motion models [59,60,61] parameterize motion information within the pre-treatment 4D datasets using principal component analysis (PCA), and can thereby accelerate the inference of DVFs. For example, Liu et al. [62] combined a subject-specific motion model with PCA coefficient prediction to achieve volumetric prediction using diagnostic MRI. Nevertheless, inferring DVFs that are not expected from the constructed subject-specific motion model might be problematic, and pre-treatment 4D MRI is not always available in clinical scenarios. To address these issues, various studies have proposed population-based motion models [63,64,65] that utilize large datasets to capture a broader range of respiratory motion patterns. Recently, Romaguera et al. [48] combined a population-based motion model with transformer network for volumetric motion prediction using 25 healthy volunteers, and achieved prediction of future 3D DVFs with a mean TRE of 1.2 ± 0.7 mm.

Inference of 3D volumetric information in real-time can also be achieved via deep learning-based fast image reconstruction [66, 67]. Terpstra et al. [68] trained a multi-resolution convolutional neural network for inferring 3D DVFs, achieving a TRE of 1.87 ± 1.65 mm. Similarity, Shao et al. [69] proposed a deep learning-based deformable registration network for downsampled 4D-MRI image reconstruction with sub-second latency. Xiao et al. [70] proposed a downsampling-invariant deformable registration model for inferring 3D DVFs, obtaining a reconstruction time of less than 500 ms. Liu et al. [71, 72] developed a geometry-informed deep learning framework for inferring 3D volumetric information with sub-second acquisition time, and incorporated implicit neural representation learning with prior information to enable fast volumetric image reconstruction from orthogonal cine MR images. These studies offer alternative approaches for inferring 3D volumetric information and facilitating further volumetric prediction in MRIgRT.

Discussion

Several clinical studies [73,74,75,76,77] have demonstrated that the superior treatment accuracy provided by MRIgRT is associated with improved clinical outcomes. Notably, Neylon et al. [78] found that large intrafraction motion in patients correlates with increased toxicity, which highlights the importance of real-time motion management in MRIgRT. Furthermore, real-time beam adaptation with either gating or MLC-tracking can provide a reduction of the CTV-PTV margin, boosting the potential of dose escalation [27, 79,80,81]. To maintain the accuracy of MLC-tracking, respiratory motion prediction becomes imperative to alleviate the system latency inherent to MRIgRT systems. In contrast to traditional real-time tracking systems, MRIgRT has the potential to achieve more advanced contour prediction and volumetric prediction, mainly relying on AI-based methods.

However, the implementation of AI in health and medicine faces challenges such as data limitations and modeling divergence [82]. For AI-based respiratory motion prediction in MRIgRT, complex clinical workflows and therefore reduced patient numbers limit the collection of large datasets, making it challenging to gather the amount of data required for robust model training, validation, and testing. Additionally, many studies utilized private datasets for the AI modeling [83, 84], and the lack of data transparency and code availability may further reduce the replicability of the reported results [85]. For example, Lombardo et al. [28, 34] reported that an LSTM outperformed a linear method, whereas Li et al. [24] recently obtained the opposite result. Therefore, in future work, the authors aim to establish a publicly available benchmark dataset for AI model comparison within AI-based respiratory motion prediction in MRIgRT.

Inspired by the University of California, Riverside (UCR) time series classification archive [86,87,88,89,90], this publicly available benchmark dataset should aim to standardize the modeling approaches, thereby promoting fair comparisons and accelerating advancements for AI-based respiratory motion prediction in MRIgRT. To ensure diversity in respiratory motion patterns, the data collection for this publicly available benchmark dataset should follow these criteria: (1) including irregular respiratory motions; (2) comprising multiple tumor sites to represent varying moving anatomies; and (3) incorporating multi-institutional data with varying image quality to assess the robustness of the prediction methods. To foster collaboration and reproducibility, this publicly available benchmark dataset will include detailed documentation, such as guidelines for data usage, model training, and evaluation metrics. Researchers will be encouraged to share their code and results through an open repository, enabling the community to build upon each other’s work and validate findings independently.

The reported average or median tracking accuracies of shift prediction methods (Tables 2 and 3) achieved in-silico were consistently within 3 mm. However, these in-silico studies did not evaluate and report the uncertainties associated with rapid target localization algorithms [69, 91,92,93]. This indicates that the accuracy in clinical scenarios might be inferior to the in-silico one, highlighting the need of developing more advanced prediction methods. As reported in Ginn et al. [22], their proposed contour prediction method outperformed common shift prediction methods. In contrast, Lombardo et al. [56] concluded that a shift prediction method overall outperformed both contour and image prediction methods. Also, their results indicated that patients with smaller respiratory motion are more likely to benefit from using the contour prediction methods. The execution time of their LSTM-shift model, ConvLSTM, and ConvLSTM-STL were 17 ± 3 ms, 14 ± 1 ms, and 45 ± 1 ms, respectively, which is clinically acceptable. Additionally, contour prediction and volumetric prediction can also provide the information of surrounding organs at risk (OARs), which may enhance the sparing of these critical OARs.

Model optimization strategies can also influence the final performance of AI-based respiratory motion prediction methods [54]. An interesting model optimization strategy is adaptive learning (also called continuous or online learning), which continuously updates the model weights to enable adaptation to recent respiratory motion patterns [94, 95]. Sun et al. [96] have confirmed that adaptive learning improves the performance of shift prediction models among 202 respiratory motions obtained from a real-time position management (RPM) device. Lombardo et al. [28, 56] also found that LSTMs with adaptive learning outperformed standard LSTM for shift prediction and contour prediction in MRIgRT. However, unlike RPM which directly provides respiratory motion signals, 2D cine MR images require additional preprocessing to extract these signals. For this reason, in their experimental study on a prototype MR-linac, Lombardo et al. [34] utilized a template matching algorithm for target centroid position extraction, and further confirmed the efficacy of adaptive learning in boosting LSTM performance for shift prediction during MRI-guided MLC tracking.

A straightforward approach to evaluate the end-to-end uncertainties in real-time motion management systems is to conduct experimentally studies. Uijtewaal et al. [16, 31] have experimentally demonstrated the combination of IMRT/VMAT with MLC-tracking on a prototype of a commercial MR-linac, obtaining 2%/1 mm pseudo-3D gamma passing-rates of 22–77% without MLC-tracking and 92–100% with MLC-tracking. However, this 3D gamma analysis might not be applicable for evaluating contour prediction and volumetric prediction. Lombardo et al. [34] proposed an alternative EPID-based performance analysis to evaluate the end-to-end uncertainties of MLC-tracking on a protype research MR-linac. The 2D cine MR images offered by MR-linacs allow for precise target localization, yet constrained by low temporal resolution. Integrating MR-linacs with the high temporal resolution of optical surface systems might enhance the tracking accuracy of MRIgRT [97].

Conclusions

In conclusion, AI-based methods have extended respiratory motion prediction from shift prediction to contour/image and volumetric prediction for MRIgRT. However, the inconsistent results observed in literature underscore the need of establishing a benchmark dataset for comparing traditional and AI-based respiratory motion prediction methods in MRIgRT. Additionally, investigating the impact of uncertainties associated with the real-time target localization step prior to the application of AI-based respiratory motion prediction might represent another research direction in MRIgRT.