In oncology, medical imaging is crucial for diagnosis, treatment planning and therapy execution. Treatment responses can be complex and varied and are known to involve factors of treatment, patient characteristics and tumor microenvironment. Longitudinal image analysis is able to track temporal changes, aiding in disease monitoring, treatment evaluation, and outcome prediction. This allows for the enhancement of personalized medicine. However, analyzing longitudinal 2D and 3D images presents unique challenges, including image registration, reliable segmentation, dealing with variable imaging intervals, and sparse data. This review presents an overview of techniques and methodologies in longitudinal image analysis, with a primary focus on outcome modeling in radiation oncology.

Pathological complete response (pCR) is an essential criterion for adjusting follow-up treatment plans for patients with breast cancer (BC). The value of the visual geometry group and long short-term memory (VGG-LSTM) network using time-series dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) for pCR identification in BC is unclear.

To identify pCR to neoadjuvant chemotherapy (NAC) using deep learning (DL) models based on the VGG-LSTM network.

Retrospective.

Center A: 235 patients (47.7 ± 10.0 years) were divided 7:3 into training (n = 164) and validation set (n = 71). Center B: 150 patients (48.5 ± 10.4 years) were used as test set.

3-T, T2-weighted spin-echo sequence imaging, and gradient echo DCE sequence imaging.

Patients underwent MRI examinations at three sequential time points: pretreatment, after three cycles of treatment, and prior to surgery, with tumor regions of interest manually delineated. Histopathology was the gold standard. We used VGG-LSTM network to establish seven DL models using time-series DCE-MR images: pre-NAC images (t0 model), early NAC images (t1 model), post-NAC images (t2 model), pre-NAC and early NAC images (t0 + t1 model), pre-NAC and post-NAC images (t0 + t2 model), pre-NAC, early NAC and post-NAC images (t0 + t1 + t2 model), and the optimal model combined with the clinical features and imaging features (combined model). The models were trained and optimized on the training and validation set, and tested on the test set.

The DeLong, Student's t-test, Mann-Whitney U, Chi-squared, Fisher's exact, Hosmer-Lemeshow tests, decision curve analysis, and receiver operating characteristics analysis were performed. P < 0.05 was considered significant.

Compared with the other six models, the combined model achieved the best performance in the test set yielding an AUC of 0.927.

The combined model that used time-series DCE-MR images, clinical features and imaging features shows promise for identifying pCR in BC.

Stage 4.

Patients may undergo anatomical changes during radiotherapy, leading to an underdosing of the target or overdosing of the organs at risk (OARs).

This study developed a deep-learning method to predict the tumor response of patients with nasopharyngeal carcinoma (NPC) during treatment. This method can predict the anatomical changes of a patient.

The participants included 230 patients with NPC. The data included planning computed tomography (pCT) and routine cone-beam CT (CBCT) images. The CBCT image quality was improved to the CT level using an advanced method. A long short-term memory network-generative adversarial network (LSTM-GAN) is proposed, which can harness the forecasting ability of LSTM and the generation ability of GAN. Four models were trained to predict the anatomical changes that occurred in weeks 3-6 and named LSTM-GAN-week 3 to LSTM-GAN-week 6. The pCT and CBCT were used as input, and the tumor target volumes (TVs) and OARs were delineated on the predicted and real images (ground truth). Finally, the models were evaluated using contours and dosimetry parameters.

The proposed method predicted the anatomical changes, with a dice similarity coefficient above 0.94 and 0.90 for the TVs and surrounding OARs, respectively. The dosimetry parameters were close between the prediction and ground truth. The deviations in the prescription, minimum, and maximum doses of the tumor targets were below 0.5 Gy. For serial organs (brain stem and spinal cord), the deviations in the maximum dose were below 0.6 Gy. For parallel organs (bilateral parotid glands), the deviations in the mean dose were below 0.8 Gy.

The proposed method can predict the tumor response to radiotherapy in the future such that adaptation can be scheduled on time. This study provides a proactive mechanism for planning adaptation, which can enable personalized treatment and save clinical time by anticipating and preparing for treatment strategy adjustments.

Radiotherapy aims to achieve a high tumor control probability while minimizing damage to normal tissues. Personalizing radiotherapy treatments for individual patients, therefore, depends on integrating physical treatment planning with predictive models of tumor control and normal tissue complications. Predictive models could be improved using a wide range of rich data sources, including tumor and normal tissue genomics, radiomics, and dosiomics. Deep learning will drive improvements in classifying normal tissue tolerance, predicting intra-treatment tumor changes, tracking accumulated dose distributions, and quantifying the tumor response to radiotherapy based on imaging. Mechanistic patient-specific computer simulations ('digital twins') could also be used to guide adaptive radiotherapy. Overall, we are entering an era where improved modeling methods will allow the use of newly available data sources to better guide radiotherapy treatments.

Purpose To develop a machine learning approach for classifying disease progression in chest radiographs using weak labels automatically derived from radiology reports. Materials and Methods In this retrospective study, a twin neural network was developed to classify anatomy-specific disease progression into four categories: improved, unchanged, worsened, and new. A two-step weakly supervised learning approach was employed, pretraining the model on 243 008 frontal chest radiographs from 63 877 patients (mean age, 51.7 years ± 17.0 [SD]; 34 813 [55%] female) included in the MIMIC-CXR database and fine-tuning it on the subset with progression labels derived from consecutive studies. Model performance was evaluated for six pathologic observations on test datasets of unseen patients from the MIMIC-CXR database. Area under the receiver operating characteristic (AUC) analysis was used to evaluate classification performance. The algorithm is also capable of generating bounding-box predictions to localize areas of new progression. Recall, precision, and mean average precision were used to evaluate the new progression localization. One-tailed paired t tests were used to assess statistical significance. Results The model outperformed most baselines in progression classification, achieving macro AUC scores of 0.72 ± 0.004 for atelectasis, 0.75 ± 0.007 for consolidation, 0.76 ± 0.017 for edema, 0.81 ± 0.006 for effusion, 0.7 ± 0.032 for pneumonia, and 0.69 ± 0.01 for pneumothorax. For new observation localization, the model achieved mean average precision scores of 0.25 ± 0.03 for atelectasis, 0.34 ± 0.03 for consolidation, 0.33 ± 0.03 for edema, and 0.31 ± 0.03 for pneumothorax. Conclusion Disease progression classification models were developed on a large chest radiograph dataset, which can be used to monitor interval changes and detect new pathologic conditions on chest radiographs. Keywords: Prognosis, Unsupervised Learning, Transfer Learning, Convolutional Neural Network (CNN), Emergency Radiology, Named Entity Recognition Supplemental material is available for this article. © RSNA, 2024 See also commentary by Alves and Venkadesh in this issue.

For many decades, the management of cancer has utilized radiation therapy, which continues to evolve with technology to improve patient outcomes. However, despite the standardization of treatment plans and the establishment of best clinical practices based on prospective, randomized trials and adherence to National Comprehensive Cancer Network (NCCN) guidelines, the outcomes from radiation therapy are highly variable and dependent on a number of factors, including patient demographics, tumor characteristics/histology, and treatment parameters. In this study, we attempt to use available patient data and treatment parameters at the time of radiation therapy to predict future outcomes using artificial intelligence (AI).

Six thousand five hundred ninety-five cases of patients who completed radiation treatment were selected retrospectively and used to train artificial neural networks (ANNs) and baseline models (i.e., logistic regression, random forest, support vector machines [SVMs], gradient boosting [XGBoost]) for binary classification of mortality at multiple time points ranging from six months to five years post-treatment. A hyperparameter grid search was used to identify the optimal network architecture for each time point, using sensitivity as the primary outcome metric.

The median age was 75 years (range: 2-102 years). There were 63.8% females and 36.1% males. The results indicate that ANNs were able to successfully perform binary mortality prediction with an accuracy greater than random chance and greater sensitivity than baseline models used. The best-performing algorithm was the ANN, which achieved a sensitivity of 83.00% ± 4.89% for five-year mortality.

The neural network was able to achieve higher sensitivity than Logistic Regression, SVM Random Forest, and XGBoost across all output target variables, demonstrating the utility of a neural network model for mortality prediction on the provided dataset.

Pediatric glioma recurrence can cause morbidity and mortality; however, recurrence pattern and severity are heterogeneous and challenging to predict with established clinical and genomic markers. Resultingly, almost all children undergo frequent, long-term, magnetic resonance (MR) brain surveillance regardless of individual recurrence risk. Deep learning analysis of longitudinal MR may be an effective approach for improving individualized recurrence prediction in gliomas and other cancers but has thus far been infeasible with current frameworks. Here, we propose a self-supervised, deep learning approach to longitudinal medical imaging analysis, temporal learning, that models the spatiotemporal information from a patient's current and prior brain MRs to predict future recurrence. We apply temporal learning to pediatric glioma surveillance imaging for 715 patients (3,994 scans) from four distinct clinical settings. We find that longitudinal imaging analysis with temporal learning improves recurrence prediction performance by up to 41% compared to traditional approaches, with improvements in performance in both low- and high-grade glioma. We find that recurrence prediction accuracy increases incrementally with the number of historical scans available per patient. Temporal deep learning may enable point-of-care decision-support for pediatric brain tumors and be adaptable more broadly to patients with other cancers and chronic diseases undergoing surveillance imaging.

Recent advances in MRI-guided radiation therapy (MRgRT) and deep learning techniques encourage fully adaptive radiation therapy (ART), real-time MRI monitoring, and the MRI-only treatment planning workflow. Given the rapid growth and emergence of new state-of-the-art methods in these fields, we systematically review 197 studies written on or before December 31, 2022, and categorize the studies into the areas of image segmentation, image synthesis, radiomics, and real time MRI. Building from the underlying deep learning methods, we discuss their clinical importance and current challenges in facilitating small tumor segmentation, accurate x-ray attenuation information from MRI, tumor characterization and prognosis, and tumor motion tracking. In particular, we highlight the recent trends in deep learning such as the emergence of multi-modal, visual transformer, and diffusion models.

Adaptive radiotherapy (ART) is an advanced technology in modern cancer treatment that incorporates progressive changes in patient anatomy into active plan/dose adaption during the fractionated treatment. However, the clinical application relies on the accurate segmentation of cancer tumors on low-quality on-board images, which has posed challenges for both manual delineation and deep learning-based models. In this paper, we propose a novel sequence transduction deep neural network with an attention mechanism to learn the shrinkage of the cancer tumor based on patients' weekly cone-beam computed tomography (CBCT). We design a self-supervised domain adaption (SDA) method to learn and adapt the rich textural and spatial features from pre-treatment high-quality computed tomography (CT) to CBCT modality in order to address the poor image quality and lack of labels. We also provide uncertainty estimation for sequential segmentation, which aids not only in the risk management of treatment planning but also in the calibration and reliability of the model. Our experimental results based on a clinical non-small cell lung cancer (NSCLC) dataset with sixteen patients and ninety-six longitudinal CBCTs show that our model correctly learns weekly deformation of the tumor over time with an average dice score of 0.92 on the immediate next step, and is able to predict multiple steps (up to 5 weeks) for future patient treatments with an average dice score reduction of 0.05. By incorporating the tumor shrinkage predictions into a weekly re-planning strategy, our proposed method demonstrates a significant decrease in the risk of radiation-induced pneumonitis up to 35% while maintaining the high tumor control probability.

MRI-guided radiation therapy (MRgRT) offers a precise and adaptive approach to treatment planning. Deep learning applications which augment the capabilities of MRgRT are systematically reviewed. MRI-guided radiation therapy offers a precise, adaptive approach to treatment planning. Deep learning applications which augment the capabilities of MRgRT are systematically reviewed with emphasis placed on underlying methods. Studies are further categorized into the areas of segmentation, synthesis, radiomics, and real time MRI. Finally, clinical implications, current challenges, and future directions are discussed.

Brain tumor is one of the leading causes of cancer death. The high-grade brain tumors are easier to recurrent even after standard treatment. Therefore, developing a method to predict brain tumor recurrence location plays an important role in the treatment planning and it can potentially prolong patient's survival time. There is still little work to deal with this issue. In this paper, we present a deep learning-based brain tumor recurrence location prediction network. Since the dataset is usually small, we propose to use transfer learning to improve the prediction. We first train a multi-modal brain tumor segmentation network on the public dataset BraTS 2021. Then, the pre-trained encoder is transferred to our private dataset for extracting the rich semantic features. Following that, a multi-scale multi-channel feature fusion model and a nonlinear correlation learning module are developed to learn the effective features. The correlation between multi-channel features is modeled by a nonlinear equation. To measure the similarity between the distributions of original features of one modality and the estimated correlated features of another modality, we propose to use Kullback-Leibler divergence. Based on this divergence, a correlation loss function is designed to maximize the similarity between the two feature distributions. Finally, two decoders are constructed to jointly segment the present brain tumor and predict its future tumor recurrence location. To the best of our knowledge, this is the first work that can segment the present tumor and at the same time predict future tumor recurrence location, making the treatment planning more efficient and precise. The experimental results demonstrated the effectiveness of our proposed method to predict the brain tumor recurrence location from the limited dataset.

To simultaneously register all the longitudinal images acquired in a radiotherapy course for analyzing patients' anatomy changes for adaptive radiotherapy (ART).

To address the unique needs of ART, we designed Seq2Morph, a novel deep learning-based deformable image registration (DIR) network. Seq2Morph was built upon VoxelMorph which is a general-purpose framework for learning-based image registration. The major upgrades are (1) expansion of inputs to all weekly cone-beam computed tomography (CBCTs) acquired for monitoring treatment responses throughout a radiotherapy course, for registration to their planning CT; (2) incorporation of 3D convolutional long short-term memory between the encoder and decoder of VoxelMorph, to parse the temporal patterns of anatomical changes; and (3) addition of bidirectional pathways to calculate and minimize inverse consistency errors (ICEs). Longitudinal image sets from 50 patients, including a planning CT and 6 weekly CBCTs per patient, were utilized for network training and cross-validation. The outputs were deformation vector fields for all the registration pairs. The loss function was composed of a normalized cross-correlation for image intensity similarity, a DICE for contour similarity, an ICE, and a deformation regularization term. For performance evaluation, DICE and Hausdorff distance (HD) for the manual versus predicted contours of tumor and esophagus on weekly basis were quantified and further compared with other state-of-the-art algorithms, including conventional VoxelMorph and large deformation diffeomorphic metric mapping (LDDMM).

Visualization of the hidden states of Seq2Morph revealed distinct spatiotemporal anatomy change patterns. Quantitatively, Seq2Morph performed similarly to LDDMM, but significantly outperformed VoxelMorph as measured by GTV DICE: (0.799±0.078, 0.798±0.081, and 0.773±0.078), and 50% HD (mm): (0.80±0.57, 0.88±0.66, and 0.95±0.60). The per-patient inference of Seq2Morph took 22 s, much less than LDDMM (∼30 min).

Seq2Morph can provide accurate and fast DIR for longitudinal image studies by exploiting spatial-temporal patterns. It closely matches the clinical workflow and has the potential to serve both online and offline ART.

This study aimed to assess interfraction stability of the delivered dose distribution by exhale-gated volumetric modulated arc therapy (VMAT) or intensity-modulated arc therapy (IMAT) for lung cancer and to determine dominant prognostic dosimetric and geometric factors.

Clinical target volume (CTV_Plan) from the planning CT was deformed to the exhale-gated daily CBCT scans to determine CTV_i, treated by the respective dose fraction. The equivalent uniform dose of the CTV_i was determined by the power law (gEUD_i) and cell survival model (EUD_iSF) as effectiveness measure for the delivered dose distribution. The following prognostic factors were analyzed: (I) minimum dose within the CTV_i (D_{min_i}), (II) Hausdorff distance (HDD_i) between CTV_i and CTV_Plan, (III) doses and deformations at the point in CTV_Plan at which the global minimum dose over all fractions per patient occurs (PD_{min_global_i}), and (IV) deformations at the point over all CTV_i margins per patient with the largest Hausdorff distance (HDPw_orst). Prognostic value and generalizability of the prognostic factors were examined using cross-validated random forest or multilayer perceptron neural network (MLP) classifiers. Dose accumulation was performed using back deformation of the dose distribution from CTV_i to CTV_Plan.

Altogether, 218 dose fractions (10 patients) were evaluated. There was a significant interpatient heterogeneity between the distributions of the normalized gEUD_i values (p<0.0001, Kruskal-Wallis tests). Accumulated gEUD over all fractions per patient was 1.004-1.023 times of the prescribed dose. Accumulation led to tolerance of ~20% of fractions with gEUD_i <93% of the prescribed dose. Normalized D_min >60% was associated with predicted gEUD values above 95%. D_min had the highest importance for predicting the gEUD over all analyzed prognostic parameters by out-of-bag loss reduction using the random forest procedure. Cross-validated random forest classifier based on D_min as the sole input had the largest Pearson correlation coefficient (R=0.897) in comparison to classifiers using additional input variables. The neural network performed better than the random forest classifier, and the gEUD values predicted by the MLP classifier with D_min as the sole input were correlated with the gEUD values characterized by R=0.933 (95% CI, 0.913-0.948). The performance of the full MLP model with all geometric input parameters was slightly better (R=0.952) than that based on D_min (p=0.0034, Z-test).

Accumulated dose distributions over the treatment series were robust against interfraction CTV deformations using exhale gating and online image guidance. D_min was the most important parameter for gEUD prediction for a single fraction. All other parameters did not lead to a markedly improved generalizable prediction. Dosimetric information, especially location and value of D_min within the CTV _i , are vital information for image-guided radiation treatment.

Objective.In the field of radiation oncology, the benefit of MRI goes beyond that of providing high soft-tissue contrast images for staging and treatment planning. With the recent clinical introduction of hybrid MRI linear accelerators it has become feasible to map physiological parameters describing diffusion, perfusion, and relaxation during the entire course of radiotherapy, for example. However, advanced data analysis tools are required for extracting qualified prognostic and predictive imaging biomarkers from longitudinal MRI data. In this study, we propose a new prediction framework tailored to exploit temporal dynamics of tissue features from repeated measurements. We demonstrate the framework using a newly developed decomposition method for tumor characterization.Approach.Two previously published MRI datasets with multiple measurements during and after radiotherapy, were used for development and testing:T₂-weighted multi-echo images obtained for two mouse models of pancreatic cancer, and diffusion-weighted images for patients with brain metastases. Initially, the data was decomposed using the novel monotonous slope non-negative matrix factorization (msNMF) tailored for MR data. The following processing consisted of a tumor heterogeneity assessment using descriptive statistical measures, robust linear modelling to capture temporal changes of these, and finally logistic regression analysis for stratification of tumors and volumetric outcome.Main Results.The framework was able to classify the two pancreatic tumor types with an area under curve (AUC) of 0.999,P< 0.001 and predict the tumor volume change with a correlation coefficient of 0.513,P= 0.034. A classification of the human brain metastases into responders and non-responders resulted in an AUC of 0.74,P= 0.065.Significance.A general data processing framework for analyses of longitudinal MRI data has been developed and applications were demonstrated by classification of tumor type and prediction of radiotherapy response. Further, as part of the assessment, the merits of msNMF for tumor tissue decomposition were demonstrated.

Information in digital mammogram images has been shown to be associated with the risk of developing breast cancer. Longitudinal breast cancer screening mammogram examinations may carry spatiotemporal information that can enhance breast cancer risk prediction. No deep learning models have been designed to capture such spatiotemporal information over multiple examinations to predict the risk. In this study, we propose a novel deep learning structure, LRP-NET, to capture the spatiotemporal changes of breast tissue over multiple negative/benign screening mammogram examinations to predict near-term breast cancer risk in a case-control setting. Specifically, LRP-NET is designed based on clinical knowledge to capture the imaging changes of bilateral breast tissue over four sequential mammogram examinations. We evaluate our proposed model with two ablation studies and compare it to three models/settings, including 1) a "loose" model without explicitly capturing the spatiotemporal changes over longitudinal examinations, 2) LRP-NET but using a varying number (i.e., 1 and 3) of sequential examinations, and 3) a previous model that uses only a single mammogram examination. On a case-control cohort of 200 patients, each with four examinations, our experiments on a total of 3200 images show that the LRP-NET model outperforms the compared models/settings.

Predicting whether a lung nodule will grow, remain stable or regress over time, especially early in its follow-up, would help doctors prescribe personalized treatments and better surgical planning. However, the multifactorial nature of lung tumour progression hampers the identification of growth patterns. In this work, we propose a deep hierarchical generative and probabilistic network that, given an initial image of the nodule, predicts whether it will grow, quantifies its future size and provides its expected semantic appearance at a future time. Unlike previous solutions, our approach also estimates the uncertainty in the predictions from the intrinsic noise in medical images and the inter-observer variability in the annotations. The evaluation of this method on an independent test set reported a future tumour growth size mean absolute error of 1.74 mm, a nodule segmentation Dice's coefficient of 78% and a tumour growth accuracy of 84% on predictions made up to 24 months ahead. Due to the lack of similar methods for providing future lung tumour growth predictions, along with their associated uncertainty, we adapted equivalent deterministic and alternative generative networks (i.e., probabilistic U-Net, Bayesian test dropout and Pix2Pix). Our method outperformed all these methods, corroborating the adequacy of our approach.

A small dataset commonly affects generalization, robustness, and overall performance of deep neural networks (DNNs) in medical imaging research. Since gathering large clinical databases is always difficult, we proposed an analytical method for producing a large realistic/diverse dataset. Clinical brain PET/CT/MR images including full-dose (FD), low-dose (LD) corresponding to only 5 % of events acquired in the FD scan, non-attenuated correction (NAC) and CT-based measured attenuation correction (MAC) PET images, CT images and T1 and T2 MR sequences of 35 patients were included. All images were registered to the Montreal Neurological Institute (MNI) template. Laplacian blending was used to make a natural presentation using information in the frequency domain of images from two separate patients, as well as the blending mask. This classical technique from the computer vision and image processing communities is still widely used and unlike modern DNNs, does not require the availability of training data. A modified ResNet DNN was implemented to evaluate four image-to-image translation tasks, including LD to FD, LD+MR to FD, NAC to MAC, and MRI to CT, with and without using the synthesized images. Quantitative analysis using established metrics, including the peak signal-to-noise ratio (PSNR), structural similarity index metric (SSIM), and joint histogram analysis was performed for quantitative evaluation. The quantitative comparison between the registered small dataset containing 35 patients and the large dataset containing 350 synthesized plus 35 real dataset demonstrated improvement of the RMSE and SSIM by 29% and 8% for LD to FD, 40% and 7% for LD+MRI to FD, 16% and 8% for NAC to MAC, and 24% and 11% for MRI to CT mapping task, respectively. The qualitative/quantitative analysis demonstrated that the proposed model improved the performance of all four DNN models through producing images of higher quality and lower quantitative bias and variance compared to reference images.

To develop a novel deep learning algorithm of sequential analysis, Seq2Seq, for predicting weekly anatomical changes of lung tumor and esophagus during definitive radiotherapy, incorporate the potential tumor shrinkage into a predictive treatment planning paradigm, and improve the therapeutic ratio.

Seq2Seq starts with the primary tumor and esophagus observed on the planning CT to predict their geometric evolution during radiotherapy on a weekly basis, and subsequently updates the predictions with new snapshots acquired via weekly CBCTs. Seq2Seq is equipped with convolutional long short term memory to analyze the spatial-temporal changes of longitudinal images, trained and validated using a dataset including sixty patients. Predictive plans were optimized according to each weekly prediction and made ready for weekly deployment to mitigate the clinical burden of online weekly replanning.

Seq2Seq tracks structural changes well: DICE between predicted and actual weekly tumor and esophagus were (0.83 ± 0.10, 0.79 ± 0.14, 0.78 ± 0.12, 0.77 ± 0.12, 0.75 ± 0.12, 0.71 ± 0.17), and (0.72 ± 0.16, 0.73 ± 0.11, 0.75 ± 0.08, 0.74 ± 0.09, 0.72 ± 0.14, 0.71 ± 0.14), respectively, while the average Hausdorff distances were within 2 mm. Evaluating dose to the actual weekly tumor and esophagus, a 4.2 Gy reduction in esophagus mean dose while maintaining 60 Gy tumor coverage was achieved with the predictive weekly plans, compared to the plan optimized using the initial tumor and esophagus alone, primarily due to noticeable tumor shrinkage during radiotherapy.

It is feasible to predict the longitudinal changes of tumor and esophagus with the Seq2Seq, which could lead to improving the efficiency and effectiveness of lung adaptive radiotherapy.

Lung cancer treatment is constantly evolving due to technological advances in the delivery of radiation therapy. Adaptive radiation therapy (ART) allows for modification of a treatment plan with the goal of improving the dose distribution to the patient due to anatomic or physiologic deviations from the initial simulation. The implementation of ART for lung cancer is widely varied with limited consensus on who to adapt, when to adapt, how to adapt, and what the actual benefits of adaptation are. ART for lung cancer presents significant challenges due to the nature of the moving target, tumor shrinkage, and complex dose accumulation because of plan adaptation. This article presents an overview of the current state of the field in ART for lung cancer, specifically, probing topics of: patient selection for the greatest benefit from adaptation, models which predict who and when to adapt plans, best timing for plan adaptation, optimized workflows for implementing ART including alternatives to re-simulation, the best radiation techniques for ART including magnetic resonance guided treatment, algorithms and quality assurance, and challenges and techniques for dose reconstruction. To date, the clinical workflow burden of ART is one of the major reasons limiting its widespread acceptance. However, the growing body of evidence demonstrates overwhelming support for reduced toxicity while improving tumor dose coverage by adapting plans mid-treatment, but this is offset by the limited knowledge about tumor control. Progress made in predictive modeling of on-treatment tumor shrinkage and toxicity, optimizing the timing of adaptation of the plan during the course of treatment, creating optimal workflows to minimize staffing burden, and utilizing deformable image registration represent ways the field is moving toward a more uniform implementation of ART.

Objective.Modern radiotherapy stands to benefit from the ability to efficiently adapt plans during treatment in response to setup and geometric variations such as those caused by internal organ deformation or tumor shrinkage. A promising strategy is to develop a framework, which given an initial state defined by patient-attributes, can predict future states based on pre-learned patterns from a well-defined patient population.Approach.Here, we investigate the feasibility of predicting patient anatomical changes, defined as a joint state of volume and daily setup changes, across a fractionated treatment schedule using two approaches. The first is based on a new joint framework employing quantum mechanics in combination with deep recurrent neural networks, denoted QRNN. The second approach is developed based on a classical framework, which models patient changes as a Markov process, denoted MRNN. We evaluated the performance characteristics of these two approaches on a dataset of 125 head and neck cancer patients, which was supplemented by synthetic data generated using a generative adversarial network. Model performance was evaluated using area under the receiver operating characteristic curve (AUC) scores.Main results.The MRNN framework had slightly better performance than the QRNN framework, with MRNN (QRNN) validation AUC scores of 0.742±0.021 (0.675±0.036), 0.709±0.026 (0.656±0.021), 0.724±0.036 (0.652±0.044), and 0.698±0.016 (0.605±0.035) for system state vector sizes of 4, 6, 8, and 10, respectively. Of these, only the results from the two higher order states had statistically significant differences(p<0.05).A similar trend was also observed when the models were applied to an external testing dataset of 20 patients, yielding MRNN (QRNN) AUC scores of 0.707 (0.623), 0.687 (0.608), 0.723 (0.669), and 0.697 (0.609) for states vectors sizes of 4, 6, 8, and 10, respectively.Significance.These results suggest that both stochastic models have potential value in predicting patient changes during the course of adaptive radiotherapy.

Radiotherapy presents unique challenges and clinical requirements for longitudinal tumor and organ-at-risk (OAR) prediction during treatment. The challenges include tumor inflammation/edema and radiation-induced changes in organ geometry, whereas the clinical requirements demand flexibility in input/output sequence timepoints to update the predictions on rolling basis and the grounding of all predictions in relationship to the pre-treatment imaging information for response and toxicity assessment in adaptive radiotherapy.

To deal with the aforementioned challenges and to comply with the clinical requirements, we present a novel 3D sequence-to-sequence model based on Convolution Long Short-Term Memory (ConvLSTM) that makes use of series of deformation vector fields (DVFs) between individual timepoints and reference pre-treatment/planning CTs to predict future anatomical deformations and changes in gross tumor volume as well as critical OARs. High-quality DVF training data are created by employing hyper-parameter optimization on the subset of the training data with DICE coefficient and mutual information metric. We validated our model on two radiotherapy datasets: a publicly available head-and-neck dataset (28 patients with manually contoured pre-, mid-, and post-treatment CTs), and an internal non-small cell lung cancer dataset (63 patients with manually contoured planning CT and 6 weekly CBCTs).

The use of DVF representation and skip connections overcomes the blurring issue of ConvLSTM prediction with the traditional image representation. The mean and standard deviation of DICE for predictions of lung GTV at weeks 4, 5, and 6 were 0.83 ± 0.09, 0.82 ± 0.08, and 0.81 ± 0.10, respectively, and for post-treatment ipsilateral and contralateral parotids, were 0.81 ± 0.06 and 0.85 ± 0.02.

We presented a novel DVF-based Seq2Seq model for medical images, leveraging the complete 3D imaging information of a relatively large longitudinal clinical dataset, to carry out longitudinal GTV/OAR predictions for anatomical changes in HN and lung radiotherapy patients, which has potential to improve RT outcomes.

Many neurological diseases are characterized by gradual deterioration of brain structure andfunction. Large longitudinal MRI datasets have revealed such deterioration, in part, by applying machine and deep learning to predict diagnosis. A popular approach is to apply Convolutional Neural Networks (CNN) to extract informative features from each visit of the longitudinal MRI and then use those features to classify each visit via Recurrent Neural Networks (RNNs). Such modeling neglects the progressive nature of the disease, which may result in clinically implausible classifications across visits. To avoid this issue, we propose to combine features across visits by coupling feature extraction with a novel longitudinal pooling layer and enforce consistency of the classification across visits in line with disease progression. We evaluate the proposed method on the longitudinal structural MRIs from three neuroimaging datasets: Alzheimer's Disease Neuroimaging Initiative (ADNI, N=404), a dataset composed of 274 normal controls and 329 patients with Alcohol Use Disorder (AUD), and 255 youths from the National Consortium on Alcohol and NeuroDevelopment in Adolescence (NCANDA). In allthree experiments our method is superior to other widely used approaches for longitudinal classification thus making a unique contribution towards more accurate tracking of the impact of conditions on the brain. The code is available at https://github.com/ouyangjiahong/longitudinal-pooling.

We propose a treatment planning framework that accounts for weekly lung tumor shrinkage using cone beam computed tomography (CBCT) images with a deep learning-based model.

Sixteen patients with non-small-cell lung cancer (NSCLC) were selected with one planning CT and six weekly CBCTs each. A deep learning-based model was applied to predict the weekly deformation of the primary tumor based on the spatial and temporal features extracted from previous weekly CBCTs. Starting from Week 3, the tumor contour at Week N was predicted by the model based on the input from all the previous weeks (1, 2 … N - 1), and was evaluated against the manually contoured tumor using Dice coefficient (DSC), precision, average surface distance (ASD), and Hausdorff distance (HD). Information about the predicted tumor was then entered into the treatment planning system and the plan was re-optimized every week. The objectives were to maximize the dose coverage in the target region while minimizing the toxicity to the surrounding healthy tissue. Dosimetric evaluation of the target and organs at risk (heart, lung, esophagus, and spinal cord) was performed on four cases, comparing between a conventional plan (ignoring tumor shrinkage) and the shrinkage-based plan.

he primary tumor volumes decreased on average by 38% ± 26% during six weeks of treatment. DSCs and ASD between the predicted tumor and the actual tumor for Weeks 3, 4, 5, 6 were 0.81, 0.82, 0.79, 0.78 and 1.49, 1.59, 1.92, 2.12 mm, respectively, which were significantly superior to the score of 0.70, 0.68, 0.66, 0.63 and 2.81, 3.22, 3.69, 3.63 mm between the rigidly transferred tumors ignoring shrinkage and the actual tumor. While target coverage metrics were maintained for the re-optimized plans, lung mean dose dropped by 2.85, 0.46, 2.39, and 1.48 Gy for four sample cases when compared to the original plan. Doses in other organs such as esophagus were also reduced for some cases.

We developed a deep learning-based model for tumor shrinkage prediction. This model used CBCTs and contours from previous weeks as input and produced reasonable tumor contours with a high prediction accuracy (DSC, precision, HD, and ASD). The proposed framework maintained target coverage while reducing dose in the lungs and esophagus.

The purpose of this work is to create a decision support methodology to predict when patients undergoing radiotherapy treatment for locally advanced lung cancer would potentially benefit from adaptive radiotherapy. The proposed methodology seeks to eliminate the manual subjective review by developing an automated statistical learning model to predict when tumor regression would trigger implementation of adaptive radiotherapy based on quantified anatomic changes observed in individual patients on-treatment cone beam computed tomographies (CTs). This proposed process seeks to improve the efficacy and efficiency of both the existing manual and automated adaptive review processes for locally advanced stage III lung cancer.

A predictive algorithm was developed as a decision support tool to determine the potential utility of mid-treatment adaptive radiotherapy based on anatomic changes observed on 1158 daily CBCT images across 43 patients. The anatomic changes on each axial slice within specified regions-of-interest were quantified into a single value utilizing imaging similarity criteria comparing the daily CBCT to the initial simulation CT. The range of the quantified metrics for each fraction across all axial slices are reduced to specified quantiles, which are used as the predictive input to train a logistic regression algorithm. A "ground-truth" of the need for adaptive radiotherapy based on tumor regression was evaluated systematically on each of the daily CBCTs and used as the classifier in the logistic regression algorithm. Accuracy of the predictive model was assessed utilizing both a tenfold cross validation and an independent validation dataset, with the sensitivity, specificity, and fractional accuracy compared to the ground-truth.

The sensitivity and specificity for the individual daily fractions ranged from 87.9%-94.3% and 91.9%-98.6% for a probability threshold of 0.2-0.5, respectively. The corresponding average treatment fraction difference between the model predictions and assessed ART "ground-truth" ranged from -2.25 to -0.07 fractions, with the model predictions consistently predicting the potential need for ART earlier in the treatment course. By initially utilizing a lower probability threshold, the higher sensitivity minimizes the chance of false negative by alerting the clinician to review a higher number of questionable cases.

The proposed methodology accurately predicted the first fraction at which individual patients may benefit from ART based on quantified anatomic changes observed in the on-treatment volumetric imaging. The generalizability of the proposed method has potential to expand to additional modes of adaptive radiotherapy for lung cancer patients with observed underlying anatomic changes.

During radiation therapy (RT) of head and neck (HN) cancer, the shape and volume of the parotid glands (PG) may change significantly, resulting in clinically relevant deviations of delivered dose from the planning dose. Early and accurate longitudinal prediction of PG anatomical changes during the RT can be valuable to inform decisions on plan adaptation. We developed a deep neural network for longitudinal predictions using the displacement fields (DFs) between the planning computed tomography (pCT) and weekly cone beam computed tomography (CBCT). Sixty-three HN patients treated with volumetric modulated arc were retrospectively studied. We calculated DFs between pCT and week 1-3 CBCT by B-spline and Demon deformable image registration (DIR). The resultant DFs were subsequently used as input to our novel network to predict the week 4 to 6 DFs for generating predicted weekly PG contours and weekly dose distributions. For evaluation, we measured dice similarity (DICE), and the uncertainty of accumulated dose. Moreover, we compared the detection accuracies of candidates for adaptive radiotherapy (ART) when the trigger criteria were mean dose difference more than 10%, 7.5%, and 5%, respectively. The DICE of ipsilateral/contralateral PG at week 4 to 6 using the prediction model trained with B-spline were 0.81 [Formula: see text] 0.07/0.81 [Formula: see text] 0.04 (week 4), 0.79 [Formula: see text] 0.06/0.81 [Formula: see text] 0.05 (week 5) and 0.78 [Formula: see text] 0.06/0.82 [Formula: see text] (week 6). The DICE with the Demons model were 0.78 [Formula: see text] 0.08/0.82 [Formula: see text] 0.03 (week 4), 0.77 [Formula: see text] 0.07/0.82 [Formula: see text] 0.04 (week 5) and 0.75 [Formula: see text] 0.07/0.82 [Formula: see text] 0.02 (week 6). The dose volume histogram (DVH) analysis with the predicted accumulated dose showed the feasibility of predicting dose uncertainty due to the PG anatomical changes. The AUC of ART candidate detection with our predictive model was over 0.90. In conclusion, the proposed network was able to predict future anatomical changes and dose uncertainty of PGs with clinically acceptable accuracy, and hence can be readily integrated into the ART workflow.

Acute esophagitis (AE) occurs among a significant number of patients with locally advanced lung cancer treated with radiotherapy. Early prediction of AE, indicated by esophageal wall expansion, is critical, as it can facilitate the redesign of treatment plans to reduce radiation-induced esophageal toxicity in an adaptive radiotherapy (ART) workflow. We have developed a novel machine learning framework to predict the patient-specific spatial presentation of the esophagus in the weeks following treatment, using magnetic resonance imaging (MRI)/ cone-beam CT (CBCT) scans acquired earlier in the 6 week radiotherapy course. Our algorithm captures the response patterns of the esophagus to radiation on a patch level, using a convolutional neural network. A recurrence neural network then parses the evolutionary patterns of the selected features in the time series, and produces a predicted esophagus-or-not label for each individual patch over future weeks. Finally, the esophagus is reconstructed, using all the predicted labels. The algorithm is trained and validated by means of ∼ 250 000 patches taken from MRI scans acquired weekly from a variety of patients, and tested using both weekly MRI and CBCT scans under a leave-one-patient-out scheme. In addition, our approach is externally validated using a publicly available dataset (Hugo 2017). Using the first three weekly scans, the algorithm can predict the condition of the esophagus over the succeeding 3 weeks with a Dice coefficient of 0.83 ± 0.04, estimate esophagus volume highly (0.98), correlated with the actual volume, using our institutional MRI/CBCT data. When evaluated using the external weekly CBCT data, the averaged Dice coefficient is 0.89 ± 0.03. Our novel algorithm may prove useful in enabling radiation oncologists to monitor and detect AE in its early stages, and could potentially play an important role in the ART decision-making process.

This brief review summarizes the major applications of artificial intelligence (AI), in particular deep learning approaches, in molecular imaging and radiation therapy research. To this end, the applications of artificial intelligence in five generic fields of molecular imaging and radiation therapy, including PET instrumentation design, PET image reconstruction quantification and segmentation, image denoising (low-dose imaging), radiation dosimetry and computer-aided diagnosis, and outcome prediction are discussed. This review sets out to cover briefly the fundamental concepts of AI and deep learning followed by a presentation of seminal achievements and the challenges facing their adoption in clinical setting.

To design a convolutional recurrent neural network (CRNN) that calculates three-dimensional (3D) positions of lung tumors from continuously acquired cone beam computed tomography (CBCT) projections, and facilitates the sorting and reconstruction of 4D-CBCT images.

Under an IRB-approved clinical lung protocol, kilovoltage (kV) projections of the setup CBCT were collected in free-breathing. Concurrently, an electromagnetic signal-guided system recorded motion traces of three transponders implanted in or near the tumor. Convolutional recurrent neural network was designed to utilize a convolutional neural network (CNN) for extracting relevant features of the kV projections around the tumor, followed by a recurrent neural network for analyzing the temporal patterns of the moving features. Convolutional recurrent neural network was trained on the simultaneously collected kV projections and motion traces, subsequently utilized to calculate motion traces solely based on the continuous feed of kV projections. To enhance performance, CRNN was also facilitated by frequent calibrations (e.g., at 10° gantry rotation intervals) derived from cross-correlation-based registrations between kV projections and templates created from the planning 4DCT. Convolutional recurrent neural network was validated on a leave-one-out strategy using data from 11 lung patients, including 5500 kV images. The root-mean-square error between the CRNN and motion traces was calculated to evaluate the localization accuracy.

Three-dimensional displacement around the simulation position shown in the Calypso traces was 3.4 ± 1.7 mm. Using motion traces as ground truth, the 3D localization error of CRNN with calibrations was 1.3 ± 1.4 mm. CRNN had a success rate of 86 ± 8% in determining whether the motion was within a 3D displacement window of 2 mm. The latency was 20 ms when CRNN ran on a high-performance computer cluster.

CRNN is able to provide accurate localization of lung tumors with aid from frequent recalibrations using the conventional cross-correlation-based registration approach, and has the potential to remove reliance on the implanted fiducials.

To design a deep learning algorithm that automatically delineates lung tumors seen on weekly magnetic resonance imaging (MRI) scans acquired during radiotherapy and facilitates the analysis of geometric tumor changes.

This longitudinal imaging study comprised 9 lung cancer patients who had 6-7 weekly T2-weighted MRI scans during radiotherapy. Tumors on all scans were manually contoured as the ground truth. Meanwhile, a patient-specific adaptive convolutional neural network (A-net) was developed to simulate the workflow of adaptive radiotherapy and to utilize past weekly MRI and tumor contours to segment tumors on the current weekly MRI. To augment the training data, each voxel inside the volume of interest was expanded to a 3 × 3 cm patch as the input, whereas the classification of the corresponding patch, background or tumor, was the output. Training was updated weekly to incorporate the latest MRI scan. For comparison, a population-based neural network was implemented, trained, and validated on the leave-one-out scheme. Both algorithms were evaluated by their precision, DICE coefficient, and root mean square surface distance between the manual and computerized segmentations.

Training of A-net converged well within 2 h of computations on a computer cluster. A-net segmented the weekly MR with a precision, DICE, and root mean square surface distance of 0.81 ± 0.10, 0.82 ± 0.10, and 2.4 ± 1.4 mm, and outperformed the population-based algorithm with 0.63 ± 0.21, 0.64 ± 0.19, and 4.1 ± 3.0 mm, respectively.

A-net can be feasibly integrated into the clinical workflow of a longitudinal imaging study and become a valuable tool to facilitate decision- making in adaptive radiotherapy.