thesis on brain tumor

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

View all journals
My Account Login
Explore content
About the journal
Publish with us
Sign up for alerts
Open access
Published: 12 September 2022

DACBT: deep learning approach for classification of brain tumors using MRI data in IoT healthcare environment

Amin ul Haq 1 ,
Jian Ping Li 1 ,
Shakir Khan 2 ,
Mohammed Ali Alshara 2 ,
Reemiah Muneer Alotaibi 2 &
CobbinahBernard Mawuli 1

Scientific Reports volume 12 , Article number: 15331 ( 2022 ) Cite this article

9508 Accesses

43 Citations

10 Altmetric

Metrics details

Computational biology and bioinformatics
Engineering
Mathematics and computing

The classification of brain tumors (BT) is significantly essential for the diagnosis of Brian cancer (BC) in IoT-healthcare systems. Artificial intelligence (AI) techniques based on Computer aided diagnostic systems (CADS) are mostly used for the accurate detection of brain cancer. However, due to the inaccuracy of artificial diagnostic systems, medical professionals are not effectively incorporating them into the diagnosis process of Brain Cancer. In this research study, we proposed a robust brain tumor classification method using Deep Learning (DL) techniques to address the lack of accuracy issue in existing artificial diagnosis systems. In the design of the proposed approach, an improved convolution neural network (CNN) is used to classify brain tumors employing brain magnetic resonance (MR) image data. The model classification performance has improved by incorporating data augmentation and transfer learning methods. The results confirmed that the model obtained high accuracy compared to the baseline models. Based on high predictive results we suggest the proposed model for brain cancer diagnosis in IoT-healthcare systems.

Segment anything in medical images

Prediction of tumor origin in cancers of unknown primary origin with cytology-based deep learning

Virtual reality-empowered deep-learning analysis of brain cells

Introduction.

Brain tumor (BT) is a series medical problem, and many people are suffering from it globally 1 . Because of its critical nature, brain tumours are one of the most dangerous types of brain cancer. Compared to other cancers from brain cancer less number of people are suffering 2 . Meningioma, Glioma, Pituitary, and Acoustic Neuroma are examples of brain tumors. In medical observation, the rates of Meningioma, GLioma, and Pituitary tumours in all brain tumors are 15%, 45%, and 15%, respectively 3 . A brain tumor has long-term and psychological consequences for the patient. Brain tumors are caused by tissue abnormalities that develop within the brain or the central spine, interfering with normal brain function. There are two types of brain tumors: benign and malignant. Benign brain tumors are not cancerous and grow slowly. They do not spread and are not common. Malignant brain tumours contain cancer cells and grow rapidly in one region of the brain and spread to other parts of the brain and spine.

The diagnosis of brain cancer is significantly necessary for early stage for effective treatment and recovery. In this regards to classify brain tumors and identify brain cancer, different non-invasive are developed in literature by researchers and medical experts in Internet of Things (IoT) healthcare industries. In the deigning of computer automatic diagnostic systems (CADS) for brain cancer detection Machine Learning (ML) and Deep Learning (DL) models are commonly used. The diagnosis of brain cancer using images data using the DL Convolution neural network (CNN) model has grown in popularity, and the CNN model is commonly used for image classification and analysis, particularly for medical image data analysis 4 . The CNNs model can extract more related features from data for accurate image classification 2 , 5 , 6 . Furthermore, data augmentation and transfer learning techniques can also improve the predictive capability of deep learning models to effective classify the brain tumors and diagnosis brain cancer in IoT healthcare industries 6 , 7 .

In the literature, various methods have been proposed for brain cancer diagnosis using ML and DL learning approaches by different scholars. Zacharaki et al. 8 designed a brain cancer diagnosis system to classify various grades of Glioma employing SVM and KNN machine learning model and respectively achieved 85% and 88% classification accuracy. Cheng et al. 9 proposed a classification approach for brain tumor classification and augmented the tumor region for improving the classification performance. They employed three techniques for feature extraction such as Gray level co-occurrence matrix, a bag of words, and an intensity histogram. Their proposed method obtained 91.28% classification accuracy.

Haq et al. 6 proposes an AI-based intelligent integrated framework (CNN-LSTM) for brain tumors classification and diagnosis in the IoT healthcare industry. In the integrated framework design, they have incorporated the CNN model to extract features from medical MRI data automatically. The extracted features are passed to Long short-term memory (LSTM) model to learn the dependencies in the features and finally predict the class for the tumor. Further they applied brain MRI data sets for the assessment of the proposed integrated model. Massive data is one requirement for an effective deep learning model. Since the size of our original data set is small, they utilized data augmentation approaches to increase the data set size, thereby improving the model result during training. Also used the train-test splits Cross-validation approach for hyperparameter tuning and best model selection to ensure proper model fitting. For model assessment, used well-known evaluation measures. They compared the predictive outputs of the proposed CNN-LSTM model with previous methods in the Medical Internet of Things (MIoT) healthcare industry and the model obtained high predictive performance.

Paul et al. 4 employed axial brain tumor images for convolution neural network training. In the proposed method they used two convolution layers, two max-pooling layers, and lastly, two fully connected layers for the final classification process. The proposed approach obtained 91.43% classification accuracy. El-dahshan et al. 10 designed a brain tumors classification method for 80 brain images MRI classification. They used discrete wavelet transform and PCA algorithms for reducing dimensions of data. To classify the normal and abnormal tumors, they used ANN and KNN machine learning classifiers. The classifiers ANN and KNN, achieved 97% and 98% classification accuracy respectively.

In another study, Afshar et al. 11 proposed a brain tumor classification method employing a capsule network that combined MRI images of the brain and coarse tumor boundaries and 90.89% accuracy achieved by the proposed method. Anaraki et al. 12 developed an integrated framework for brain tumor classification, and in the proposed technique, they integrated CNN and GA, and designed GA-CNN framework and obtained 94.2% accuracy. Khan et al. 13 proposed brain tumors classification method employing transfer learning techniques (CNN-Transfer learning) and achieved 94.82% accuracy 14 . The proposed multi-classification method employing ensemble of deep features and ML algorithms and obtained high performance.

According to the review of the literature, current brain cancer diagnosis techniques still lack a robust predictive capability in terms of accuracy to correctly diagnose brain cancer for proper treatment and recovery. To address this issue, a novel robust method for accurately diagnosing brain cancer for proper treatment and recovery in IoT healthcare industries is required. Furthermore, the artificial intelligence based brain cancer diagnosis systems also reduce the financial costs of healthcare department.

In this study, we created an improved CNN model for the classification of brain MR images to diagnosis brain cancer in IoT healthcare industries. In the development of the proposed model, we used Convolution neural network model to classify brain tumors types (Meningioma, Glioma and Pituitary) employing MR images data. The CNN model is more suitable for the Meningioma, Glioma, and pituitary classification using brain tumors images data and its extract more deep features from images data for final classification. To further improve the CNN model predictive capability, we have incorporated a transfer-learning (TL) techniques for proper training of the CNN architecture, the brain MR images data is insufficient. In transfer learning, we used the well-known pre-trained models ResNet-50, VGG-16, Inception V3, DenseNet201, Xception, and MobilleNet. The weights generated of these pre trained models individually transferred to CNN architecture for effective training o CNN model. For the fine-tuning process, the model was trained with brain MR images data set. The generated weights of pre trained models improving CNN model final predictive performance. Additionally, the data augmentation technique is incorporated to increase the data set size for effective training of the model. We also used held-out cross-validation (CV) and performance evaluation metrics. The performance of the model compared with base lines models. The experimental results confirmed that the proposed model generated higher predictive results and it could be applied in IoT-healthcare systems easily.

Innovations of this study summarized as follows:

In IoT healthcare systems, an improved model based on CNN and TL for classifying brain tumors using MR image data is proposed for diagnosis of brain cancer.

To increase the predictive accuracy of the CNN model, TL techniques are used because the brain tumor image data is insufficient for effective training of the CNN model. Pre-trained models ResNet-50, VGG-16, Inception V3, DenseNet201, Xception, and MobilleNet are used to train with the well-known ImageNet data set for generating trained parameters (weights). The weights of these pre tained models are individually transfer to CNN model effective training. Fine-tuning the model CNN with brain tumor images data along with transferred weights final classification.

To improve model performance, the data augmentation technique is used to increase the size of the data set for effective model training.

When compared to baseline methods, our model has a high predictive performance.

The rest of the paper is organized as follows: In “ Materials and method ” section data set and proposed model methodology have explored. In “ Experiments ” section the experiments are reported. In “ Discussion ” section, we discussed the significance of the work. The conclusion and research direction of future work are reported in “ Conclusion ” section.

Materials and method

We used a brain tumor data set (BTDS) from China’s Nanfang hospital and general hospital, as well as Tianjing medical university, in this study (2005 to 2010) 9 , and new versions in 2017 have been published. T1-Weighted Contrast-Enhanced images (TWCEI) of 233 subjects with meningioma, glioma, and pituitary tumours are included in this data set. The data set is freely accessible via the Kaggle repository 15 . We also used the Brain MRI Images Data Set (BMIDS) for cross dataset validation, which contains 253 MRI brain images. The tumor class in the data set has 155 images, while the non-tumor class has 98 images 16 .

Background of convolutional neural network (CNN) architecture

Deep Learning model convolutional neural networks is a kind of Feed-Forward Neural Network 17 . Convolutions can capture translation invariance, which means that the filter is independent of position that significantly reduces the number of parameters. The CNN model have Convolutional, Pooling, and fully connected layers. Different functions are accomplished by these layers, such as dimensionality reduction, feature extractors, and classification. During the convolution operation of the forward pass, the filter is slide on the input shape and compute the map of activation, which computing the point-wise value of each output. Further add these output to achieve the activation of that point. Designed a Sliding Filter (SF) using convolution as a linear operator, and expressed as a dot product for fast deployment. Let consider x and w are input and the kernel function, the convolution process $(x*w)(a)$ on time index t can be mathematically expressed in Eq. ( 1 ).

In Eq. ( 1 ) a is in $\text{ R}^n$ for any $n \ge 1$ . While Parameter t is discrete. In this case, the discrete convolution can be expressed as in Eq. ( 2 ):

However, usually use 2 or 3-dimensional convolutions in CNN model. In case of 2-dimensional image I as input and K is a two dimensional kernel and the convolution can be mathematically expressed as in Eq. ( 3 ):

If the case is 3 dimensional data image, then the convolution process can be written mathematically in Eq. ( 4 ) as follow:

In addition to gain non-linearities, two activation functions can be incorporate suc as Sigmoid and ReLU. The sigmoid activation fumction non-linearity is expressed mathematically in Eq. ( 5 ):

The sigmoid non-linearity activation function is suitable when need the output to be include in the range of [0,1]. Furthermore, the sigmoid function is monotone growing which means $\lim \limits _{n \rightarrow +\infty } \theta (x)=1$ , and $\lim \limits _{n \rightarrow +\infty } \theta (x)=0$ . However, this fact may be cause vanishing gradients, when the input x is not near to 0, the neuron will be more and the gradient of $\theta (x)$ will nearly to zero and will make successive optimization difficult.

The second activation function is relu which is mathematically defined in Eq. ( 6 ):

The gradient of of $relu(x)=1$ for $x>0$ and $relu^-(x)=0$ for $x<0$ . The relu convergence capability of is good then sigmoid non-linearities.

The CNN model Pooling layers are utilized to produce a statistics summary of its inputs and deduced the dimensionality without missing important information. There are different types of pooling. In the layer of Max-Pooling generate the extreme values in individually rectangular neighborhood of individual point i.e i, j, k for data of three dimensional of individual feature of input respectively, while the average values generated by the average pooling layer.

The last layer is fully connected with n and m respectively input and output sizes. The output layer is expressed by the parameters such as a weight matrix i.e $W \in M_{m, n}$ with m rows, and n columns and a bias vector $b \in {\textbf {R}}^m$ . The input vector $x \in {\textbf {R}}^n$ , the fully connected output layer FC along function of activation f is expressed mathematically in Eq. ( 7 ) as:

In Eq. ( 7 ) Wx is the product matrix while the function f is used component wise.

The last layers fully connected employed for classification of problems. The CNN model architecture last layer is fully connected layers and CNN output is flattened and showed as a single vector.

Convolution neural network for brain tumors classification

Recently, CNN models generated significant outcomes in numerous domains, such as NLP, image classification 18 , and diagnosis systems. In contrast to MLPs, CNN reduces the number of neurons and parameters, which results in lower complexity and faster adaptation.

The CNN model has significant applications in the classification of medical images 18 , 19 . In this paper we developed the CNN networks architecture with 4 alternating convolutional layers and max-pooling layers and a dropout layer after each Conv/pooling pair. The last pooling layer connected fully layer with 256 neurons, ReLU activation function, dropout layer, and sigmoid activation function are employed for classification of brain MR images (Meningioma, Glioma, and Pituitary). In addition, we have used the optimization algorithm Stochastic Gradient Descend (SGD) 20 . The CNN architecture is given in Fig. 1 .

CNN model architecture for classification of Brain tumors.

Improve CNN model for brain tumors classification

To improve CNN model predictive accuracy, we employed Data augmentation (DA) and Transfer learning (TL) techniques. The data augmentation can resolve the problem of insufficient data for model training. To expand the data amount, the zooming technique is used on original image data to produce images data with the similar label. The new created data set is used for fine tuning of the model. Th The transfer learning (TL) techniques widely used in image classification tasks 21 , cancer sub-type recognition 22 and medical images filtering 23 . In this work, we used the transfer learning ResNet-50, VGG-16, Inception V3, DenseNet201, Xception, and MobilleNet models to enhanced the predictive performance of the proposed CNN model. The ResNet-50, VGG-16, Inception V3, DenseNet201, Xception, and MobilleNet pre-train models were trained on imageNet data set and transferred the trained parameters weights of these models individually to CNN model for effective training, and fine-tuned the model using the brain tumor augmented MR images data set for final classification of the CNN model.

Model cross validation and evaluation criteria

The holdout cross-validation 6 , 24 , 25 mechanism was used for training and validation of the model. In hold out CV data is randomly assign to two sets $d_0$ and $d_1$ . The $d_0$ and $d_1$ use for training and testing of the model respectively. In hold out CV the training data set is usually large as compare to testing data set. The is train on $d_0$ and testing on $d_1$ . The holdout CV is suitable validation method in case when the data set is very plenty. In this study brain tumor MRI Images data set was divided into 70% for training and $30\%$ for teasing of the model. The performance evaluation metrics Accuracy (Acc), Sensitivity (Sn), Specificity (Sp), Precision (Pr), F1-Score (F1-S), and Matthews Correlation Coefficient (MCC) 26 , 27 , 28 , 29 are used for model evaluation.

Proposed brain tumors classification model

NCNN models are now popular for image classification problems. A large image data set is more suitable for the CNN model’s effective training, as it allows the model to extract more related features during the training process for accurate image classification. The CNN model’s performance suffers as a result of the scarcity of large image data sets, particularly in the medical domain. However, to enhance the proposed CNN classifier performance, data augmentation and transfer learning 6 , 21 , 30 , 31 techniques are incorporated. We have used transfer learning pre-trained models ResNet-50, VGG-16, Inception V3, DenseNet201, Xception, and MobilleNet along with data augmentation technique zooming. The imagesNet data set has been employed for pre-trained of ResNet-50, VGG-16, Inception V3, DenseNet201, Xception, and MobilleNet models, and the generated weights (trained parameters) of these models were transferred for the effective training of the CNN model individually. Brain tumor MRI data set was used for fine-tuning of CNN model and for final classification of the model in IoT healthcare system.

Furthermore, the proposed CNN model was trained and tested on a data set of brain tumour MR images, and its performance was compared to that of the transfer learning technique. A heldout cross-validation mechanism is used in the proposed method for model training and testing, with 70% used for training and 30% for model validation. The data augmentation 20 technique was used to augment the original dataset by using the zooming method, which improves the model generalisation capability. The integration of data augmentation and transfer learning greatly enhanced the predictive accuracy of the CNN model. The evaluation criteria of the model different assessment metrics have used.

The data set X ( i , i ) embedded into the CNN classifier,We used data transformations to increase the size of the data set so that we could train the model. Furthermore, the number of epochs E , model parameters w , Learning Rate (LR) $\eta$ , size of batch b , and the number of layers in both CNN were configured accordingly. For the optimization of our model parameters, we have used the stochastic gradient descent algorithm (SGD). The pseudo-code of the proposed model is given in algorithm 1 and flow chart in Fig. 2 .

Flow chart of proposed tumor classification framework in IoT healthcare systems. The pre trained CNN models (ResNet-50, VGG-16, Inception V3, DenseNet201, Xception, and MobilleNet) are trained with image-net dataset and the generated weights of these pre trained models are individually transferred to proposed CNN model for effective training. While the augmented data set is used for fine-tuning of the ResNet-CNN model for final classification of brain tumors.

Experiments

Experimental setup.

We conducted various experiments to test the feasibility of our proposed model in IoT healthcare system. The proposed model was tested using a brain tumour image data set in this study. To improve the proposed CNN model predictive performance, we have employed (ResNet-50, VGG-16, Inception V3, DenseNet201, Xception, and MobilleNet) CNN pre trained models with imagenet dataset to produce high trained parameters (weights) and then transferred trained parameters weights of pre trained models to the CNN model individually for effective training of the model. For fine-tuning the CNN model, the brain tumor images data set was employed for final classification. The brain tumor data have 233 subjects and 3064 slices, which belong to three classes, i.e., Meningioma, Glioma, and Pituitary. This data set is very Small for effective training of the CNN model. In addition to tackle the problem of small brain tumor data method of data augmentation 20 has used to augment the original data set. Data augmentation technique (zooming) is used, and all three types of images (Meningioma, Glioma, and Pituitary) are zoomed horizontally and vertically and added with existing images. The new augmented data set image size of three kinds of images is 6128. Held out technique is used for model training and validation, and respectively 70% and 30% data are employed for training and validation of the model for all experiments. To effectively optimize the model SGD Optimization algorithm is used 20 . In addition, other parameters such as learning rate (LR) $\alpha$ , SGD = 0.0001, epochs = 100, batch size = 120, outer and inner activation function = ReLu is used in all experiments. It is worth noting that for the final prediction layer our CNN model, the softmax activation function was used. Evaluation metrics are incorporated to evaluate the model performance.

All experiments used a laptop and a Google collaborator with GPU. All experiments required Python v3.7, and the CNN model was created using Keras framework v2.2.4 as a high-level API and Tensor flow v1.12 as the back end. All experiments were repeated numerous times to obtain consistent results. All experiment results were tabulated and graphed.

Results and analysis

Results of data pre-processing.

The brain tumor data set (BTDS) is obtained from the Kaggle repository 15 . T1-weighted contrast-enhanced images of 233 meningioma, glioma, and pituitary tumour patients are included in this data set. The Brain Tumor data contains 233 subjects and 3064 slices, with meningioma subjects accounting for 82 with slices 708, glioma subjects accounting for 91 with slices 1426, and pituitary subjects accounting for 60 with slices 930. Thus, the total number of subjects in the data is 233, and the total number of slices is 3064. In order to reduce the dimension of $512\times 512\times 1$ into $224\times 224\times 1$ for effective training of model.

To handle imbalance problem in data set because Brain tumor data set has the different number of three subjects slices. The distribution of the data is different, and it creates a problem of over fitting the model. To balance the meningioma, glioma, and pictutitary in the data set, we incorporate the data augmentation 20 method to augment the original dataset by using random zooming. All slices are being zoomed, and a new data set with 6128 slices has been created. The ratio of samples in an original data set is shown in Fig. 3 . The data set has three subfolders for meningioma, glioma, and pictutitary images. Held out techniques is used for model training and validation because the new data set is very big and heldout validation is suitable in case of plenty dataset. The data set has splitted into 70% and 30% for training and validation of the model respectively. The cross-validation method has also been employed for an augmented data set.

We also used the Brain MRI Images Data Set (BMIDS) for cross dataset validation, which contains 253 MRI brain images. The tumor class in the data set has 155 images, while the non-tumor class has 98 images.

Ratio of samples in data set.

Results of the proposed CNN model, on original and augmented data sets

The performance of the proposed CNN model is evaluated using the original and augmented brain tumour MR image data sets. The CNN model is configured with essential hyper-parameters such as optimizer SGD with a Learning Rate (LR) of 00.0001, epochs 100, and size of batch was 120. The 70% data for training and 30% for the testing of the model is used. Different evaluation matrices were used for model performance evaluation. The input image size $264\times 264\times 1$ is used for training and evaluation of the proposed CNN model. All these hyper-parameters values and the output of the experimental results have been reported in Table 1 .

Table 1 presented the proposed CNN model obtained 97.40% accuracy, 98.03% specificity, 95.10% sensitivity, 99.02% Precision, 97.75% MCC, and 97.26% F1-score respectively on original brain tumor MR images data set. The 97.40% accuracy demonstrated that our CNN architecture accurately classifies the three classes of brain tumors (meningioma, glioma, and pictutitary). The 98.03% specificity shows that the Proposed CNN model is a highly suitable detecting model for healthy subjects recognition, while 95.10% sensitivity presents that the model significantly detected the affected subjects. The MCC value was 97.75%, which gives confusion metrics a good summary.

On the other hand, the CNN model gained very excellent performance when trained and evaluated on an augmented data set. The CNN model obtained 98.56% accuracy, 100.00% specificity, 98.09% sensitivity, and 98.00% MCC when trained and evaluated on an augmented data set. The accuracy of the model improved from 97.40 to 98.56% which demonstrated the importance of the data augmentation process. Also, it illustrated that model needs more data for effective training of the CNN model.

From the experimental results, we concluded that the proposed CNN model effectively classified the brain tumor types, and the augmentation process further improved the model CNN performance because the CNN model more data for extract more related features for classification. The high accuracy of the proposed CNN model might be due to the suitable architecture of the CNN model and proper fitting of essential parameters of the model and data augmentation.

CNN model performance evaluation with cross dataset

We have evaluated the predictive performance of CNN model with independent cross dataset. We trained the proposed CNN model with original and augmented brain tumor data set and validated with independent Brain MRI Images Data Set (BMIDS). The model is configured with essential hyper-parameters such as optimizer SGD with a Learning Rate (LR) of 00.0001, epochs 100, and size of batch was 120. Different evaluation matrices were used for model performance evaluation. The input image size $264\times 264\times 1$ is used for training and evaluation of the proposed CNN model. The experimental results of model with cross data are reported in Table 2 .

Table 2 presented that the proposed CNN model obtained 97.96% accuracy, 99.00% specificity, 97.30% sensitivity, 98.18% Precision, 98.00% MCC, and 99.02% F1-score when trained on original brain tumor MR images data set (BTDS) and validated with independent data set (BMIDS).

Other other side the model achieved 98.97% accuracy, 99.89% specificity, 99.39% sensitivity, 98.89% Precision, 99.40% MCC, and 99.30% F1-score when trained with augmented data set (BTDS) and validated with independent data set (BMIDS). Hence, from experimental results we observed that model predictive and generalization capability improved when trained and validated with independent data sets.

Results of the transfer learning models (ResNet-50, VGG-16, Inception V3, DenseNet201, Xception, and MobilleNet) on original and augmented data sets

The performances of transfer learning (ResNet-50, VGG-16, Inception V3, DenseNet201, Xception, and MobilleNet) models have checked on original and augmented data sets. Theses models have configured other essential hyper-parameters such as optimizer SGD with learning rate 0.0001, the number of epoch 100, batch size 120. The input image size $264\times 264\times 1$ is used for training and evaluation of the proposed model. The (ResNet-50, VGG-16, Inception V3, DenseNet201, Xception, and MobilleNet) models are evaluated using different performance evaluation metric. The (ResNet-50, VGG-16, Inception V3, DenseNet201, Xception, and MobilleNet) models hyper-parameters values and the output of the experimental results have reported in Table 3 .

Table 3 show that the ResNet-50 model obtained 97.03% accuracy, 97.04% specificity, 93.10% sensitivity, 94.21% Precision, 93.23% MCC, and 95.00% F1-score respectively on original brain tumor data set. The 95.30% accuracy show that the ResNet-50 model accurately classifies the three classes of brain tumors (meningioma, glioma, and pictutitary). The 97.04% specificity shows that the ResNet-50 model is a highly suitable detecting model for healthy subjects recognition, while 93.10% sensitivity show that the model accurately detected the affected subjects.

The predictive Performance of transfer learning model ResNet-50 very high when model trained and evaluated with augmented data set. According to Table 3 the transfer learning model ResNet-50 obtained 98.07% accuracy, 99.30% specificity, 100.00% sensitivity, 96.07% precision, 96.00% MCC, and 97.00% F1-S, when trained and evaluated on augmented data set.

The VGG-16 model with original and augmented data sets obtained 94.77% accuracy, 96.30% specificity, 94.67% sensitivity, 93.43% precision, 91.90% MCC, 96.61% F1-S, and 95.97% accuracy, 96.95% specificity, 99.40% sensitivity, 96.84% precision, 92.98% MCC, and 96.80% F1-S respectively.

Inception V3 obtained 93.23% accuracy, 96.89% specificity, 95.00% sensitivity, 96.08% precision, 95.56% MCC, and 97.87% F1-s, with original data set. While on augmented data set Inception V3 obtained 96.03% accuracy, 97.03% specificity, 97.00% sensitivity, 97.01% precision, 96.05% MCC, 98.00% F1-S. DenseNet201 model obtained 96.76% accuracy on original data set and increase it 97.43% accuracy with augmented data set.

The Xception model with original data set achieved 93.00% accuracy, 97.03% specificity, 98.00% sensitivity, 97.09% precision, 99.32% MCC, 97.23% F1-S and obtained 95.60% accuracy, 98.98% specificity, 96.00% sensitivity, 98.04% precision, 99.98% MCC, and 98.00% F1-S with augmented data set. MobilleNet model obtained 96.76% accuracy with original data set and 97.87% with augmented data set. Among all models the ResNet-50 model performance in terms of accuracy is high with augmented data set. The model improved accuracy from 95.30 to 98.07% with data augmentation. The other evaluation metrics values also improved with data augmentation. From the experimental results, we concluded that the data augmentation process increased the training of ResNet-50 and model effectively classified the brain tumor types.

Results of the integrated frameworks (ResNet-50-CNN, VGG-16-CNN, Inception V3-CNN, DenseNet201-CNN, Xception-CNN, and MobilleNet-CNN) on original and augmented data sets

The integrated frameworks (ResNet-50-CNN, VGG-16-CNN, Inception V3-CNN, DenseNet201-CNN, Xception-CNN, and MobilleNet-CNN) performances have checked on original and augmented data sets. Furthermore, we have incorporated the TL ResNet-50, VGG-16, Inception V3, DenseNet201, Xception, and MobilleNet CNN architectures with imageNet data set to generate high weights and then transferred trained parameters weights of these pre trained models to the CNN model individually for effective training of CNN model. For fine-tuning of the CNN model, the brain tumors original and augmented data sets have used for final classification. The models have configured with concern hyper-parameters such as optimizer SGD with learning rate 0.0001, the number of epoch 100, batch size 120. The proposed framework performance has been evaluated employing various matrices. The input image size $264\times 264\times 1$ has been used for training and evaluation of the proposed model. All these hyper-parameters values and the output of the experimental results of (ResNet-50-CNN, VGG-16-CNN, Inception V3-CNN, DenseNet201-CNN, Xception-CNN, and MobilleNet-CNN) models have reported in Table 4 .

Table 4 presented that the ResNet50-CNN model obtained 99.10% accuracy, 100.00% specificity, 89.60% sensitivity, 98.75% Precision, 98.66% MCC, and 99.5% F1-score respectively on original brain tumor data set. The 99.10% accuracy demonstrated that the architecture accurately classifies the three classes of brain tumors (meningioma, glioma, and pictutitary). The 100% specificity shows that the Proposed model is a highly suitable detecting model for healthy subjects recognition, while 89.60% sensitivity presents that the model significantly detected the affected subjects.

On the other hand, the model obtained very high performance when it trained and evaluated on the augmented data set. The integrated CNN and transfer learning model (ResNet-50-CNN) obtained 99.90% accuracy, 99.08% specificity, 96.13% sensitivity, and 99.10% MCC when trained and evaluated on augmented data set.

The VGG-16-CNN model with original and augmented data sets obtained 96.78% accuracy, 99.23% specificity, 95.00% sensitivity, 96.99% precision, 98.93% MCC, 97.98% F1-S, and 97.88% accuracy, 98.00% specificity, 100.00% sensitivity, 96.98% precision, 98.79% MCC, and 99.00% F1-S respectively.

Inception V3-CNN model obtained 97.00% accuracy, 99.00% specificity, 99.87% sensitivity, 98.92% precision, 95.76% MCC, 98.09% F1-S with original data set. While on augmented data set Inception V3 obtained 98.02% accuracy, 100.00% specificity, 98.67% sensitivity, 97.56% precision, 99.00% MCC, and 97.30% F1-S.

DenseNet201-CNN model obtained 97.00% accuracy on original data set and increase it 97.90% accuracy with augmented data set. Hence, the integrated model DenseNet201-CNN improved accuracy 97.00–97.90% = 0.90% with data augmentation process.

The Xception-CNN model with original data set achieved 98.20% accuracy, 98.88% specificity, 97.40% sensitivity, 99.00% precision, 99.10% MCC, 98.65% F1-S, and obtained 98.97% accuracy, 99.00% specificity, 98.60% sensitivity, 97.24% precision, 97.99% MCC, 99.30% F1-S with augmented data set. MobilleNet-CNN model obtained 98.08% accuracy with original data set and 98.56% with augmented data set. The improved accuracy 98.08% to 98.56% when model fine tuned with augmented data set.

From above anlaysis we conculded that among all the ResNet-50-CNN, VGG-16-CNN, Inception V3-CNN, DenseNet201-CNN, Xception-CNN, and MobilleNet-CNN, the predictive performance of ResNet-50-CNN model is high in terms of accuracy. The accuracy of the model improved from 99.10 to 99.90% which is illustrated the importance of the data augmentation and transfer learning process. Hence we concluded that the ResNet-50-CNN model effectively classify the brain tumor types. The high accuracy of the proposed integrated diagnosis framework might be due to the suitable architecture of the model and proper fitting of essential parameters of the model and data augmentation. In addition, the proposed integrated model (ResNet-50-CNN) accuracy has compared with CNN model and transfer learning ResNet-50 model in Table 5 on augmented data set and graphically shown in Fig. 4 .

CNN, ResNet-50 and Integrated (ResNet-50-CNN) models accuracy comparison with augmented Brain tumor data set. The CNN model obtained accuracy’s with augmented data is 98.97%, while ResNet-50 obtained 96.07% and Integrated model ResNet-CNN obtained high predictive accuracy 99.90% with augmented data. Thus, the proposed integrated ResNet-CNN model is suitable for effective classification of brain tumors and could assist clinical professionals to diagnosis brain cancer accurately and efficiently. Due to the high performance of proposed ResNet-CNN method we recommend it for diagnosis of brain cancer in IoT-healthcare.

Accuracy comparison of the proposed (ResNet-CNN) model with state of-the-art models

We have compared our ResNet-50-CNN (ResNet-CNN) model performance in terms of accuracy with state-of-the-art methods in Table 6 . Table 6 and Fig. 5 presented that proposed model obtained 99.89% accuracy, which is high as compared to state-of-the-art techniques. The high performance of the proposed method demonstrated that it is correctly classified brain tumors (meningioma, glioma, and pictutitary), and it can easily be deployed in IoT-health care for the classification of brain tumors.

ResNet-CNN model performance comparison with baseline models show that our model predictive performance in terms of accuracy is high from baseline models. The ResNet-CNN model cloud accurately and efficiently classify the brain tumors and assist medical experts to interpret the images of brain tumors to diagnosis brain cancer.

Space and time complexity

Also, in Tables 3 , 4 , and 6 , we present both the models space and complexity of the various proposed methods used in the prediction of Brain cancer. Since the proposed models are convolutional deep learning methods, the space complexities are analyzed in terms of the each model’s trainable parameters. For the time complexity, the model’s training time is used. It could be deduced from Table 3 that VGG-16 has the worst space complexity since its trainable parameter is 138.4 million, whiles MobileNet has the best space time complexity. Moreover for the time complexity, the Xception model has the worst time complexity because its training time is 4.3 h. Because of the difficulty of accessing the models of the competing methods in Table 4 , we could not experimentally analyze the complexity of the models in terms of algorithmic run-time. It is more likely that almost all the methods with the deep learning techniques, the convolutional neural networks will have a worse space and time complexity because of the significant number of parameters and matrix computation that come with the models’ architecture. Irrespective of the worst case time and space complexity, our proposed model has an accuracy performance gain as compared to all competing methods. The time complexity is the training time (in hours) of the models as reported in Tables 3 , 4 , and 6 . The space and time complexity of our model are $\mathscr {O}(cwh + 1)f$ and $\mathscr {O}(f*u*m)$ respectively.

Brain Tumor Classification using MR images are critical in the detection of brain cancer In IoT healthcare systems. Artificial intelligence (AI) based computer automatic diagnostic systems (CAD) can effectively different diagnose diseases in IoT healthcare system. Deep learning techniques are widely used in CAD systems to diagnose critical diseases 32 , especially convolutional neural networks. The CNN model is mostly used for medical image classification 18 , 19 . The CNN model extracts deep features from image data, and these features played an important role in final image classification. For the diagnosis of brain cancer, various methods have been proposed by researchers using brain MR image data and deep learning models. However, these existing methods have lack of accuracy of diagnosis. In order to tackle this problem, a new method is necessary to diagnose the disease accurately and efficiently IoT healthcare systems.

In this study, we have proposed a CNN model for the accurate classification of brain tumor using Brain MR images. In the design of the proposed method, we have applied the deep learning CNN model for the classification of tumors meningioma, gLioma, and pituitary. The CNN model extracts more deep features from image data for final classification. To further improve the CNN model predictive capability, we have incorporated a transfer learning mechanism because, for proper training of the CNN architecture, the brain MR images data is insufficient. In transfer learning, we have used the well-known pre-trained models (ResNet-50, VGG-16, Inception V3, DenseNet201, Xception, and MobilleNet) with big imageNet data set to generate high parameters (weights). These generated weights of models individually transferred to CNN model for effective training. For the fine-tuning process, the model was trained with brain MR images data set. Also, the data augmentation method is employed to increase the data set size for effective training of the model. Furthermore, we have used held-out cross-validation and performance evaluation metrics. We also used cross data set for cehcking the propoed CNN model predictice performance.

According to Tables 2 , 3 , 4 and 6 the proposed method obtained high results as compared to baseline methods. The high performance of the proposed ResNet-CNN model might be due to the proper setting of model parameters such as learning rate, batch size, number of the epoch, and pre-processing, and data augmentation. We recommend the proposed method for meningioma, gLioma, and pituitary classification. Furthermore, the proposed method would be applied for diagnosis of a brain cancer in IoT-Healthcare systems easily.

For accurate medical image classification, the CNN model is played a significant role, and in most CAD systems CNN model is used for the analysis of medical image data. In research study, we have proposed a deep learning-based diagnosis approach for brain tumor classification. In the proposed method, we have used a deep CNN model for the classification of tumor types Meningioma, Glioma, and Pituitary employing brain tumor MR images data. To enhance the predictive capability of the CNN model, we have incorporated transfer learning and data augmentation techniques. The experimental results show that the proposed integrated diagnosis framework ResNet-CNN has obtained 99.90% accuracy as compared to baseline methods. The high predictive outcomes of the proposed method might be due to the effective pre-processing of data and the adjustment of other parameters of the model such as numbers of layers, optimizer and activation functions, transfer learning, and data augmentation. Due to the high performance of the proposed ResNet-CNN model, it could be applicable for the classification of brain tumors and diagnosis of brain cancer in IoT-Healthcare. In the future, we will use other brain tumors datasets and other deep learning techniques to diagnose brain tumors.

Data availibility

The data sets we used in this study are available on the kaggle machine learning repository at linked below: (1) Brain tumor dataset ( https://www.kaggle.com/datasets/awsaf49/brain-tumor ), and (2) Brain MRI Images for Brain Tumor Detection data set ( https://www.kaggle.com/datasets/navoneel/brain-mri-images-for-brain-tumor-detection ). All methods were performed in accordance with the relevant guidelines and regulations.

Roser, M. & Ritchie, H. Cancer. Our World in Data (2015). https://ourworldindata.org/cancer .

Khan, H. A., Jue, W., Mushtaq, M. & Mushtaq, M. U. Brain tumor classification in MRI image using convolutional neural network. Math. Biosci. Eng 17 , 6203–6216 (2020).

Article MathSciNet Google Scholar

Swati, Z. N. K. et al. Content-based brain tumor retrieval for MR images using transfer learning. IEEE Access 7 , 17809–17822 (2019).

Article Google Scholar

Sultan, H. H., Salem, N. M. & Al-Atabany, W. Multi-classification of brain tumor images using deep neural network. IEEE Access 7 , 69215–69225 (2019).

Bishop, C. M. Pattern Recognition and Machine Learning (Springer, 2006).

MATH Google Scholar

Haq, A. U. H. et al. IIMFCBM: Intelligent integrated model for feature extraction and classification of brain tumors using MRI clinical imaging data in IoT-healthcare. IEEE J. Biomed. Health Inform. https://doi.org/10.1109/JBHI.2022.3171663 (2022).

Article PubMed Google Scholar

Haq, A. U. & Li, J. P. Stacking approach for accurate invasive ductal carcinoma classification. Comput. Electr. Eng. 100 , 107937 (2022).

Zacharaki, E. I. et al. Classification of brain tumor type and grade using MRI texture and shape in a machine learning scheme. Magn. Reson. Med. 62 , 1609–1618 (2009).

Cheng, J. et al. Enhanced performance of brain tumor classification via tumor region augmentation and partition. PLoS ONE 10 , e0140381 (2015).

El-Dahshan, E.-S.A., Hosny, T. & Salem, A.-B.M. Hybrid intelligent techniques for MRI brain images classification. Digit. Signal Process. 20 , 433–441 (2010).

Afshar, P., Plataniotis, K. N. & Mohammadi, A. Capsule networks for brain tumor classification based on mri images and coarse tumor boundaries. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 1368–1372 (IEEE, 2019).

Anaraki, A. K., Ayati, M. & Kazemi, F. Magnetic resonance imaging-based brain tumor grades classification and grading via convolutional neural networks and genetic algorithms. Biocybern. Biomed. Eng. 39 , 63–74 (2019).

Swati, Z. N. K. et al. Brain tumor classification for MR images using transfer learning and fine-tuning. Comput. Med. Imaging Graph. 75 , 34–46 (2019).

Kang, J., Ullah, Z. & Gwak, J. MRI-based brain tumor classification using ensemble of deep features and machine learning classifiers. Sensors 21 , 2222 (2021).

Article ADS Google Scholar

BT. Brain tumor. Kaggle: Machine Learning Repository. https://www.kaggle.com/awsaf49/brain-tumor (Accessed 12 Mar 2021) (2022).

BT. Brain tumor. Kaggle: Machine Learning Repository https://www.kaggle.com/navoneel/brain-mri-images-for-brain-tumor-detection (Accessed 10 Mar 2022) (2022).

Zhang, S., Yao, L., Sun, A. & Tay, Y. Deep learning based recommender system: A survey and new perspectives. ACM Comput. Surv. 52 , 5 (2019).

Google Scholar

Cai, J., Lu, L., Xie, Y., Xing, F. & Yang, L. Improving deep pancreas segmentation in CT and MRI images via recurrent neural contextual learning and direct loss function. arXiv preprint arXiv:1707.04912 (2017).

Wolz, R. et al. Automated abdominal multi-organ segmentation with subject-specific atlas generation. IEEE Trans. Med. Imaging 32 , 1723–1730 (2013).

Goodfellow, I., Bengio, Y., Courville, A. & Bengio, Y. Deep Learning Vol. 1 (MIT Press, 2016).

Schwarz, M., Schulz, H. & Behnke, S. RGB-D object recognition and pose estimation based on pre-trained convolutional neural network features. In 2015 IEEE International Conference on Robotics and Automation (ICRA) , 1329–1335 (IEEE, 2015).

Hajiramezanali, E., Dadaneh, S. Z., Karbalayghareh, A., Zhou, M. & Qian, X. Bayesian multi-domain learning for cancer subtype discovery from next-generation sequencing count data. arXiv preprint arXiv:1810.09433 (2018).

Bickel, S. ECML-PKDD discovery challenge 2006 overview. In ECML-PKDD Discovery Challenge Workshop , 1–9 (2006).

Haq, A. U., Li, J. P., Memon, M. H., Nazir, S. & Sun, R. A hybrid intelligent system framework for the prediction of heart disease using machine learning algorithms. Mob. Inf. Syst. 2018 (2018).

Li, J. P. et al. Heart disease identification method using machine learning classification in e-healthcare. IEEE Access 8 , 107562–107582 (2020).

Yurttakal, A. H., Erbay, H., İkizceli, T., Karaçavus, S. & Çinarer, G. A comparative study on segmentation and classification in breast MRI imaging. IIOAB J. 9 , 23–33 (2018).

Gallego-Ortiz, C. & Martel, A. L. Improving the accuracy of computer-aided diagnosis for breast MR imaging by differentiating between mass and nonmass lesions. Radiology 278 , 679–688 (2016).

Yang, Q., Li, L., Zhang, J., Shao, G. & Zheng, B. A new quantitative image analysis method for improving breast cancer diagnosis using DCE-MRI examinations. Med. Phys. 42 , 103–109 (2015).

Haq, A. U. et al. Detection of breast cancer through clinical data using supervised and unsupervised feature selection techniques. IEEE Access 9 , 22090–22105 (2021).

Khan, M. A., Javed, M. Y., Sharif, M., Saba, T. & Rehman, A. Multi-model deep neural network based features extraction and optimal selection approach for skin lesion classification. In 2019 International Conference on Computer and Information Sciences (ICCIS) , 1–7 (IEEE, 2019).

Shin, H.-C. et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging 35 , 1285–1298 (2016).

Haq, A. U. & Li, J. P. A survey of deep learning techniques based Parkinson’s disease recognition methods employing clinical data. Expert Syst. Appl. 208 , 118045 (2022).

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant No. 61370073), the National High Technology Research and Development Program of China, the project of Science and Technology Department of Sichuan Province (Grant No. 2021YFG0322).

Author information

Authors and affiliations.

School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China

Amin ul Haq, Jian Ping Li & CobbinahBernard Mawuli

College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University, Riyadh, 11432, Saudi Arabia

Shakir Khan, Mohammed Ali Alshara & Reemiah Muneer Alotaibi

You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization, A.U.H., and J.P.L.; methodology, A.U.H. and J.P.L. C.M.; software, A.U.H.; validation, A.U.H., and J.P.L.; C.M. formal analysis, A.U.H., and J.P.L.; investigation, A.U.H., and S.K.; resources, A.U.H., and J.P.L., S.K.; data curation, A.U.H., Mohammed Ali Alshara, C.M. and R.M.A.; writing-original draft preparation, A.U.H.; writing-review and editing, A.U.H., M.A.A., and S.K.; visualization, A.U.H.; supervision, J.P.L.; project administration, A.U.H.; funding acquisition, J.P.L. All authors have read and agreed to the published version of the manuscript.

Corresponding authors

Correspondence to Amin ul Haq or Jian Ping Li .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Haq, A.u., Li, J.P., Khan, S. et al. DACBT: deep learning approach for classification of brain tumors using MRI data in IoT healthcare environment. Sci Rep 12 , 15331 (2022). https://doi.org/10.1038/s41598-022-19465-1

Download citation

Received : 04 November 2021

Accepted : 30 August 2022

Published : 12 September 2022

DOI : https://doi.org/10.1038/s41598-022-19465-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Advanced ai-driven approach for enhanced brain tumor detection from mri images utilizing efficientnetb2 with equalization and homomorphic filtering.

A. M. J. Zubair Rahman
Muskan Gupta
Suresh Guluwadi

BMC Medical Informatics and Decision Making (2024)

DDFC: deep learning approach for deep feature extraction and classification of brain tumors using magnetic resonance imaging in E-healthcare system

Abdus Saboor
Jian Ping Li
Saad Abdullah Alajlan

Scientific Reports (2024)

Revolutionizing heart disease prediction with quantum-enhanced machine learning

S. Venkatesh Babu
Jeffin Gracewell

An integrative machine learning framework for classifying SEER breast cancer

P. Manikandan
C. Ponnuraja

Scientific Reports (2023)

Segmentation and classification of brain tumors using fuzzy 3D highlighting and machine learning

Khalil Mowlani
Mehdi Jafari Shahbazzadeh
Maliheh Hashemipour

Journal of Cancer Research and Clinical Oncology (2023)

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

Explore articles by subject
Guide to authors
Editorial policies

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Deep learning based brain tumor segmentation: a survey

Survey and State of the Art
Open access
Published: 09 July 2022
Volume 9 , pages 1001–1026, ( 2023 )

Cite this article

You have full access to this open access article

Zhihua Liu 1 ,
Lei Tong 1 ,
Long Chen 1 ,
Zheheng Jiang 2 ,
Feixiang Zhou 1 ,
Qianni Zhang 3 ,
Xiangrong Zhang 4 ,
Yaochu Jin 5 &
Huiyu Zhou ORCID: orcid.org/0000-0003-1634-9840 1

23k Accesses

78 Citations

1 Altmetric

Explore all metrics

Brain tumor segmentation is one of the most challenging problems in medical image analysis. The goal of brain tumor segmentation is to generate accurate delineation of brain tumor regions. In recent years, deep learning methods have shown promising performance in solving various computer vision problems, such as image classification, object detection and semantic segmentation. A number of deep learning based methods have been applied to brain tumor segmentation and achieved promising results. Considering the remarkable breakthroughs made by state-of-the-art technologies, we provide this survey with a comprehensive study of recently developed deep learning based brain tumor segmentation techniques. More than 150 scientific papers are selected and discussed in this survey, extensively covering technical aspects such as network architecture design, segmentation under imbalanced conditions, and multi-modality processes. We also provide insightful discussions for future development directions.

Brain tumor detection and classification using machine learning: a comprehensive survey

UNet++: A Nested U-Net Architecture for Medical Image Segmentation

Convolutional neural networks: an overview and application in radiology

Avoid common mistakes on your manuscript.

Introduction

Medical imaging analysis has been commonly involved in basic medical research and clinical treatment, e.g. computer-aided diagnosis [ 34 ], medical robots [ 126 ] and image-based applications [ 84 ]. Medical image analysis provides useful guidance for medical professionals to understand diseases and investigate clinical challenges in order to improve health-care quality. Among various tasks in medical image analysis, brain tumor segmentation has attracted much attention in the research community, which has been continuously studied (illustrated in Fig. 1 a). In spite of tireless efforts of researchers, as a key challenge, accurate brain tumor segmentation still remains to be solved, due to various challenges such as location uncertainty, morphological uncertainty, low contrast imaging, annotation bias and data imbalance. With the promising performance made by powerful deep learning methods, a number of deep learning based methods have been applied upon brain tumor segmentation to extract feature representations automatically and achieve accurate and stable performance as illustrated in Fig. 1 b.

Growth of scientific attention on deep learning based brain tumor segmentation. a Keyword frequency map in MICCAI from 2018 to 2020. The size of the keyword is proportional to the frequency of the word. We observe that ‘brain’, ‘tumor’, ‘segmentation’, and ‘deep learning’, have drawn large research interests in the community. b Blue line represents the number of deep learning based solutions in The Multimodal Brain Tumor Segmentation Challenge (BraTS) in each year. Red line represents the Top-1 whole tumor dice score of the test set in each year. Researchers shift their interests to deep learning based segmentation methods due to the powerful feature learning ability and systematic performance due to deep learning techniques since 2012 (green dashed line). Best viewed in colors

Glioma is one of the most primary brain tumors that stems from glial cells. World Health Organization (WHO) reports that glioma can be graded into four different levels based on microscopic images and tumor behaviors [ 92 ]. Grade I and II are Low-Grade-Gliomas (LGGs) which are close to benign with slow-growing pace. Grade III and IV are High-Grade-Gliomas (HGGs) which are cancerous and aggressive. Magnetic Resonance Imaging (MRI) is one of the most common imaging methods used before and after surgery, aiming at providing fundamental information for the treatment plan.

Image segmentation plays an active role in gliomas diagnosis and treatment. For example, an accurate glioma segmentation mask may help surgery planning, postoperative observations and improve the survival rate [ 6 , 7 , 94 ]. To quantify the outcome of image segmentation, we define the task of brain tumor segmentation as follows: Given an input image from one or multiple image modality (e.g. multiple MRI sequences), the system aims to automatically segment the tumor area from the normal tissues by classifying each voxel or pixel of the input data into a pre-set tumor region category. Finally, the system returns the segmentation map of the corresponding input. Figure 2 shows one exemplar HGG case with a multi-modality MRI as input and corresponding ground truth segmentation map.

Exemplar input dataset with different MRI modalities and corresponding ground truth segmentation map. Each frame represents a unique MRI modality. The last frame on the right is the ground truth with corresponding manual segmentation annotation. Different colors represent different tumor sub-regions, i.e., gadolinium (GD) enhancing tumor (green), peritumoral edema (yellow) and necrotic and non-enhancing tumor core (NCR/ECT) (red)

Difference from previous surveys

A number of notable brain tumor segmentation surveys have been published in the last few years. We present recent relevant surveys with details and highlights in Table 1 . Among them, the closest survey papers to ours are presented by Ghaffari et al. [ 44 ], Biratu et al. [ 11 ] and Magadza et al. [ 93 ]. The authors in [ 11 , 44 , 93 ] covered a majority of solutions from BraTS2012 to BraTS2018 challenges, lacking, however, an analyses based on methodology category and highlights. Two recent surveys by Kapoor et al. [ 71 ] and Hameurlaine et al. [ 51 ] also focused on the overview of classic brain tumor segmentation methods. However, both of them lacked the technical analysis and discussion of deep learning based segmentation methods. A survey of early state-of-the-art brain tumor segmentation methods before 2013 was presented in [ 49 ], where most of the proposals before 2013 combined conventional machine learning models with hand-crafted features. Liu et al. [ 86 ] reported a survey on MRI based brain tumor segmentation in 2014. This survey does not include deep learning based methods as well. Nalepa et al. [ 98 ] analyzed the technical details and impacts of different kinds of data augmentation methods with the application to brain tumor segmentation, while ours focuses on the technical analysis of deep learning based brain tumor segmentation methods.

There is a number of representative survey papers published with similar topics in recent years. Litjens et al. [ 84 ] summarized recent medical image analysis applications with deep learning techniques. This survey gave a broad study on medical image analysis including several state-of-the-art deep learning based brain tumor segmentation methods before 2017. Bernal et al. [ 9 ] reported a review focusing on the use of deep convolutional neural networks for brain image analysis. This review only highlights the application of deep convolutional neural networks. Other important learning strategies such as segmentation under imbalance condition and learning from multi-modality were not mentioned. Akkus et al. [ 2 ] presented a survey on deep learning for brain MRI segmentation. Recently, Esteva et al. [ 40 ] presented a survey on deep learning for health-care applications. This survey summarized how deep learning and generalized methods promote health-care applications. For a broader view of object detection and semantic segmentation, a survey was recently published in [ 87 ], providing the implications on object detection and semantic segmentation.

Narrowly speaking, the word “deep learning” means using deep neural network models with stacked functional layers [ 48 ]. Neural networks are able to learn high dimensional hierarchical features and approximate any continuous functions [ 83 , 135 ]. Considering the achievements and recent advances of deep neural networks, several surveys have reported the developed deep learning techniques, such as [ 50 , 77 ].

Scope of this survey

Challenges in segmentation of brain glioma tumors. a Shows glioma tumor exemplars with various sizes and locations inside the brain. b , c Show the statistical information of the training set in the multimodal brain tumor segmentation challenge 2017 (BraTS2017). The left hand side of b shows the FLAIR and T2 intensity projection, and the right hand side shows the T1ce and T1 intensity projection. c is the pie chart of the training data with labels, where the top figure shows the HGG labels while the bottom figure shows the LGG labels. We here experience region and label imbalance problems. Best viewed in colors

In this survey, we have collected and summarized the research studies reported on over one hundred scientific papers. We have examined major journals in the scientific community such as Medical Image Analysis and IEEE Transactions on Medical Imaging. We also evaluated proceedings of major conferences, such as ISBI, MICCAI, IPMI, MIDL, CVPR, ECCV and ICCV, to retain frontier medical imaging research outcomes. We reviewed annual challenges and their related competition entries such as The Multimodal Brain Tumor Segmentation Challenge (BraTS). In addition, some pre-printed versions of the established methods are also included as a source of information.

The goal of this survey is to present a comprehensive technical review of deep learning based brain tumor segmentation methods, according to architectural categories and strategy comparisons. We wish to explore how different architectures affect the segmentation performance of deep neural networks and how different learning strategies can be further improved for various challenges in brain tumor segmentation. We cover diverse high level inspects, including effective architecture design, imbalance segmentation and multi-modality process. The taxonomy of this survey is made (Fig. 5 ) such that our categorization can help the reader to understand the technical similarities and differences between segmentation methods. The proposed taxonomy may also enable the reader to identify open challenges and future research directions.

We first present the background information of deep learning based brain tumor segmentation in “Background” section and the rest of this survey is organized as follows: In “Designing effective segmentation networks” section, we review the design paradigm of effective segmentation modules and network architectures. In “Segmentation under imbalanced condition” section, we categorize, explore and compare the solutions for tackling the data imbalance issue, which is a long-standing problem in brain tumor segmentation. As multi-modality provides promising solutions towards accurate brain tumor segmentation, we finally review the methods of utilizing multi-modality information in “Utilizing multi modality information” section. We conclude this paper in “Conclusion” section. We also build up a regularly maintained project page to accommodate the updates related to this survey. Footnote 1

The evolution of brain tumor segmentation with selective milestones over the past decade. Best viewed in colors

Research challenges

Despite significant progress that has been made in brain tumor segmentation, state-of-the-art deep learning methods still experience difficulties with several challenges to be solved. The challenges associated with brain tumor segmentation can be categorized as follows:

Location uncertainty Glioma is mutated from gluey cells which surround nerve cells. Due to the wide spatial distribution of gluey cells, either High-Grade Glioma (HGG) or Low-Grade Glioma (LGG) may appear at any location inside the brain.

Morphological uncertainty Different from a rigid object, the morphology, e.g. shape and size, of different brain tumors varies with large uncertainty. As the external layer of a brain tumor, edema tissues show different fluid structures, which barely provide any prior information for describing the tumor’s shapes. The sub-regions of a tumor may also vary in shape and size.

Low contrast High resolution and high contrast images are expected to contain diverse image information [ 88 ]. Due to the image projection and tomography process, MRI images may be of low quality and low contrast. The boundary between biological tissues tends to be blurred and hard to detect. Cells near the boundary are hard to be classified, which makes precise segmentation more difficult and harder to achieve.

Annotation bias Manual annotation highly depends on individual experience, which can introduce an annotation bias during data labeling. As shown in Fig. 3 (a), it seems that some annotations tend to connect all the small regions together while the other annotations can label individual voxels precisely. The annotation biases have a huge impact on the segmentation algorithm during the learning process [ 28 ].

Imbalanced issue As shown in Fig. 3 b, c, there exists an imbalanced number of voxels in different tumor regions. For example, the necrotic/non-enhancing tumor core (NCR/ECT) region is much smaller than the other two regions. The imbalanced issue affects the data-driven learning algorithm as the extracted features may be highly influenced by large tumor regions [ 15 ].

Progress in the past decades

Representative research milestones of brain tumor segmentation are shown in Fig. 4 . In the late 90s’, researchers like Zhu et al. [ 158 ] started to use a Hopfield Neural Network with active contours to extract the tumor boundary and dilate the tumor region. However, training a neural network was highly constrained due to the computational resource limitation and technical supporting. From late 90s’ to early 20s’, most of the brain tumor segmentation methods focused on traditional machine learning algorithms with hand-crafted features, such as expert systems with multi-spectral histogram [ 32 ], segmentation with templates [ 72 , 108 ], graphical models with intensity histograms [ 33 , 133 ], tumor boundary detection from latent atlas [ 95 ]. These early works pioneered the use of machine learning in solving brain tumor segmentation problems. However, early attempts have significant shortcomings. First, most of the early works only focused on the segmentation of the whole tumor region, that is, the segmentation result has only one category. Compared with recent brain tumor segmentation algorithms, early works are formulated with strong conditions, relying on unrealistic assumptions. Second, manually designed feature engineering is constrained by prior knowledge, which cannot be fully generalized. Last but not least, early research works fail to address some challenges such as appearance uncertainty and data imbalance.

Our proposed taxonomy of deep learning based brain tumor segmentation methods. Best viewed in colors

With the revolutionary breakthrough by deep learning technology [ 74 ], researchers began to focus on using deep neural networks to solve various practical problems. Pioneering works from Zikic et al. [ 160 ], Havaei et al. [ 52 ] and Pereira et al. [ 106 ] intend to design customized deep convolutional neural network (DCNN) to achieve accurate brain tumor segmentation. With breakthrough brought by Fully Convolutional Network (FCN) [ 90 ] and U-Net [ 111 ], later innovations [ 59 , 147 ] on brain tumor segmentation focus on building fully convolutional encoder-decoder networks without fully connected layers to achieve end-to-end tumor segmentation.

A long-standing challenge in brain tumor segmentation is data imbalance. To effectively deal with the imbalance problem, researchers try different solutions, such as network cascade and ensemble [ 64 , 67 , 130 ], multi-task learning [ 97 , 150 ], and customized loss functions [ 120 ]. Another solution is to fully utilize information from multi-modality. Recent research focused on modality fusion [ 142 ] and dealing with modality missing [ 152 ].

Based on the evolution, we generally categorize the existing deep learning based brain tumor segmentation methods into three categories, i.e., methods with effective architectures, methods for dealing with imbalanced condition and methods of utilizing multi-modality information. Figure 5 shows a taxonomy of the research work in deep learning based brain tumor segmentation.

Contributions of this survey

A large number of deep learning based brain tumor segmentation methods have been published with promising results. Our paper, as a platform, provides a comprehensive and critical survey of state-of-the-art brain tumor segmentation methods. We anticipate that this survey supplies useful guidelines and coherent technical insights to academia and industry. The major contributions of this survey can be summarized as follows:

We present a comprehensive review to categorize and outline deep learning based brain tumor segmentation methods with a structured taxonomy of various important technical innovation perspectives.

We present the reader with a summarization of research progress of deep learning base brain tumor segmentation with detailed background information and system comparisons (e.g. Tables 1 , 5 ).

We carefully and extensively compares existing methods based on results from public accessible challenges and datasets (e.g. Tables 2 , 3 , 4 ), with critical summaries and insightful discussions.

Designing effective segmentation networks

Compared with complex feature engineering pipelines to extract useful features, recent deep learning mainly relies on designing effective deep neural networks to automatically extract high-dimensional discriminative features. Designing effective modules and network architectures has become one of the most important factors for achieving accurate segmentation performance. In this section, we reviewed two important design guidelines for deep learning based brain tumor segmentation: designing effective modules and designing network architecture.

There are mainly two principles to follow when designing effective modules. One is to learn high level semantics and localize precious targets, through the enlargement of the receptive field [ 81 , 91 , 144 ], attention mechanism [ 60 , 131 , 150 ] feature fusion update [ 85 , 154 ] and other forms. The other way is to reduce the amount of the network parameters and speed up during training and inference, thereby saving computational time and resources [ 4 , 14 , 22 , 29 , 105 , 117 , 117 ].

The design of the network architecture is mainly reflected in the transition from a single-channel network to a multi-channel network, from a network with fully connected layers to a fully convolutional network, from a simple network to a deep cascaded network. The purpose is to deepen the network, enhance the feature learning ability of the network and completes more precise segmentation. In the following, we divide theses methods and review them comprehensively. A systematical comparison between various network architectures and modules is shown in Fig. 6 .

Structural comparison between representative methods based on designing effective network modules and architectures. From top-left to bottom-right: (a1) CNN in [ 160 ], (b1) CNN with (b2) residual convolution module [ 19 ], (c1) CNN with (c2) full resolution residual unit [ 66 ], (d1) CNN with (d2) dense connection module [ 114 ], (e1) CNN with (e2) residual dilation block [ 91 ], (f1) CNN with (f2) atrous convolution feature pyramid module [ 155 ], (g1) FCN with (g2) multi-fiber unit [ 22 ], (h1) FCN with (h2) reversible block [ 14 ] and (i1) FCN with (i2) modality fusion module [ 153 ]. Best viewed in colors

Designing specialized modules

Modules for higher accuracy.

Numerous methods for brain tumor segmentation focus on designing effective modules inside neural networks, aiming to stabilize training, learning informative, discriminative features for accurate segmentation. Early design attempts followed the pattern of well-known networks such as AlexNet [ 74 ] and gradually increased the network depth by stacking convolutional blocks. Early research works such as [ 39 , 110 , 147 ] stacked several blocks with convolutional layers composed of a large kernel size (typically greater than 5), pooling layers and activation layers together. Blocks with a large size convolution kernel enable us to capture details with a large number of parameters to be trained. Other research works such as [ 106 , 160 ] followed the pattern of VGG [ 119 ] to build convolutional layers with a small sized kernel (typically 3) as basic block. Further research work such as [ 52 ] stacked hybrid blocks with a combination of different kernel sizes, where large sized kernels tend to find global features (such as tumor location and size) with a large receptive field and small kernels tend to contain local features (such as boundary and texture) with a small receptive field. As stacking two $3\times 3$ convolutional layers leads to equal sized reception fields while maintaining less parameters, compared with a single $5\times 5$ layer, most recent tumor segmentation works constructed basic network blocks, based on stacking $3\times 3$ layers, and started to extend to volumetric reconstruction in MRI with $3\times 3\times 3$ kernels [ 18 , 62 ].

As the number of stacked layers increases, the network is getting deeper, causing the issue of gradient explosion and vanishing during the training process. In order to stabilize system training and reach higher segmentation accuracy, early brain tumor segmentation methods such as [ 21 ] and [ 19 ] followed ResNet [ 53 ] and introduced residual connection into module design. Residual connection helps solving the problem of gradient vanishing and explosion, by adding the input of a convolution module to its output, which avoids degradation and converges faster with better accuracy. Now, residual connection has become one of the standard operations for designing modules and complex network architectures. In the following works [ 45 , 114 , 132 , 156 ], the authors followed DenseNet [ 57 ] and expanded residual connection to dense connection. Although dense connection design looks more conducive to gradient back-propagation, the complex close connection structure can cause multiple usage of the computing memory during the network training.

By stacking convolution modules and using residual connections inside and outside modules, neural networks can be deeper and features can be learnt with higher dimensions. However, this process may lead to the sacrifice of spatial resolution. The resolution of high dimensional feature maps is much smaller than that of the original data. In order to preserve the spatial resolution of feature whilst still expanding the receptive field, [ 81 , 91 , 144 ] replaced the standard convolution layer with the dilated convolution layer [ 139 ]. The dilated convolution comes up with several benefits. First, dilation convolution enlarges the receptive field without introducing additional parameters. Larger receptive fields are helpful for segmenting large-area targets, such as edema. Second, dilated convolution avoids the loss of spatial resolution. Thus, the position of the object to be segmented can be accurately localized in the original input space. However, the problem of incorrect localization and segmentation of small structures remains to be solved. In response, [ 30 ] proposed to design a multi-scale dilation convolution or atrous spatial pyramid pooling module, capturing the semantic context that describes subtle details of the object.

Modules for efficient computation

Designing and stacking complex modules help effectively learn high-dimensional discriminative features and achieve precise segmentation, but it requires high computational resources and long training and inference time. In response to this request, many works have adopted lightweight ideas in module design. With similar accuracy, fewer computing resources are required by lightweight modules, training and inference time is shorter, and the speed is faster. [ 3 ] is one of the earliest research works aiming at speeding up brain tumor segmentation. The authors of [ 3 ] reordered the input data (a data sample rotated by 6-degrees) so that the samples with high visual similarity are placed closer in the memory, in an attempt to speed up I/O communication. Instead of managing the input data, [ 29 ] chose to build a U-Net variant with decreased down-sampling channels to reduce the computational cost.

The above-mentioned works used less computational resources, but lose learning information and decreased segmentation accuracy. Inspired by reversible residual network [ 14 , 47 ] introduced reversible blocks into U-Net where each layer’s activation can be collected from the previous layer’s output during the backward pass process. Thus, no additional memory is used to store intermediate activation and hence reduce memory cost. [ 105 ] further extend reversible blocks by introducing Mobile Reversible Convolution Blocks (MBConvBlock) used in MobileNetV2 [ 112 ] and EfficientNet [ 125 ]. In addition to the reversible computation design, MBConvBlock replaced standard convolutions with depthwise separable convolutions. Depthwise separable convolutions first split the computation of feature maps accordingly using depthwise convolution and merge the feature maps together using $1\times 1\times 1$ pointwise convolutions, which further reduced parameters compared with the standard convolution. Later research works, including 3DESPNet [ 101 ] and DMFNet [ 22 ], further extend this idea with dilated convolutions, requiring less computational resources while preserving most spatial resolutions.

Designing effective architectures

A major factor that promotes prosperity and development of deep neural networks in various fields is to invest efforts in designing intelligent and effective network architectures. We divide most deep learning based brain tumor segmentation networks into single/multiple path networks and encoder–decoder networks according to the characteristics of network structures. Single and multiple path networks are used to extract features and classify the center pixels of the input patch. Encoder-Decoder networks are designed in an end-to-end fashion, that is, the encoder enables deep feature to be extracted from part of or the entire image, and then the decoder conducts feature-to-segmentation mapping. In the following subsections, we conduct a systematic analysis and comparison of variant architecture designs.

Multi-path architecture

A high level comparison between single-path and two-path CNN. Best viewed in colors

Here we refer network path as the flow of data processing (Fig. 7 ). Many research works, e.g. [ 106 , 128 , 160 ] use single path networks due to computational efficiency. Compared with single path networks, multi-path networks can extract different features from different pathways of different scales. The extracted features are combined (added or concatenated) together for further processing. A common interpretation is that a large scale path (path with a large size’s kernel or input etc.) allows networks to learn global features. Small scale’s paths (paths with a small size’s kernel or input etc.) allows networks to learn features known as local features. Similar to the functionality mentioned in the previous section, global features tend to provide global information such as tumor location, size and shape while local features provide descriptive details such as tumor texture and boundary.

The work of Havaei et al. [ 52 ] is one of the early multi-path network based solutions. The author reported a novel two pathway structure that learns local tumor information as well as global contexts. The local pathway uses a $7\times 7$ convolution kernel and the global pathway uses a $13\times 13$ convolution kernel. In order to utilize CNN architectures, the authors designed several variant architectures that concatenate CNN outputs. Castillo et al. [ 19 ] used a 3 pathway CNN to segment brain tumors. Different from [ 52 ] that used kernels in different scales, [ 19 ] inputs each path with different sizes’ patches e.g. patches with low ( $15\times 15$ ), medium( $17\times 17$ ) and normal ( $27\times 27$ ) resolutions. Thus, each path can learn specific features under the condition of different spatial resolutions. Inspired by [ 52 ], Akil et al. [ 1 ] extended the network structure with overlapping patch prediction methods, where the center of the target patch is associated with the neighboring overlapping patches.

Instead of building multi-path networks with different sizes’ kernels, other research works attempt to learn local-to-global information from the input directly. For example, Kamnitsas et al. [ 69 ] presented a dual pathway network which considers the input with different sizes’ patches, known as the normal resolution input of size $25 \times 25 \times 25$ and the low resolution input of size $19 \times 19 \times 19$ . Different from [ 19 ], the authors in [ 69 ] applied small convolution kernels with a size of $3 \times 3 \times 3$ on both pathways. Later research works by Zhao et al. [ 90 ] also designed a multi-scale CNN with a large scale path with the input size of $48 \times 48$ , a middle scale path with the input size of $18 \times 18$ and a small scale path with the input size of $12 \times 12$ .

Encoder–decoder architecture

A high level comparison between different fully convolutional networks (FCNs). Best viewed in colors

The input of the single and multiple path network for brain tumor segmentation is a patch or a certain area of the image, and the output is the classification outcome of the patch or the classification outcome of the central pixel of the input. It is very challenging to promote an accurate mapping from the patch level to the category label. First of all, the segmentation performance of single and multiple path network is easily affected by the size and quality of the input patch. A small sized input patch holds incomplete spatial information, while a large sized patch requires more computational resources. Secondly, the feature-to-label mapping is mostly conducted by the last fully connected layer. A simple fully connected layer cannot fully represent the feature space where complicated fully connected layers may overload the computer’s memory. Last but not least, this feature-to-label mapping is not of an end-to-end mode, which significantly increases the optimization cost. To tackle these problems, recent research works start to use fully convolutional network (FCN) [ 90 ] and U-Net [ 111 ] based encoder-decoder networks, establish an end-to-end fashion from the input image to the output segmentation map, and further improve the segmentation performance of networks.

Jesson et al. [ 62 ] extended standard FCN by using a multi-scale loss function. One limitation of FCN is that FCN does not explicitly model the contexts in the label domain. In [ 62 ], the FCN variant minimized the multi-scale loss by combining higher and lower resolution feature maps to model the contexts in both image and label domains. In [ 116 ], researchers proposed a boundary aware fully convolutional neural network, including two branches for up-sampling. The boundary detection branch aims to learn and model boundary information of the whole tumor as a binary classification problem. The region detection branch learns to detect and classify sub-region classes of the tumor. The outputs from the two branches are concatenated and fed to a block of two convolutional layers with a softmax classification layer.

One important mutant of FCN is U-Net [ 111 ]. U-Net consists of a contracting path to capture features and a symmetric expanding path that enables precise localization. One advantage of using U-Net, compared against traditional FCN, is the skip connections between the contracting and the expanding paths. The skip connections pass feature maps from the contracting path to the expanding path and concatenate the feature maps from the two paths directly. The original image data through skip connections can help the layers in the contracting path recover details. Several research works have been proposed for brain tumor segmentation based on U-Net. For example, Brosch et al. [ 13 ] used a fully convolutional network with skip connections to segment multiple sclerosis lesions. Isensee et al. [ 58 ] reported a modified U-Net for brain tumor segmentation, where the authors used a dice loss function and extensive data augmentation to successfully avoid over-fitting. In [ 35 ], the authors used zero padding to keep the identical output dimension for all the convolutional layers in both down-sampling and up-sampling paths. Chang et al. [ 21 ] reported a fully convolutional neural network with residual connections. Similar to skip connection, the residual connection allows both low- and high-level feature maps to contribute towards the final segmentation.

In order to extract information from the original volumetric data, Milletari et al. [ 96 ] introduced a modified 3D version of U-Net, called V-Net, with a customized dice coefficient loss function. Beers et al. [ 8 ] introduced 3D U-Nets based on sequential tasks, which uses the entire tumor ground truth as an auxiliary channel to detect enhancing tumors and tumor cores. In the post-processing stage, the authors employed two additional U-Nets that serve to enhance prediction for better classification outcomes. The input patches consist of seven channels: four anatomical MR and three label maps corresponding to the entire tumor, enhancing tumor, and tumor core.

In this section, we review and compare the work focused on module and network architecture design in brain tumor segmentation. Table 2 shows the results generated by methods focused on module and network architecture design in brain tumor segmentation. We drawn key information of these research works and list it below.

By designing custom modules, the accuracy and speed of the network can be improved.

By designing a customized architecture, it can help the network learn features at different scales, which is one of the most important steps to achieve accurate brain tumor segmentation.

The design of modules and networks heavily relies on human experience. In the future, we anticipate the application of network architecture search for searching effective brain tumor segmentation architectures [ 5 , 73 , 157 , 159 ].

Most of the existing network architecture designs do not combine domain knowledge about brain tumor, such as modeling degree information and physically inspired morphological information within tumor segmentation network.

The structure of cascaded convolutional networks for brain tumor segmentation, modified from the original structure reported in [ 130 ]. WNet, TNet and ENet are used for segmenting the whole tumor, tumor core and enhancing tumor core, respectively

Segmentation under imbalanced condition

One of the long standing challenges for brain tumor segmentation is the data imbalance issue. As shown in Fig. 3 c, imbalance is mainly reflected in the number of pixels in the sub-regions of the brain tumor. In addition, there is also an imbalance issue in patient samples, that is, the number of the HGG cases is much more than that of the LGG cases. At the same time, labeling biases introduced by manual experts can also be treated as a special form of data imbalance (different experts have different standards, resulting in imbalanced labeling results). Data imbalance plays a significant effect on learning algorithms especially deep networks. The main manifestation is that learning models trained with imbalanced data tend to learn more about the dominant groups, e.g. to learn the morphology of the edema area, and to learn HGG instead of LGG patients) [ 36 , 65 , 120 ].

Numerous works have presented many improvement strategies to address data imbalance. According to core components of these strategies, we divide the existing methods into three categories: multi-network driven , multi-task driven and custom loss function driven approaches.

Multi-network driven approaches

Even if complex modules and architectures have been designed to ensure the learning of high-dimensional discriminative features, a single network often suffers from the problem of data imbalance. Inspired by the methods such as multi-expert systems, people have started to construct complex network systems to effectively deal with data imbalance and achieved promising segmentation performance. Common multi-network systems can be divided into network cascade and network ensemble, according to data flows shared between multiple networks.

Network cascade

The definition of network cascade is that, in a serially connected network, the output of an upstream network is passed to the downstream network as input. This topology simulates the coarse-to-fine strategy, that is, the upstream network extracts rough information or features, and the downstream network subdivides the input and achieves a fine-grained segmentation.

The earliest work of adopting the cascade strategy was undertaken by Wang et al. [ 130 ] (Fig. 9 ). In their work, the author proposed to connect three networks in series. First, WNet segmented Whole Tumor, and output the segmentation result of Whole Tumor to TNet, and TNet traces Tumor Core. Finally, the segmentation result of TNet is handed over to ENet for the segmentation of Enhancing Tumor. This design logic is inspired by the attributes of the tumor sub-region, where it is assumed that Whole Tumor, Tumor Core, and Enhancing Tumor are included one by one. Therefore, the segmentation output of the upstream network is the Region-of-Interest (RoI) of the downstream network. The advantage of this practice is to avoid the interference caused by the unbalanced data. The introduction of astropic convolution and the manually cropped input effectively reduces the amount of network parameters. But there are two disadvantages: First, the segmentation effect of the downstream network is heavily dependent on the performance of the upstream network. Second, only the upstream segmentation result is considered as the input so that the downstream network cannot use other image areas as auxiliary information, which is not conducive to other tasks such as tumor location detection. Similarly, Hua et al. [ 56 ] also proposed a network cascade based on the physical inclusion characteristics of tumor. Unlike Wang et al. [ 56 , 130 ] replaced the cascade unit with a V-Net, which is suitable for 3D segmentation to improve performance. Fang et al. [ 43 ] trained two networks to act as upstream networks at the same time according to different characteristics highlighted by different modalities, respectively training for Flair and T1ce. The results of the two upstream networks can be passed to the downstream network for final segmentation. Jia et al. [ 63 ] replaced upstream and downstream networks with HRNet [ 123 ] to learn feature maps with higher spatial resolutions.

Combining 3D networks for cascading can bring better segmentation performance, but the combination of multiple 3D networks requires a large amount of parameters and high computational resources. In response to this, Li [ 79 ] proposed a cascading model that mixes 2D and 3D networks. 2D networks learn from multi views slices of a volume to obtain the segmentation mask of the whole tumor. Then, the whole tumor mask and the original 3D volume are fed to the downstream 3D U-Net. The downstream network pairs tumor core and enhancing tumor for fine segmentation. Li et al. [ 80 ] also adopted a similar method by connecting multiple U-Nets in series for coarse-to-fine segmentation. The segmentation results at each stage is associated with different loss functions. Vu et al. [ 129 ] further introduced dense connection between the upstream and downstream networks to enhance feature expression. The two-stage cascaded U-Net designed by Jiang et al. [ 64 ] has been further enhanced at the output end. In addition to the single network architecture, they also tried two different segmentation modules (interpolation and deconvolution) at the output end.

In addition to cascaded coarse-to-fine segmentation, there are other attempts to introduce other auxiliary functions. Liu designed a novel strategy in [ 89 ] to pass the segmentation result of the upstream network to the downstream network. The downstream network reconstructs the original input image according to the segmentation result of the upstream network. The loss of the recovery network is also back-propagated to the upstream segmentation network, in order to help the upstream network to outline the tumor area. Cirillo et al. [ 31 ] introduced adversarial training to tumor segmentation. The generator network constitutes the upstream network, and the discriminator network is used as the downstream network to determine whether a segmentation map is from ground truth or not. Chen et al. [ 23 ] introduced left and right symmetry characteristics of the brain to the system. The added left and right similarity masks at the connection of the upstream and downstream networks can improve the robustness of network segmentation.

Network ensemble

One main drawback of using a single deep neural network is that its performance is heavily influenced by the hyper-parameter choices. This refers to a limited generalization capability of the deep neural network. Cascaded network intends to aggregate multiple networks’ output in a coarse-to-fine strategy, however downstream networks’ performance heavily relies on the upstream network, which still limits the capability of a cascaded system. In order to achieve a more robust and more generalized tumor segmentation, the segmentation output from multiple networks can be aggregated together with a high variance, known as network ensemble. Network ensemble enlarges the hypothesis space of the parameters to be trained by aggregating multiple networks and avoids falling into local optimum caused by data imbalance.

Early research works presented in multi-path network (“Multi-path architecture” section) such as Castillo et al. [ 19 ], Kamnitsas et al. [ 68 , 69 ] can be regarded as a simplified form of network ensemble, where each path can be treated as a sub-network. The features extracted by the sub-network are then ensembled and processed for the final segmentation. In this section, we pay more attention to explicit ensemble of segmentation results from multiple sub-networks, rather than implicit ensemble of the features extracted by sub-paths.

Ensembles of multiple models and architectures (EMMA) [ 67 ] is one of the earliest well-structured works using ensemble deep neural networks for brain tumor segmentation. EMMA ensembles segmentation results from DeepMedic [ 68 ], FCN [ 90 ] and U-Net [ 111 ] and associated the final segmentation with the highest confidence score. Kao et al. [ 70 ] ensembles 26 neural networks for tumor segmentation and survival prediction. [ 70 ] introduced brain parcellation atlas to produce a location prior information for tumor segmentation. Lachinov et al. [ 75 ] ensembles two variant U-Net [ 59 , 97 ] and a cascaded U-Net [ 76 ]. The final ensemble result out-performs each single network 1–2%.

The structure of multi-task networks for brain tumor segmentation. Image courtesy of [ 97 ]. The shared encoder learns generalized feature representation and the reconstruction decoder performs multi-task as regularization

Instead of feeding sub-networks with the same input, Zhao et al. [ 146 ] averaged ensembles 3 2D-FCNs where each FCN takes different view slices as input. Similarly, Sundaresan et al. [ 124 ] averaged ensembles 4 2D-FCNs, where each FCN is designed for segmenting a specific tumor region. Chen et al. [ 24 ] used a DeconvNet [ 100 ] to generate a primary segmentation probability map and another multi-scale convolutional label evaluation net is used to evaluate previously generated segmentation maps. False positives can be reduced using both the probability map and the original input image. Hu et al. [ 55 ] ensembles a 3D cascaded U-Net with a multi-modality fusion structure. The proposed two-level U-Net in [ 55 ] aims to outline the boundary of tumors and the patch-based deep network associates tumor voxels with predicted labels.

Ensemble can be regarded as a boosting strategy for improving final segmentation results by aggregating results from multiple homogeneous networks. The winner of the BraTS2018 [ 97 ] ensembles 10 models, which further boosted the performance with 1% on dice score compared with the best single network segmentation. Similar benefits brought by ensembling can be observed from Silva et al. [ 118 ] as well. BraTS2019 winner [ 64 ] also adopted an ensemble strategy where the final result is generated by ensembling 12 models, which slightly improves the result (around 0.6−1%) compared with the best single model’s performance.

Multi-task driven approaches

Most of the work described above only perform single-task learning, that is, design and optimize a network for precise brain tumor segmentation only. The disadvantage of single-task learning is that the training target of a single-task may ignore the potential information in some tasks. Information from related tasks may improve the performance of tumor segmentation. Therefore, in recent years, many research works have started from the perspective of multi-task learning, introducing auxiliary tasks on the basis of precise segmentation of brain tumors. The main setting of multi-task learning is a low-level feature representation that can be shared among multiple tasks. There are two advantages from the shared representation. One is to share the learnt domain-related information with each other through shallow shared representations so as to promote learning and to enhance the ability to obtain updated information. The second is mutual restraint. When multi-task learning performs gradient back-propagation, it will take into account the feedback of multiple tasks. Since different tasks may have different noise patterns, the model that learns multiple tasks at the same time will learn a more general representation, which reduces the risk of over-fitting and increases the generalization ability of the system.

Early attempts such as [ 115 , 149 ] adapt the idea of multi-task learning and split the brain tumor segmentation task into three different sub-region segmentation tasks, i.e. segmenting whole tumor, tumor core and enhancing tumor individually. In [ 149 ], the author incorporated three sub-region segmentation tasks into an end-to-end holistic network, and exploited the underlying relevance among the three sub-region segmentation tasks. In [ 115 ], the author designed three different loss functions, corresponding to the segmentation loss of whole tumor, tumor core and enhancing tumor. In addition, more recent works introduce auxiliary tasks different from image segmentation. The learnt features from other tasks will support accurate segmentation. In [ 116 ], the author additionally introduces a boundary localization task. The features extracted by the shared encoder are not only suitable for tumor segmentation, but also for tumor boundary localization. Precise boundary localization can assist in minimizing the searching space and defining precise boundaries during tumor segmentation. [ 99 ] introduced the idea of first detecting and then segmenting, that is, detecting the location of tumors, and then performing precise tumor segmentation.

Another commonly used auxiliary task is to reconstruct the input data, that is, the encoded feature representation can be restored to the original input using an auxiliary decoder. [ 97 ] is the first method to introduce reconstruction as an auxiliary task to brain tumor segmentation. [ 134 ] introduced two auxiliary tasks, reconstruction and enhancement, to further enhance the ability of feature representation. [ 61 ] introduced three auxiliary tasks, including reconstruction, edge segmentation and patch comparison. These works regard the auxiliary task as a regularization to the main brain tumor segmentation task. Most multi-task designs use shared encoder to extract features and independent decoders to process different tasks. From the perspective of parameter update, the role of auxiliary task is to further regularize shared encoder’s parameter. Different from L1 or L2 that explicitly regularize parameter numbers and values, the auxiliary task shared low-level sub-space with main task. During training, auxiliary task is helpful for the network to train in the direction that simultaneously optimize the auxiliary task and the main segmentation task, which reduces the search space of the parameters, makes the extracted features more generalized for accurate segmentation [ 17 , 41 , 113 , 143 ].

Customized loss function driven approaches

During network training, the gradient is likely dominated by the excessively large sample given an imbalanced dataset. Therefore, a number of works propose a custom loss function to regulate gradients during the training of a brain tumor segmentation model. Designing a custom loss function aims to reduce the weights of the easy-to-classify samples in the loss function, whilst increasing the weights of the hard samples, so that the model is more focused on the samples of a small proportion, reducing the impact of gradient bias generated while learning from imbalanced datasets.

The illustration of cross-modality feature learning framework. Image courtesy from [ 141 ]

Early research works tend to uses the standard loss functions, e.g. categorical cross-entropy [ 106 ], cross-entropy [ 127 ], and dice loss [ 20 ]. [ 109 ] is the first attempt to customise the loss function. In [ 109 ], the authors enhance the loss function to give more weights to the edge pixels, which significantly improve segmentation accuracy at classifying tumor boundaries. Experimental results show that the weighted loss function for edge pixels helps to improve the performance of segmentation dice by $2-4\%$ . Later on, [ 116 ] proposed a customised cross-entropy loss for boundary pixels while using an auxiliary task that includes boundary localization. In [ 89 ], the reconstruction task is adopted as regularization, so the loss function aims at improving pixel-wise reconstruction accuracy. In [ 85 ], the space loss function was designed to ensure that the learnt features can keep spatial information as much as possible. [ 99 ] further used a focal loss to deal with imbalanced issues. [ 58 ] used a multi-class dice loss, that is, the smaller the proportion of the category, the higher the error weight during back-propagation. In [ 62 ], a multi-scale loss function was added to perform in-depth supervision on the features of different scales at each stage of the encoder, helping the network to learn the features in multi-scale resolutions that are more conducive to object segmentation. In [ 43 ], from the perspective of a modal, two types of losses were designed for T1ce and Flair respectively. [ 29 ] proposed a weighted combination of the dice loss, the edge loss and the mask loss. The result shows that the combined losses can improve dice performance by about $2\%$ . [ 104 ] also proposed a combination loss set, which includes the categorical cross-entropy and the soft dice loss.

Table 3 shows the results generated by methods focused on dealing with data imbalance in brain tumor segmentation. From the above comparison, we can find several interesting observations.

From the perspective of the network, the strategy to solve the imbalance problem is mainly to combine the output of multiple networks. Commonly used combination methods include network cascade and network ensemble. But these strategies all depend on the performance of each network. The consumption of the computing resources is also increased proportionally to the number of the network candidates.

From the perspective of a task, the strategy to solve the imbalance problem is to set up auxiliary tasks for the regulating networks so that the networks can make full use of the existing data and learn more generalized features that are beneficial to the auxiliary tasks as well as the segmentation task.

From the perspective of the loss function, the strategy to solve the imbalance problem is to use a custom loss function or an auxiliary loss function. By weighting the hard samples, the networks are regulated to pay more attention to the small data.

Utilizing multi modality information

Multi-modality imaging has played a key role in medical image analysis and applications. Different modalities of MRI emphasize on different tissues. Effectively utilizing of multi-modality information is one of the key factors in MRI-based brain tumor segmentation. According to the completeness of the available modalities, we divide the multi-modality brain tumor segmentation into two scenes: leveraging information based on multiple modalities and limited information processing with missing modality.

The structure of the modality-aware feature embedding module. Image courtesy of [ 142 ]

Learning with multiple modalities

In this paper, we follow the BraTS competition standard, that is, a complete multi-modality set refers the input data modalities include but not limit to T1, T1ce, T2, and Flair. In order to effectively use multi-modality information, existing works focus on effectively learning multi-modality information. The designed learning methods can be classified into three categories based on their purposes: Learning to Rank , Learning to Pair and Learning to Fuse .

Learning to Rank Modalities In multi-modality processing, the existing data modality is sorted by relevance based on the learning task, so that the network can focus on learning the modality with high relevance. This definition can be re-named as modality-task modeling. Early work from [ 110 ] can be treated as basic learning to rank formation. In [ 110 ], the author transformed each modality to a single CNN. In [ 110 ], each CNN corresponds to a different modality and the features extracted by CNN are independent of each other. The loss returned by the final classifier is similar to the scoring of the input data and the segmentation is undertaken according to the score. A similar processing method was used in [ 142 ]. For each of the two modalities, two independent networks were used for modeling relationship matching, and the parameters of each network are affected by the influence of different supervision losses. [ 141 ] extracted features of different embedding modalities (as shown in Fig. 11 ), modeled the relationship between the modalities and the segmentation of different tumor sub-regions, so that the data of different modalities were weighted and sorted corresponding to individual tasks.

The structure of the modality correlation module. Image courtesy of [ 153 ]

Learning to Pair Modalities Learning to rank modalities refers to the sorting of the modality-task relation for a certain segmentation task. Another commonly used modeling is the modality-modality pairing, which selects the best combination from multi-modality data to achieve precise segmentation. [ 82 ] is one of the early works to model the modality-modality relationship. The authors paired every two modalities and sent all the pairing combinations to the downstream network. [ 141 ] further strengthens the modality-modality pairing relationship through the cross-modal feature transition module and the modal pairing module. In the cross-modality feature transition module, the authors converted the input and output from one modality’s data to the concatenation of a modality pair. In the cross-modality feature fusion module, the authors converted the single-modality feature learning to the single-modality-pair feature learning, which predicts the segmentation masks of each single-modality-pair.

Learning to Fuse Modalities More recent works focus on learning to fuse multi-modality. Different from the modality ranking and pairing, modality fusion is to fuse features from each modality for accurate segmentation. The early fusion method is relatively simple, usually concatenates or adds features learned from different modalities. In [ 110 ], the authors used 4 networks to extract features from each modality and concatenates the extracted modality aware features. The features after concatenation are sent to Random Forest to classify the central pixel of the input patch. In [ 43 ], features from T1ce and Flair were added and sent to the downstream network for entire tumor segmentation. Similarly, in [ 141 ], modality aware feature extraction is performed and sent to the downstream network for further learning. These two fusion methods do not introduce additional parameters and are very simple and efficient. In [ 141 ], even though the authors fused the features from more complex cross-modal feature pairing and single-modal feature pairing modules. In addition, there are other works such as [ 82 , 127 ] that used additional convolutional modules to combine and learn features from different modalities so as to accomplish modality fusion.

Although concatenation and addition are used, these two fusion methods do not change the semantics of learned features and cannot highlight or suppress features. To tackling this problem, many research works in recent years have adopted attention mechanisms to strengthen the learnt features. [ 60 , 85 , 131 , 154 ] used a spatial and channel attention based fusion module. The proposed attention mechanism highlights useful features and suppresses redundant features, resulting in accurate segmentation.

Dealing with missing modalities

The modality learning methods mentioned above work in a complete multi-modality set. For example, in BraTS, we obtain the data of four modalities: T1, T1ce, T2, and FLAIR. However, in actual application scenarios, it is very difficult to obtain complete and high-quality multi-modality datasets, refers to as missing modality scenarios. Yu et al. [ 137 ] is one of the earliest works targeting learning under missing modality. The authors in Yu et al. [ 137 ] constructed the only available modal T1 and used generative adversarial networks to generate the missed modalities. In Yu et al. [ 137 ], the authors used the existing T1 modality as input to generate Flair modality. The generated Flair data is sent as a supplement with the original T1 data to the downstream segmentation network. [ 151 , 153 ] learnt the implicit relationship between modalities and examined all possible missing scenarios. The results show that multi-modality have an important influence on accurate segmentation. In [ 138 ], the intensity correction algorithm was proposed for different scenarios of the single modality input. In this framework, the intensity query and correction of the data of multiple modalities makes it easier to distinguish the tumor and non-tumor regions in the synthetic data.

Table 4 shows the results generated by methods focused learning with multi-modality in deep learning based brain tumor segmentation. We can collect several common observations in utilizing the information from multi modalities.

For task-modality modeling, learning to rank modalities can help the network choose the most relative and conducive modality for accurate segmentation. Most of the research works model the implicit ranking while learning the modality aware features.

For modality-modality modeling, learning to pair modalities can help the network find the most suitable modality combination for segmentation. However, existing pairing works show modality pairs through exhaustive combination with large computing resources.

The fusion of multi-modality information can improve the expressive ability and generalization of features. Existing fusion methods have their own advantages and disadvantages. Addition or concatenation does not introduce additional parameters, but lacks the physical expression of features. Using a small network, an attention module can optimize feature expression, but introduce additional parameters and computational cost.

Missing modalities are one of the most common scenes in clinical imaging. Existing works focus on the perspective of generation, using existing modality data to generate missing modalities. However, the performance and quality of the generator modal heavily relies on the quality of the existing modality data.

Future trends

Deep learning based brain tumor segmentation methods have achieved satisfying performance, there are challenges remaining to be solved. In this section, we briefly discuss several open issues and also point out potential directions for possible future works.

Segmentation with less supervision

Most existing research methods belong to fully supervised methods, which rely on complete dataset with precious annotated segmentation masks. However, it is very challenging to obtain segmentation mask without annotation bias, which is time-consuming and labor-intensive. Recently, research attempts such as [ 78 ] evaluate the self-supervised representation for brain tumor segmentation. In the future, brain tumor segmentation methods expected to be powered by self, weak and semi-supervised training with fewer labels.

Neural architecture search based segmentation

As we discussed in Sect. “ Designing effective segmentation networks ”, the design of modules and networks heavily relies on human experience. In the future, we anticipate the combination between domain knowledge (e.g. tumor degree, tumor morphology) with neural architecture search algorithms for searching effective brain tumor segmentation networks.

Protect patient’s privacy

Current methods heavily mining the data information, especially for downstream tasks such as survival prediction for learning to segment brain tumor with patient statistics. In the future, privacy-preserved learning framework are expected to be explored aiming at protecting patients privacy [ 78 ].

Applying various deep learning methods to brain tumor segmentation is a challenging task. Automated brain tumor segmentation benefits several aspects due to the powerful feature learning ability of deep learning techniques. In this paper, we have investigated relevant deep learning based brain tumor segmentation methods and presented a comprehensive survey. We structurally categorized and summarized the deep learning based brain tumor segmentation methods. We have widely investigated this task and discussed several key aspects such as methods’ pros and cons, designing motivation and performance evaluation.

http://github.com/ZhihuaLiuEd/SoTA-Brain-Tumor-Segmentation .

Akil M, Saouli R, Kachouri R et al (2020) Fully automatic brain tumor segmentation with deep learning-based selective attention using overlapping patches and multi-class weighted cross-entropy. Med Image Anal 63:101692

Google Scholar

Akkus Z, Galimzianova A, Hoogi A, Rubin DL, Erickson BJ (2017) Deep learning for brain MRI segmentation: state of the art and future directions. J Digit Imag 30(4):449–459

Andermatt S, Pezold S, Cattin P (2016) Multi-dimensional gated recurrent units for the segmentation of biomedical 3d-data. In: Deep learning and data labeling for medical applications, pp. 142–151. Springer

Andermatt S, Pezold S, Cattin P (2017) Multi-dimensional gated recurrent units for brain tumor segmentation. In: International MICCAI BraTS Challenge. Pre-Conference Proceedings, pp. 15–19

Bae W, Lee S, Lee Y, Park B, Chung M, Jung KH (2019) Resource optimized neural architecture search for 3d medical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 228–236. Springer

Baid U, Ghodasara S, Mohan S, Bilello M, Calabrese E, Colak E, Farahani K, Kalpathy-Cramer J, Kitamura FC, Pati S, et al. (2021) The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314

Bakas S, Akbari H, Sotiras A, Bilello M, Rozycki M, Kirby JS, Freymann JB, Farahani K, Davatzikos C (2017) Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Sci Data 4(1):1–13

Beers A, Chang K, Brown J, Sartor E, Mammen C, Gerstner E, Rosen B, Kalpathy-Cramer J (2017) Sequential 3d u-nets for biologically-informed brain tumor segmentation. arXiv preprint arXiv:1709.02967

Bernal J, Kushibar K, Asfaw DS, Valverde S, Oliver A, Martí R, Lladó X (2018) Deep convolutional neural networks for brain image analysis on magnetic resonance imaging: a review. Artificial intelligence in medicine

Bernal J, Kushibar K, Asfaw DS, Valverde S, Oliver A, Martí R, Lladó X (2019) Deep convolutional neural networks for brain image analysis on magnetic resonance imaging: a review. Artif Intell Med 95:64–81

Biratu ES, Schwenker F, Ayano YM, Debelee TG (2021) A survey of brain tumor segmentation and classification algorithms. J Imag 7(9):179

de Brebisson A, Montana G (2015) Deep neural networks for anatomical brain segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 20–28

Brosch T, Tang LY, Yoo Y, Li DK, Traboulsee A, Tam R (2016) Deep 3d convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation. IEEE Trans Med Imag 35(5):1229–1239

Brügger R, Baumgartner CF, Konukoglu E (2019) A partially reversible u-net for memory-efficient volumetric image segmentation. In: International conference on medical image computing and computer-assisted intervention, pp. 429–437. Springer

Bulo SR, Neuhold G, Kontschieder P (2017) Loss max-pooling for semantic image segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7082–7091. IEEE

van der Burgh HK, Schmidt R, Westeneng HJ, de Reus MA, van den Berg LH, van den Heuvel MP (2017) Deep learning predictions of survival based on mri in amyotrophic lateral sclerosis. NeuroImage Clin 13:361–369

Caruana R (1997) Multitask learning. Mach Learn 28(1):41–75

MathSciNet Google Scholar

Casamitjana A, Puch S, Aduriz A, Sayrol E, Vilaplana V (2016) 3d convolutional networks for brain tumor segmentation. Proceedings of the MICCAI Challenge on Multimodal Brain Tumor Image Segmentation (BRATS) pp. 65–68

Castillo LS, Daza LA, Rivera LC, Arbeláez P (2017) Volumetric multimodality neural network for brain tumor segmentation. In: 13th international conference on medical information processing and analysis, vol. 10572, p. 105720E. International Society for Optics and Photonics

Catà M, Casamitjana Díaz A, Sanchez Muriana I, Combalia M, Vilaplana Besler V (2017) Masked v-net: an approach to brain tumor segmentation. In: 2017 International MICCAI BraTS Challenge. Pre-conference proceedings, pp. 42–49

Chang PD (2016) Fully convolutional deep residual neural networks for brain tumor segmentation. In: International workshop on Brainlesion: Glioma, multiple sclerosis, stroke and traumatic brain injuries, pp. 108–118. Springer

Chen C, Liu X, Ding M, Zheng J, Li J (2019) 3d dilated multi-fiber network for real-time brain tumor segmentation in mri. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 184–192. Springer

Chen H, Qin Z, Ding Y, Tian L, Qin Z (2020) Brain tumor segmentation with deep convolutional symmetric neural network. Neurocomputing 392:305–313

Chen L, Bentley P, Rueckert D (2017) Fully automatic acute ischemic lesion segmentation in dwi using convolutional neural networks. NeuroImage Clin 15:633–643

Chen S, Ding C, Liu M (2019) Dual-force convolutional neural networks for accurate brain tumor segmentation. Pattern Recognition 88:90–100

Chen S, Ding C, Zhou C (2017) Brain tumor segmentation with label distribution learning and multi-level feature representation. 2017 International MICCAI BraTS Challenge

Chen W, Liu B, Peng S, Sun J, Qiao X (2018) S3d-unet: separable 3d u-net for brain tumor segmentation. In: International MICCAI Brainlesion Workshop, pp. 358–368. Springer

Chen Y, Joo J (2021) Understanding and mitigating annotation bias in facial expression recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14980–14991

Cheng X, Jiang Z, Sun Q, Zhang J (2019) Memory-efficient cascade 3d u-net for brain tumor segmentation. In: International MICCAI Brainlesion Workshop, pp. 242–253. Springer

Choudhury AR, Vanguri R, Jambawalikar SR, Kumar P (2018) Segmentation of brain tumors using deeplabv3+. In: International MICCAI Brainlesion Workshop, pp. 154–167. Springer

Cirillo MD, Abramian D, Eklund A (2020) Vox2vox: 3d-gan for brain tumour segmentation. arXiv preprint arXiv:2003.13653

Clark MC, Hall LO, Goldgof DB, Velthuizen R, Murtagh FR, Silbiger MS (1998) Automatic tumor segmentation using knowledge-based techniques. IEEE Trans Med Imag 17(2):187–201

Corso JJ, Sharon E, Dube S, El-Saden S, Sinha U, Yuille A (2008) Efficient multilevel brain tumor segmentation with integrated bayesian model classification. IEEE Trans Med Imag 27(5):629–640

Doi K (2007) Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput Med Imag Grap 31(4–5):198–211

Dong H, Yang G, Liu F, Mo Y, Guo Y (2017) Automatic brain tumor detection and segmentation using u-net based fully convolutional networks. In: annual conference on medical image understanding and analysis, pp. 506–517. Springer

Dong Q, Gong S, Zhu X (2018) Imbalanced deep learning by minority class incremental rectification. IEEE Trans Pattern Anal Mach Intel 41(6):1367–1381

Dou Q, Chen H, Yu L, Shi L, Wang D, Mok VC, Heng PA (2015) Automatic cerebral microbleeds detection from mr images via independent subspace analysis based hierarchical features. In: Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual International Conference of the IEEE, pp. 7933–7936. IEEE

Dou Q, Chen H, Yu L, Zhao L, Qin J, Wang D, Mok VC, Shi L, Heng PA (2016) Automatic detection of cerebral microbleeds from mr images via 3d convolutional neural networks. IEEE Trans Med Imag 35(5):1182–1195

Dvořák P, Menze B (2015) Local structure prediction with convolutional neural networks for multimodal brain tumor segmentation. In: International MICCAI workshop on medical computer vision, pp. 59–71. Springer

Esteva A, Robicquet A, Ramsundar B, Kuleshov V, DePristo M, Chou K, Cui C, Corrado G, Thrun S, Dean J (2019) A guide to deep learning in healthcare. Nature Med 25(1):24–29

Evgeniou T, Pontil M (2004) Regularized multi–task learning. In: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 109–117

Fang F, Yao Y, Zhou T, Xie G, Lu J (2021) Self-supervised multi-modal hybrid fusion network for brain tumor segmentation. IEEE J Biomed Health Inform. https://doi.org/10.1109/JBHI.2021.3109301

Fang L, He H (2018) Three pathways u-net for brain tumor segmentation. In: Pre-conference proceedings of the 7th medical image computing and computer-assisted interventions (MICCAI) BraTS Challenge, vol. 2018, pp. 119–126

Ghaffari M, Sowmya A, Oliver R (2019) Automated brain tumor segmentation using multimodal brain scans: a survey based on models submitted to the brats 2012–2018 challenges. IEEE Trans Biomed Eng 13:156–168

Ghaffari M, Sowmya A, Oliver R (2020) Brain tumour segmentation using cascaded 3d densely-connected u-net

Ghafoorian M, Karssemeijer N, Heskes T, Bergkamp M, Wissink J, Obels J, Keizer K, de Leeuw F.E, van Ginneken B, Marchiori E et al (2017) Deep multi-scale location-aware 3d convolutional neural networks for automated detection of lacunes of presumed vascular origin. NeuroImage: Clin 14:391–399

Gomez AN, Ren M, Urtasun R, Grosse RB (2017) The reversible residual network: Backpropagation without storing activations. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 2211–2221

Goodfellow I, Bengio Y, Courville A (2016) Deep learning

Gordillo N, Montseny E, Sobrevilla P (2013) State of the art survey on mri brain tumor segmentation. Magn Reson Imag 31(8):1426–1438

Gu J, Wang Z, Kuen J, Ma L, Shahroudy A, Shuai B, Liu T, Wang X, Wang G, Cai J et al (2018) Recent advances in convolutional neural networks. Pattern Recognit 77:354–377

Hameurlaine M, Moussaoui A (2019) Survey of brain tumor segmentation techniques on magnetic resonance imaging. Nano Biomed Eng 11(2):178–191

Havaei M, Davy A, Warde-Farley D, Biard A, Courville A, Bengio Y, Pal C, Jodoin PM, Larochelle H (2017) Brain tumor segmentation with deep neural networks. Med Imag Anal 35:18–31

He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778

Henry T, Carre A, Lerousseau M, Estienne T, Robert C, Paragios N, Deutsch E (2020) Brain tumor segmentation with self-ensembled, deeply-supervised 3d u-net neural networks: a brats 2020 challenge solution. arXiv preprint arXiv:2011.01045

Hu Y, Xia Y (2017) 3d deep neural network-based brain tumor segmentation using multimodality magnetic resonance sequences. In: International MICCAI Brainlesion Workshop, pp. 423–434. Springer

Hua R, Huo Q, Gao Y, Sun Y, Shi F (2018) Multimodal brain tumor segmentation using cascaded v-nets. In: International MICCAI Brainlesion Workshop, pp. 49–60. Springer

Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708

Isensee F, Kickingereder P, Wick W, Bendszus M, Maier-Hein KH (2017) Brain tumor segmentation and radiomics survival prediction: Contribution to the brats 2017 challenge. In: International MICCAI Brainlesion Workshop, pp. 287–297. Springer

Isensee F, Kickingereder P, Wick W, Bendszus M, Maier-Hein KH (2018) No new-net. In: International MICCAI Brainlesion Workshop, pp. 234–244. Springer

Islam M, Vibashan V, Jose VJM, Wijethilake N, Utkarsh U, Ren H (2019) Brain tumor segmentation and survival prediction using 3d attention unet. In: International MICCAI Brainlesion Workshop, pp. 262–272. Springer

Iwasawa J, Hirano Y, Sugawara Y (2020) Label-efficient multi-task segmentation using contrastive learning. arXiv preprint arXiv:2009.11160

Jesson A, Arbel T (2017) Brain tumor segmentation using a 3d fcn with multi-scale loss. In: International MICCAI Brainlesion Workshop, pp. 392–402. Springer

Jia H, Cai W, Huang H, Xia Y (2020) H2nf-net for brain tumor segmentation using multimodal mr imaging: 2nd place solution to brats challenge 2020 segmentation task. In: BrainLes@ MICCAI (2)

Jiang Z, Ding C, Liu M, Tao D (2019) Two-stage cascaded u-net: 1st place solution to brats challenge 2019 segmentation task. In: International MICCAI Brainlesion Workshop, pp. 231–241. Springer

Johnson JM, Khoshgoftaar TM (2019) Survey on deep learning with class imbalance. J Big Data 6(1):1–54

Jungo A, McKinley R, Meier R, Knecht U, Vera L, Pérez-Beteta J, Molina-García D, Pérez-García VM, Wiest R, Reyes M (2017) Towards uncertainty-assisted brain tumor segmentation and survival prediction. In: International MICCAI Brainlesion Workshop, pp. 474–485. Springer

Kamnitsas K, Bai W, Ferrante E, McDonagh S, Sinclair M, Pawlowski N, Rajchl M, Lee M, Kainz B, Rueckert D, et al (2017) Ensembles of multiple models and architectures for robust brain tumour segmentation. In: International MICCAI brainlesion workshop, pp. 450–462. Springer

Kamnitsas K, Ferrante E, Parisot S, Ledig C, Nori AV, Criminisi A, Rueckert D, Glocker B (2016) Deepmedic for brain tumor segmentation. In: International workshop on Brainlesion: Glioma, multiple sclerosis, stroke and traumatic brain injuries, pp. 138–149. Springer

Kamnitsas K, Ledig C, Newcombe VF, Simpson JP, Kane AD, Menon DK, Rueckert D, Glocker B (2017) Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation. Med Imag Anal 36:61–78

Kao PY, Ngo T, Zhang A, Chen JW, Manjunath B (2018) Brain tumor segmentation and tractographic feature extraction from structural mr images for overall survival prediction. In: International MICCAI Brainlesion Workshop, pp. 128–141. Springer

Kapoor L, Thakur S (2017) A survey on brain tumor detection using image processing techniques. In: 2017 7th international conference on cloud computing, data science & engineering-confluence, pp. 582–585. IEEE

Kaus M, Warfield SK, Nabavi A, Chatzidakis E, Black PM, Jolesz FA, Kikinis R (1999) Segmentation of meningiomas and low grade gliomas in mri. In: International conference on medical image computing and computer-assisted intervention, pp. 1–10. Springer

Kim S, Kim I, Lim S, Baek W, Kim C, Cho H, Yoon B, Kim T (2019) Scalable neural architecture search for 3d medical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 220–228. Springer

Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105

Lachinov D, Shipunova E, Turlapov V (2019) Knowledge distillation for brain tumor segmentation. In: International MICCAI Brainlesion Workshop, pp. 324–332. Springer

Lachinov D, Vasiliev E, Turlapov V (2018) Glioma segmentation with cascaded unet. In: International MICCAI Brainlesion Workshop, pp. 189–198. Springer

LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436

Li, W., Milletarì, F., Xu, D., Rieke, N., Hancox, J., Zhu, W., Baust, M., Cheng, Y., Ourselin, S., Cardoso, M.J., et al.: Privacy-preserving federated brain tumour segmentation. In: International workshop on machine learning in medical imaging, pp. 133–141. Springer (2019)

Li X (2018) Fused u-net for brain tumor segmentation based on multimodal mr images. International MICCAI Brain Tumor Segmentation (BraTS) challenge pp. 290–297

Li X, Luo G, Wang K (2019) Multi-step cascaded networks for brain tumor segmentation. In: International MICCAI Brainlesion Workshop, pp. 163–173. Springer

Li X, Zhang X, Luo Z (2017) Brain tumor segmentation via 3d fully dilated convolutional networks. In: Multimodal Brain Tumor Segmentation Benchmark, Brain-lesion Workshop, MICCAI, vol. 9, p. 2017

Li Y, Shen L (2017) Deep learning based multimodal brain tumor diagnosis. In: International MICCAI Brainlesion Workshop, pp. 149–158. Springer

Lin H, Jegelka S (2018) Resnet with one-neuron hidden layers is a universal approximator. Adv Neural Inf Process Syst 31:6169–6178

Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak JA, Van Ginneken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Med Imag Anal 42:60–88

Liu C, Ding W, Li L, Zhang Z, Pei C, Huang L, Zhuang X (2020) Brain tumor segmentation network using attention-based fusion and spatial relationship constraint. arXiv preprint arXiv:2010.15647

Liu J, Li M, Wang J, Wu F, Liu T, Pan Y (2014) A survey of mri-based brain tumor segmentation methods. Tsinghua Sci Technol 19(6):578–595

Liu L, Ouyang W, Wang X, Fieguth P, Chen J, Liu X, Pietikäinen M (2020) Deep learning for generic object detection: A survey. Int J Comput Vis 128(2):261–318

MATH Google Scholar

Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC (2016) Ssd: Single shot multibox detector. In: European conference on computer vision, pp. 21–37. Springer

Liu Z, Gu D, Zhang Y, Cao X, Xue Z (2020) Automatic segmentation of non-tumor tissues in glioma mr brain images using deformable registration with partial convolutional networks. In: International MICCAI Brainlesion Workshop, pp. 41–50. Springer

Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440

Lopez MM, Ventura J (2017) Dilated convolutions for brain tumor segmentation in mri scans. In: International MICCAI Brainlesion Workshop, pp. 253–262. Springer

Louis DN, Perry A, Reifenberger G, Von Deimling A, Figarella-Branger D, Cavenee WK, Ohgaki H, Wiestler OD, Kleihues P, Ellison DW (2016) The 2016 world health organization classification of tumors of the central nervous system: a summary. Acta Neuropathol 131(6):803–820

Magadza T, Viriri S (2021) Deep learning for brain tumor segmentation: a survey of state-of-the-art. J Imag 7(2):19

Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, Burren Y, Porz N, Slotboom J, Wiest R et al (2014) The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans Med Imag 34(10):1993–2024

Menze BH, Van Leemput K, Lashkari D, Weber MA, Ayache N, Golland P (2010) A generative model for brain tumor segmentation in multi-modal images. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 151–159. Springer

Milletari F, Navab N, Ahmadi SA (2016) V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 3D Vision (3DV), 2016 Fourth International Conference on, pp. 565–571. IEEE

Myronenko A (2018) 3d mri brain tumor segmentation using autoencoder regularization. In: International MICCAI Brainlesion Workshop, pp. 311–320. Springer

Nalepa J, Marcinkiewicz M, Kawulok M (2019) Data augmentation for brain-tumor segmentation: a review. Front Comput Neurosci 13:83

Nguyen HT, Le TT, Nguyen TV, Nguyen NT (2020) Enhancing mri brain tumor segmentation with an additional classification network. arXiv preprint arXiv:2009.12111

Noh H, Hong S, Han B (2015) Learning deconvolution network for semantic segmentation. In: Proceedings of the IEEE international conference on computer vision, pp. 1520–1528

Nuechterlein N, Mehta S (2018) 3d-espnet with pyramidal refinement for volumetric brain tumor image segmentation. In: International MICCAI Brainlesion Workshop, pp. 245–253. Springer

Patenaude B, Smith SM, Kennedy DN, Jenkinson M (2011) A bayesian model of shape and appearance for subcortical brain segmentation. Neuroimage 56(3):907–922

Pawar K, Chen Z, Shah NJ, Egan G (2017) Residual encoder and convolutional decoder neural network for glioma segmentation. In: International MICCAI Brainlesion Workshop, pp. 263–273. Springer

Pawar K, Chen Z, Shah NJ, Egan GF (2019) An ensemble of 2d convolutional neural network for 3d brain tumor segmentation. In: International MICCAI Brainlesion Workshop, pp. 359–367. Springer

Pendse M, Thangarasa V, Chiley V, Holmdahl R, Hestness J, DeCoste D (2020) Memory efficient 3d u-net with reversible mobile inverted bottlenecks for brain tumor segmentation. In: International MICCAI Brainlesion Workshop, pp. 388–397. Springer

Pereira S, Pinto A, Alves V, Silva CA (2016) Brain tumor segmentation using convolutional neural networks in mri images. IEEE Trans Comput Imag 35(5):1240–1251

Pinaya WH, Gadelha A, Doyle OM, Noto C, Zugman A, Cordeiro Q, Jackowski AP, Bressan RA, Sato JR (2016) Using deep belief network modelling to characterize differences in brain morphometry in schizophrenia. Sci Rep 6:38897

Prastawa M, Bullitt E, Ho S, Gerig G (2004) A brain tumor segmentation framework based on outlier detection. Med Imag Anal 8(3):275–283

Randhawa RS, Modi A, Jain P, Warier P (2016) Improving boundary classification for brain tumor segmentation and longitudinal disease progression. In: International Workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, pp. 65–74. Springer

Rao V, Sarabi MS, Jaiswal A (2015) Brain tumor segmentation with deep learning. MICCAI Multimodal Brain Tumor Segmentation Challenge (BraTS) 59

Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention, pp. 234–241. Springer

Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018 Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510–4520 (2018)

Sener O, Koltun V (2018) Multi-task learning as multi-objective optimization. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 525–536

Shaikh M, Anand G, Acharya G, Amrutkar A, Alex V, Krishnamurthi G (2017) Brain tumor segmentation using dense fully convolutional neural network. In: International MICCAI brainlesion workshop, pp. 309–319. Springer

Shen H, Wang R, Zhang J, McKenna S (2017) Multi-task fully convolutional network for brain tumour segmentation. In: Annual Conference on Medical Image Understanding and Analysis, pp. 239–248. Springer

Shen H, Wang R, Zhang J, McKenna SJ (2017) Boundary-aware fully convolutional network for brain tumor segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 433–441. Springer

Shen H, Zhang J, Zheng W (2017) Efficient symmetry-driven fully convolutional network for multimodal brain tumor segmentation. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 3864–3868. IEEE

Silva CA, Pinto A, Pereira S, Lopes A (2020) Multi-stage deep layer aggregation for brain tumor segmentation. In: International MICCAI Brainlesion Workshop, pp. 179–188. Springer

Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

Sudre CH, Li W, Vercauteren T, Ourselin S, Cardoso MJ (2017) Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp. 240–248. Springer

Suk HI, Shen D (2016) Deep ensemble sparse regression network for alzheimer’s disease diagnosis. In: International Workshop on Machine Learning in Medical Imaging, pp. 113–121. Springer

Suk HI, Wee CY, Lee SW, Shen D (2016) State-space model with deep learning for functional dynamics estimation in resting-state fmri. NeuroImage 129:292–307

Sun K, Xiao B, Liu D, Wang J (2019) Deep high-resolution representation learning for human pose estimation. In: CVPR

Sundaresan V, Griffanti L, Jenkinson M (2020) Brain tumour segmentation using a triplanar ensemble of u-nets on mr images. In: International MICCAI Brainlesion Workshop, pp. 340–353. Springer

Tan M, Le Q (2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114. PMLR

Taylor RH, Menciassi A, Fichtinger G, Fiorini P, Dario P (2016) Medical robotics and computer-integrated surgery. In: Springer handbook of robotics, pp. 1657–1684. Springer

Tseng KL, Lin YL, Hsu W, Huang CY (2017) Joint sequence learning and cross-modality convolution for 3d biomedical segmentation. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6393–6400

Urban G, Bendszus M, Hamprecht F, Kleesiek J (2014) Multi-modal brain tumor segmentation using deep convolutional neural networks. MICCAI BraTS (brain tumor segmentation) challenge. Proceedings, winning contribution pp. 31–35

Vu MH, Nyholm T, Löfstedt T (2019) Tunet: End-to-end hierarchical brain tumor segmentation using cascaded networks. In: International MICCAI Brainlesion Workshop, pp. 174–186. Springer

Wang G, Li W, Ourselin S, Vercauteren T (2017) Automatic brain tumor segmentation using cascaded anisotropic convolutional neural networks. In: International MICCAI brainlesion workshop, pp. 178–190. Springer

Wang H, Wang G, Liu Z, Zhang S (2019) Global and local multi-scale feature fusion enhancement for brain tumor segmentation and pancreas segmentation. In: International MICCAI Brainlesion Workshop, pp. 80–88. Springer

Wang Y, Zhang Y, Hou F, Liu Y, Tian J, Zhong C, Zhang Y, He Z (2020) Modality-pairing learning for brain tumor segmentation. arXiv preprint arXiv:2010.09277

Wels M, Carneiro G, Aplas A, Huber M, Hornegger J, Comaniciu D (2008) A discriminative model-constrained graph cuts approach to fully automated pediatric brain tumor segmentation in 3-d mri. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 67–75. Springer

Weninger L, Liu Q, Merhof D (2019) Multi-task learning for brain tumor segmentation. In: International MICCAI brainlesion workshop, pp. 327–337. Springer

Yarotsky D (2017) Error bounds for approximations with deep relu networks. Neural Netw 94:103–114

Yoo Y, Tang LW, Brosch T, Li DK, Metz L, Traboulsee A, Tam R (2016) Deep learning of brain lesion patterns for predicting future disease activity in patients with early symptoms of multiple sclerosis. In: Deep Learning and Data Labeling for Medical Applications, pp. 86–94. Springer

Yu B, Zhou L, Wang L, Fripp J, Bourgeat P (2018) 3d cgan based cross-modality mr image synthesis for brain tumor segmentation. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 626–630. IEEE

Yu B, Zhou L, Wang L, Yang W, Yang M, Bourgeat P, Fripp J (2021) Sa-lut-nets: Learning sample-adaptive intensity lookup tables for brain tumor segmentation. IEEE Trans Comput Imag 40(5):1417–1427

Yu F, Koltun V, Funkhouser T (2017) Dilated residual networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 472–480

Yuan Y (2020) Automatic brain tumor segmentation with scale attention network. In: BrainLes@MICCAI

Zhang D, Huang G, Zhang Q, Han J, Han J, Wang Y, Yu Y (2020) Exploring task structure for brain tumor segmentation from multi-modality mr images. IEEE Trans Comput Imag Process 29:9032–9043

Zhang D, Huang G, Zhang Q, Han J, Han J, Yu Y (2021) Cross-modality deep feature learning for brain tumor segmentation. Pattern Recogn 110:107562

Zhang Y, Yang Q (2021) A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering

Zhao L (2017) Automatic brain tumor segmentation with 3d deconvolution network with dilated inception block. MICCAI BraTS pp. 316–320

Zhao L, Jia K (2016) Multiscale CNNs for brain tumors segmentation and diagnosis. Comput Math Methods Med 2016:8356294

Zhao X, Wu Y, Song G, Li Z, Zhang Y, Fan Y (2017) 3d brain tumor segmentation through integrating multiple 2d fcnns. In: International MICCAI Brainlesion Workshop, pp. 191–203. Springer

Zhao X, Wu Y, Song G, Li Z, Zhang Y, Fan Y (2018) A deep learning model integrating fcnns and crfs for brain tumor segmentation. Med Imag Anal 43:98–111

Zhao YX, Zhang YM, Liu CL (2019) Bag of tricks for 3d mri brain tumor segmentation. In: International MICCAI Brainlesion Workshop, pp. 210–220. Springer

Zhou C, Chen S, Ding C, Tao D (2018) Learning contextual and attentive information for brain tumor segmentation. In: International MICCAI brainlesion workshop, pp. 497–507. Springer

Zhou C, Ding C, Wang X, Lu Z, Tao D (2020) One-pass multi-task networks with cross-task guided attention for brain tumor segmentation. IEEE Trans Imag Process 29:4516–4529

Zhou T, Canu S, Vera P, Ruan S (2020) Brain tumor segmentation with missing modalities via latent multi-source correlation representation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 533–541. Springer

Zhou T, Canu S, Vera P, Ruan S (2021) Latent correlation representation learning for brain tumor segmentation with missing mri modalities. IEEE Trans Imag Process 30:4263–4274. https://doi.org/10.1109/TIP.2021.3070752

Article Google Scholar

Zhou T, Canu S, Vera P, Ruan S (2021) Latent correlation representation learning for brain tumor segmentation with missing mri modalities. IEEE Trans Imag Process 30:4263–4274

Zhou T, Ruan S, Guo Y, Canu S (2020) A multi-modality fusion network based on attention mechanism for brain tumor segmentation. In: 2020 IEEE 17th international symposium on biomedical imaging (ISBI), pp. 377–380. IEEE

Zhou Z, He Z, Jia Y (2020) Afpnet: A 3d fully convolutional neural network with atrous-convolution feature pyramid for brain tumor segmentation via mri images. Neurocomputing 402:235-244

Zhou Z, He Z, Shi M, Du J, Chen D (2020) 3d dense connectivity network with atrous convolutional feature pyramid for brain tumor segmentation in magnetic resonance imaging of human heads. Comput Biol Medicine 121:103766. https://doi.org/10.1016/j.compbiomed.2020.103766

Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J (2019) Unet++: Redesigning skip connections to exploit multiscale features in image segmentation. IEEE Trans Med Imag 39(6):1856–1867

Zhu Y, Yan Z (1997) Computerized tumor boundary detection using a hopfield neural network. IEEE Trans Med Imag 16(1):55–67

Zhu Z, Liu C, Yang D, Yuille A, Xu D (2019) V-nas: Neural architecture search for volumetric medical image segmentation. In: 2019 International Conference on 3D Vision (3DV), pp. 240–248. IEEE

Zikic D, Ioannou Y, Brown M, Criminisi A (2014) Segmentation of brain tumor tissues with convolutional neural networks. Proceedings MICCAI-BRATS 36:36–39

Download references

Acknowledgements

The authors are very grateful to the editor and reviewers for their insightful and constructive comments and suggestions, which are very helpful in improving the quality of the paper. This work was funded by the China Scholarship Council and Graduate Teaching Assistantship of University of Leicester. Yaochu Jin is supported by an Alexander von Humboldt Professorship endowed by the German Federal Ministry for Education and Research. The authors thank Prof. Guotai Wang, Prof. Dingwen Zhang and Dr. Tongxue Zhou for their detailed suggestions and discussions.

Author information

Authors and affiliations.

School of Computing and Mathematical Sciences, University of Leicester, Leicester, United Kingdom

Zhihua Liu, Lei Tong, Long Chen, Feixiang Zhou & Huiyu Zhou

School of Computing and Communications, University of Lancaster, Leicester, United Kingdom

Zheheng Jiang

School of Electronic Engineering and Computer Science, Queen Mary, University of London, London, United Kingdom

Qianni Zhang

School of Artificial Intelligence, Xidian University, Xi’an, China

Xiangrong Zhang

Faculty of Technology, Bielefeld University, Bielefeld, Germany

You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huiyu Zhou .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Liu, Z., Tong, L., Chen, L. et al. Deep learning based brain tumor segmentation: a survey. Complex Intell. Syst. 9 , 1001–1026 (2023). https://doi.org/10.1007/s40747-022-00815-5

Download citation

Received : 01 December 2021

Accepted : 17 June 2022

Published : 09 July 2022

Issue Date : February 2023

DOI : https://doi.org/10.1007/s40747-022-00815-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Brain tumor segmentation
Deep learning
Neural networks
Network design
Data imbalance
Multi-modalities
Find a journal
Publish with us
Track your research

Brain lesion detection and tumor segmentation in MRI using 3D fully convolutional networks

Downloadable content.

Jesson, Andrew
Tal Arbel (Internal/Supervisor)
This thesis presents a generalized framework for the detection of lesions and segmentation of tumors in brain magnetic resonance imaging (MRI) using fully convolutional neural networks (FCNs). The FCN framework is chosen due to its capacity to model multi-resolution context in the image domain and yield consistent semantic segmentation results. This thesis extends the FCN framework to better suit the task of brain lesion segmentation and detection by including 3D convolutions to capture the full context of MRI volumes, a curriculum on the label weights to handle class imbalance, and a multi-scale loss to promote the modelling of context in the label domain. The proposed method is evaluated on two distinct tasks: multiple sclerosis (MS) lesion detection, and brain tumor segmentation. It is shown that this method performs at a high level for both tasks even though no fundamental changes to architecture, objective function, or optimization strategy are made. For the task of MS lesion detection, the trained model achieves a true positive rate of 0.82 at a false detection rate of 0.23 on an independent test set. The method was also submitted to the 2017 MICCAI Brain Tumor Segmentation (BraTS) Challenge, where it placed in the top five out of nearly one-hundred entrants, achieving independently evaluated dices scores of 0.860 and 0.783 for segmenting tumor and tumor core on unseen test data.
Cette thèse présente une méthodologie générale pour la segmentation et la détection des lésions dans des IRMs (imagerie par résonnance magnétique) de cerveaux en utilisant des réseaux neuronaux entièrement convolutionnels (REC). Le schéma REC est choisi en raison de sa capacité à modéliser le contexte multi-résolution dans le domaine de l'image et à produire des résultats de segmentation sémantique cohérents. Cette thèse étend le schéma REC pour mieux convenir à la tâche de segmentation et de détection des lésions cérébrales en incluant des convolutions 3D pour capturer le contexte complet des volumes IRM, une décroisance programée sur les poids des étiquettes pour prendre en considération les déséquilibres de classe, et une fonction de perte multi-échelle pour promouvoir la modélisation du contexte dans le domaine de l'étiquetage. La méthode proposée est évaluée sur deux tâches distinctes : la détection des lésions de la sclérose en plaques (SEP) et la segmentation des tumeurs cérébrales. Il est démontré que cette méthode fonctionneà un niveau élevé pour les deux tâches, même si aucun changement fondamental n'est apporté à l'architecture, à la fonction objectif ou à la stratégie d'optimisation. Pour la tâche de détection de lésions SEP, le modèle proposé atteint un taux positif réel de 0,82 à un taux de détection de faux de 0,23 sur un ensemble de tests indépendants. La méthode a été soumise à la conférence MICCAI 2017 Brain Tumor Segmentation (BraTS) Challenge, où elle s'est classée parmi les cinq premiers sur prés de cent participants, obtenant des scores DICE évalués indépendamment de 0,860 et 0,783 sur les tâches de segmentation des tumeurs et du noyau tumoral sur des données de test cachées.
Electrical and Computer Engineering
McGill University
https://escholarship.mcgill.ca/concern/theses/t722hc12f
Department of Electrical and Computer Engineering
Master of Engineering
Theses & Dissertations

MRI brain tumour classification using image processing and data mining

Downloadable content.

Shen, Shan.
Strathclyde Thesis Copyright
University of Strathclyde
Doctoral (Postgraduate)
Doctor of Philosophy (PhD)
Department of Electronic and Electrical Engineering.
Bioengineering Unit.
Detecting and diagnosing brain tumour types quickly and accurately is essential to any effective treatment. The general brain tumour diagnosis procedure, biopsy, not only causes a great deal of pain to the patient but also raises operational difficulty to the clinician. In this thesis, a non-invasive brain tumour diagnosis system based on MR images is proposed. The first part is image preprocessing applied to original MR images from the hospital. Non-uniformed intensity scales of MR images are standardized relying on their statistic characteristics without requiring prior or post templates. It is followed by a non-brain region removal process using morphologic operations and a contrast enhancement between white matter and grey matter by means of histogram equalization. The second part is image segmentation applied to preprocessed MR images. A new image segmentation algorithm named IFCM is developed based on the traditional FCM algorithm. Neighbourhood attractions considered in IFCM enable this new algorithm insensitive to noise, while a neural network model is designed to determine optimized degrees of attractions. This extension can also estimate inhomogenities. Brain tissue intensities are acquired from segmentation. The final part of the system is brain tumour classification. It extracts hidden diagnosis information from brain tissue intensities using a fuzzy logic based GP algorithm. This novel method imports a fuzzy membership to implement a multi-class classification directly without converting it into several binary classification problems as with most other methods. Two fitness functions are defined to describe the features of medical data precisely. The superiority of image analysis methods in each part was demonstrated on synthetic images and real MR images. Classification rules of three types and two grades of brain tumours were discovered. The final diagnosis accuracy was very promising. The feasibility and capability of the non-invasive diagnosis system were testified comprehensively.
Doctoral thesis
10.48730/vcew-8t74
uk.bl.ethos.405500

My Shodhganga
Receive email updates
Edit Profile

Shodhganga : a reservoir of Indian theses @ INFLIBNET

Shodhganga@INFLIBNET
Anna University
Faculty of Information and Communication Engineering

Items in Shodhganga are licensed under Creative Commons Licence Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0).

“Here’s How I Knew I Had a Brain Tumor”: One Survivor’s Story After a Common Complaint Became Too Much

Glioblastoma is reported to be the most common and aggressive form of brain cancer. While other cancers can metastasize, which means they spread to other parts of the body, glioblastoma starts in the brain and stays there. According to Jose Carrillo, MD , a board-certified neurologist and neuro-oncologist at Pacific Neuroscience Institute and Associate Professor of Neurology at the Saint John’s Cancer Institute at Providence Saint John’s Health Center in Santa Monica, CA, glioblastoma can occur in any part of the brain and can be very fast-growing.

This means the survival rate for glioblastoma is low: Data suggest fewer than 7% of patients live past five years, and 1% live past 10 years. “This is one of those cancers that even cancer docs will say, ‘That’s a bad one’,” Dr. Carrillo says. “There is no known cure. These tumors invariably recur after treatment.”

How do you know if you have a brain tumor of this kind? Dr. Carillo says in the earliest stages there are usually no symptoms, but as the mass grows, it will put pressure on the brain tissue around it. The symptoms will vary depending where in the brain the tumor is locatedbut generally speaking, patients experience severe headaches, nausea, weakness, slurred speech, balance problems, and eventually seizures, he explains.

But wait: Aren’t headaches a symptom of a lot of crummy conditions, like pre-menstrual syndrome and the flu? Dr. Carillo says this on discerning when a headache symptom is more serious. “What we tell people is:

if you’ve never had headaches before and suddenly you’re getting severe headaches;
if the headaches are suddenly increasing in intensity and frequency, beyond your ‘normal’;
or if your headaches are combined with any other new, unusual symptoms,

then it’s time to call your doctor,” he says.

8 Silent Signs You Could Have a Brain Tumor

Bailey Roquemore Stutz, 29, is one of these patients who discovered she had a brain tumor after going to see a doctor for what she thought were migraines . Here’s Bailey’s story of strength and living fully.

How I knew I had a brain tumor

By Bailey Roquemore Stutz, as told to Charlotte Hilton Andersen

I should have died four years agothat is, if you believe the statistics for the very aggressive type of brain cancer I have. I don’t believe them, though. If I’ve learned one thing through my battle with glioblastoma, it’s that thinking negatively gets me nowhere but depressed. There are miracles everywhere.

Get The Healthy @ Reader’s Digest newsletter

Miracle #1: Getting my “migraine” diagnosed as a brain tumor

In 2019 I was 25 years old, trying to figure out what I wanted to do with the rest of my life. I am a certified physical therapy assistant and thought it might be fun to work as a traveling assistantI’d make more money and get to see the country. Strangely though, the job opportunity I’d found kept falling through. It would seem like everything was lined up, and then another issue would surface. I was getting frustrated, so when I started to have headaches, I just chalked them up to stress.

Yet despite trying everything from caffeine to my mom’s prescription migraine pills, the headaches just kept getting worseto the point where I would wake up in the middle of the night due to the crushing pain. As time passed, the headaches brought blurry vision and light-headedness with them. Those worried me too, but I assumed they were side effects of the migraines.

One day I couldn’t take it anymore. I asked my dad to take me to urgent care. There, the doctor agreed that I was probably just having migraines…but then a thought came to me: I should mention my uncle, who died 10 years ago from a glioblastoma multiforme brain tumor . We were told that my uncle’s type of brain tumor is not normally hereditary, and it would be incredibly rare for someone my age to get it. Fortunately, the doctor ordered an MRI just to be on the safe side.“Don’t worry,” my dad tried to assure me as we waited for the results of the imaging. “It’s just a precaution, I’m sure it will be fine.”

Ten minutes later, the doctor came in. “There’s something we need to see better,” he said, and gave me an injection of contrast dye to help the images show up more clearly.

They found three lesions. The doctor told me to go directly to the emergency room where they would admit me for more testing. Still, no one had said the words “brain cancer” yet. There are lots of things that could cause brain lesions, I was told, including toxoplasmosis, an infection you can get from catsand I had a cat!

After a week of every test imaginable to rule out any other possibilityincluding a brain biopsyI had my diagnosis . It wasn’t a disease from the cat I adored. I had a frontal lobe glioblastoma multiforme grade 4that’s medical speak for “really darn big brain tumor.”

How a Routine Eye Exam Revealed This Boy’s Brain Tumor

Miracle #2: Surviving one year

Glioblastoma multiforme is the most common type of malignant brain tumor in adults. It’s also the most aggressive type, making it the most lethal. I would soon learn that fewer than 1% of allglioblastoma multiforme patients live for more than 10 years. Due to the size of my tumor, they gave me less than a year to live. That sent a chill down my spine. My beloved uncle only made it one year after his diagnosis.

I won’t lie: The statistics scared me. I asked my father and cousin (my uncle’s son) to give me a blessinga prayer that we practice in our church to give hope and healing to people suffering from any kind of illness. I’ve always found strength and comfort in my faith, and as soon as they put their hands on my head, I felt an immense feeling of peace. I wasn’t going to die. I knew it.

My doctors sure didn’t, though. They did everything they could to “manage” my expectations, trying to keep me from becoming too hopeful. But I would need all the positivity and hope I could get as I started my treatments. At first it was chemotherapy combined with radiation, but my body didn’t tolerate the chemo. So then it was only high-dose radiationa really brutal, devastating form of radiation. Basically the whole goal was to kill all the growing cells in my body and hope that the tumor cells would die before the rest of me did. Thankfully my parents were with me every step of the way and they believed 100% in my strength and courage to beat my illness!

At the end of treatment, I told them I still knew I was going to be OK. The doctors were skeptical…but sure enough, the next MRI showed that the radiation had helped. The tumor was shrinking.

Next I tried therapy administered with an Optune, an advanced technology that uses a net of electrical wires on the head to pulse low-intensity, alternating electrical fields through the brain. It isn’t a cure but can temporarily “pause” the growth of the tumor. I responded well to that treatment, but the downside is you have to shave your entire head to use it. I spent six months totally bald.

It was worth it. By January of 2020 I’d made so much progress that I decided to stop treatment. My tumor wasn’t gone and I wasn’t “cured,” but it was small enough that it was no longer causing me any symptoms.

7 Scary Cancers You Can Help Prevent Just By Exercising

Miracle #3: My hair grew back in time for my wedding

For the next three years, my routine MRIs showed the tumor shrinking steadily. During that time, I moved to the Dallas/Fort Worth area, furthered my career and met my husband Braden through a church group.

We got married in January 2022. It was the happiest day of my life and made all the past struggles worth it to get to that moment. No matter what happens next, I will always have that moment with my husband to keep in my heart.

7 Cancers that Are Notoriously Tricky to Detect Early

Miracle #4: My undying spirit

Braden and I enjoyed a blissful, carefree first year of marriage until July of 2023, when a routine MRI showed tumor growth. I had not experienced any new symptoms, so it was not a happy surprise. It was decided I would try proton therapyanother miracle, because the place I’d happened to relocate was one of the few cities in the country where it’s available!

Proton therapy reduced my tumor, but the side effects were horrendous. I was constantly nauseated and vomiting, had brain fog, exhaustion and (my least favorite part) incontinence. I wet myself in public places, more than once.

After that, I was put on Avastin, which has already shrunk the tumor a substantial amount with far fewer side effects. I received my first course of treatment in October 2023 and am in the middle of my second course, as of the time of this publication.

Learning my brain tumor was growing again was a blow, but I refuse to let it drag me back into despair. I’m not sure what the future holdsbut I remain positive and optimistic, doing everything I can to stay as healthy and happy as I can. I combine my Western medical treatments with holistic treatments like acupuncture, diet, meditation, therapy, and working on my mind-body connection. During one session I had an incredible realization that I had always seen my body as my enemy, and that this cancer was feeding on that negativity. So I apologized to my body and even started crying as I hugged myself. Learning to see my body not as something betraying me, but as my best friend, has been life-changing.

My upbeat attitude still baffles my doctors a little, especially in the face of this latest setback. But if I could tell people one thing about finding out I had brain cancer, it would be this: No matter what you’re struggling with, keep faith that everything will work out exactly as it should. That positive attitude doesn’t take away the pain or the disease, but it does give you hope. And hope is everything.

The Avastin is helping. My parents and family are my biggest support group. Braden and I just celebrated our second wedding anniversary. Life is good.

For more wellness updates, subscribe to The Healthy by Reader’s Digest newsletter and follow The Healthy on Facebook and Instagram . Keep reading:

“Here’s How I Knew I Had Colon Cancer”: One Survivor’s Story After a Single, Subtle Symptom
“Here’s How I Knew I Had Cervical Cancer”: One Survivor’s Story of Discovering Unexpected Symptoms
Do Wireless Earbuds Harm Your Brain? A Brain Cancer Doctor Sounds Off
New Study: An Old-School Remedy Shows Impressive Promise at Preventing Serious COVID Infection

The post “Here’s How I Knew I Had a Brain Tumor”: One Survivor’s Story After a Common Complaint Became Too Much appeared first on The Healthy .

Share full article

Rachel Brougham, wearing a jacket, stands in brush on the shore of a placid lake.

Widowed Before 40 and Coping With the Financial Consequences

People who suddenly lose a spouse while young can feel unprepared for what their future looks like.

When Rachel Brougham’s husband died in a cycling accident, she said, “my future as I imagined was stolen.” Credit... Nate Ryan for The New York Times

Supported by

By Caitlin Kelly

April 27, 2024

It was April 10, 2018, and Colin Brougham hadn’t sent his usual text to his wife that he was biking home. Instead, he lay dead a few blocks away after a commuter train struck him.

“I knew he was dead before I knew he was dead,” recalled Rachel Brougham, his widow. “My son and I went to the scene, and when I was told it was him, I screamed so loud I think all of Minneapolis heard me.”

Mr. Brougham was only 39.

“My life as I knew it changed in an instant,” Ms. Brougham, now 46, said. “My future as I imagined was stolen. Grief changes your brain chemistry. It changes how you think, how you interact with others, how you work. It literally changes every single thing about your life.”

Those widowed in their 20s and 30s, few of whom may even have a will, can feel even more stunned and unprepared — who expects to die that young?

Ms. Brougham, like anyone whose spouse dies unexpectedly, suddenly faced a variety of complex financial decisions: how to handle mortgage payments, car and student loans, leases, and credit card debts. Blinded by grief, exhausted and overwhelmed, the bereaved must also plan and pay for cremation or funeral costs.

Social Security’s one-time death benefit is only $255 , while the median American funeral in 2021 cost $6,971 (with cremation) or $7,848 (with a viewing and burial), according to the National Funeral Directors Association . Social Security survivor benefits are also available for children. Ms. Brougham’s 15-year-old son, Thomas, receives $2,149 a month until he turns 18 or graduates from high school, whichever is later.

Ms. Brougham sits on a couch with a mug in her hand. She looks toward a window next to her as sunlight falls across that side of her face.

“As a certified financial planner, and someone who specializes in supporting young widows and widowers, I’ve seen firsthand the raw heartache of this unique community,” said Brian K. Seymour II, the founder and chief executive of Prosperitage Wealth in Atlanta. “Losing your partner at a young age, whether to illness or a sudden accident, throws you into a storm of grief and financial upheaval.”

Even if it feels overwhelming, Mr. Seymour recommends getting control of your financial situation immediately.

“Gather all your financial documents — bank statements, investment accounts, life insurance policies, wills — and get yourself organized,” he said. “If you feel lost, seek professional help from a fee-only fiduciary financial adviser who specializes in young widows and widowers. We understand your unique challenges and can tailor a plan that considers your income, debt, benefits and goals.”

Those who have more time to prepare — the spouse is dying of a terminal disease, for example — also face making difficult decisions amid emotional distress.

Sarah Seib, 39, whose husband, Jason Markle, died in 2022 of amyotrophic lateral sclerosis, commonly known as A.L.S. or Lou Gehrig’s disease, had a steady job with a local technology company. Mr. Markle worked for many years at Syracuse University as an undergraduate administrator, but the demands of his disease quickly turned Ms. Seib into his full-time caregiver, costing her that income even as she owed $50,000 in student debt.

As her husband’s health deteriorated, he kept working to the very end because the couple desperately needed his income and health insurance. He communicated through a Tobii Dynavox tablet, which he used by blinking. A GoFundMe campaign provided $20,000 to help with growing costs.

Mr. Markle had a 401(k) plan, but tapping into it early would have meant paying a penalty and taxes. The day he died, Ms. Seib lost access to his health insurance. Her mother, who had moved in to help Ms. Seib financially and emotionally as her husband’s health declined, still lives in Syracuse, N.Y., with her and now pays half the mortgage.

“You need help from all sides,” Ms. Seib said. “A widow’s head is not right and won’t be right for a long time.”

Francisco Rosado, a barber and D.J. who goes by Frank Rose in Orlando, Fla., lost his wife, Rebekkah Rosado, when he was 34 and she was 33. He had been her caretaker for three years as she fought a form of Hodgkin’s lymphoma, a form of blood cancer. Ms. Rosado had run a thriving wedding planning business and kept working as much as she could, but the couple sold their house to cut expenses and pay medical bills. They also received $10,000 from a GoFundMe campaign that allowed Mr. Rosado to stop working and spend time with his wife before she died.

For many people whose spouse is from another country, communicating with family abroad can add complications or welcome support — or both, as it did for Robin Truiett-Theodorson, who, in 2008, became a widow at 36 after five and a half years of marriage to Mark Theodorson, a British man.

Her father assumed her late husband’s car payments, and her family “helped me quite a bit,” she said. Her mother-in-law in Britain sent some money, and Ms. Truiett-Theodorson was grateful their home in Baltimore had no mortgage. She deferred her student debt for 18 months and consolidated her credit card debt.

Many young widows and widowers will also have to face their spouse’s debts, which can add an enormous burden if they are not discharged by creditors.

Jeanette Koncikowski was separated from her husband, Mark, when he died two years after completing chiropractic school. Both were 36, with children 5 and 9 years old. He died of a rare condition, sudden unexplained death in epilepsy, owing about $150,000 on student loans.

“In order to finance that amount, we did a mix of private and federal loans, and he was the sole signatory, later consolidated,” said Ms. Koncikowski, now 45 and living in Eden, N.Y. “At the time of his death, I was originally told by the lender that I would have to pay them back even though I did not co-sign. They said since we were married when the debt was accrued, I was responsible for the debt.”

But once she shared her separation agreement and her husband’s death certificate with the lender, the entire debt was forgiven. “It was a small saving grace in an otherwise horrific experience,” Ms. Koncikowski said.

Daniel Kopp, a certified financial planner in Sarasota, Fla., who lost a spouse when he was 31, said it mattered when the debt was taken on.

“If it was before the marriage and the couple does not live in a community property state — there are nine — then the surviving spouse would generally not be responsible for the student loans,” he said. “Community property states can make the surviving spouse be held liable for paying the private loans if they were taken on after the marriage even if the spouse did not co-sign. It’s the classic financial planning answer: It depends.”

“Student loan borrowers who die will have their federal student loans discharged by providing documentation like a death certificate,” Mr. Kopp added. “However, when it comes to private student loans, it will depend on if there was a co-signer and terms of the loan. Some private lenders will also discharge the debt, but others may attempt to get the surviving spouse to pay.”

Personal, unsecured debts like those from credit cards are generally written off by the issuing companies, Mr. Kopp said.

“I even had a widowed client that tried to pay off the $5,000 balance, and Chase sent her back the check,” he said. “Auto loans typically stay with the vehicle, so if the spouse receives the vehicle through the will, the loan would then go to the spouse.”

Everyone who has received life insurance funds after a spouse’s death knows the mixed emotions they bring.

“It was a great sense of relief — and guilt,” Ms. Brougham said. “I thought, ‘Oh, my God, my husband’s dead and now I have one million dollars.’” In fact, she received $1.575 million from both term and whole life policies, which she invested for future needs.

Mr. Rosado received $250,000 in an insurance payout, and Mr. Kopp said he had received about $300,000. This money helped free them from financial panic at the worst moment of their young lives. In addition, life insurance proceeds are not considered taxable income .

The Broughams had bought life insurance when they were 24 and 25 and Ms. Brougham was freelancing full time for a small newspaper, even though they felt the cost was unaffordable — $1,308 a year.

Being prepared, financially and emotionally, means having difficult conversations even if you feel you’re way too young to have them. The spouses of Ms. Brougham, Ms. Truiett-Theodorson, Ms. Seib and Ms. Koncikowski didn’t have a will or do advance estate planning. But Mr. Rosado’s did.

“I didn’t think death would come in my 30s,” he said. “Maybe in my 70s or 90s.”

A Guide to Making Better Financial Moves

Making sense of your finances can be complicated. the tips below can help..

The way advisers handle your retirement money is about to change: More investment professionals will be required to act in their customers’ best interest when providing advice about their retirement money.

The I.R.S. estimates that 940,000 people who didn’t file their tax returns in 2020 are due back money. The deadline for filing to get it is May 17.

Credit card debt is rising, and shopping for a card with a lower interest rate can help you save money. Here are some things to know .

Whether you’re looking to make your home more energy-efficient, install solar panels or buy an electric car, this guide can help you save money and fight climate change .

Starting this year, some of the money in 529 college savings accounts can be used for retirement if it’s not needed for education. Here is how it works .

Are you trying to improve your credit profile? You can now choose to have your on-time rent payments reported to the credit bureaus to enhance your score.

IMAGES

Steps of the proposed methodology for classification of brain tumor
Brain Tumor Detection Using Image Processing Ieee Papers
mdintraining: “ Concept map for key points from brain tumors. Dark blue
Classification Of Brain Tumor Using Deep Learning
(PDF) A survey on Brain Tumor detection using deep learning
PPT on BRAIN TUMOR detection in MRI images based on IMAGE SEGMENTATI…

VIDEO

Managing School When Your Child Has a Brain Tumor
Experts Weigh in on Treating Brain Metastases in Patients with Kidney Cancer
Matlab code for Fingerprint Recognition and Matching Using Image Processing
What is a brain tumor?
PhD defense
Brain MRI Tumor Segmentation in MATLAB

COMMENTS

Brain tumor segmentation based on deep learning and an attention
Brain tumor localization and segmentation from magnetic resonance imaging (MRI) are hard and important tasks for several applications in the field of medical analysis. As each brain imaging ...
Brain Tumor Detection and Classification from MRI images
A Thesis presented to the Faculty of California Polytechnic State University, San Luis Obispo ... Brain Tumor Detection and Classi cation from MRI images Anjaneya Teja Sarma Kalvakolanu A brain tumor is detected and classi ed by biopsy that is conducted after the brain surgery. Advancement in technology and machine learning techniques could help
(PDF) Brain Tumor Detection and Segmentation
In this thesis, an experiment has been condu cted to detect ... Brain tumors are among the most aggressive of common diseases and can lead to drastic reduction of the lifespan of those affected ...
PDF Deep Learning for Brain Tumor Classification
•Create a more generalized method for brain tumor clas-siﬁcation using deep learning •Analyze the application of tumorless brain images on brain tumor classiﬁcation •Empirically evaluate neural networks on the given datasets with per image accuracy and per patient accuracy. 2. Related Work A public brain tumor dataset was created from ...
A Deep Learning Approach for Brain Tumor Classification and
In many BTS applications, the brain tumor image segmentation is achieved by classifying pixels, thus the segmentation problem turns into a classification . The aim of the work presented in this paper is to develop and test a Deep Learning approach for brain tumor classification and segmentation using a Multiscale Convolutional Neural Network.
Detection and classification of brain tumor using hybrid deep ...
Magnetic Resonance Imaging (MRI) is a widely used non-invasive method for obtaining high-contrast grayscale brain images, primarily for tumor diagnosis. The application of Convolutional Neural ...
Brain tumor detection and classification using machine ...
Brain tumor occurs owing to uncontrolled and rapid growth of cells. If not treated at an initial phase, it may lead to death. Despite many significant efforts and promising outcomes in this domain, accurate segmentation and classification remain a challenging task. A major challenge for brain tumor detection arises from the variations in tumor location, shape, and size. The objective of this ...
Brain Tumor Detection Based on Deep Learning Approaches and Magnetic
Brain tumor photos from various categories are shown as examples in Figure 1. For each type of brain cancer (glioma, pituitary, and meningioma), Table 1 provides the number of pictures in various views such as axial, coronal and sagittal. It is important to keep in mind that medical photos, in contrast to natural images, are more complicated ...
DACBT: deep learning approach for classification of brain tumors using
The classification of brain tumors (BT) is significantly essential for the diagnosis of Brian cancer (BC) in IoT-healthcare systems. Artificial intelligence (AI) techniques based on Computer aided ...
PDF Brain Tumor Segmentation Using Deep Learning
Brain Tumor Segmentation Using Deep Learning Master's thesis in Biomedical Engineering Linus Lagergren and Carl Rosengren Department of Electrical Engineering CHALMERS UNIVERSITY OF TECHNOLOGY Gothenburg, Sweden 2020. Master's thesis 2020 BrainTumorSegmentation UsingDeepLearning
Brain Tumor Detection and Classification on MR Images by a Deep Wavelet
In classification, we applied a deep wavelet auto-encoder (DWAE) model. In this stage, the segmented MR brain image is resized by 256 × 256 × 1 dimension for faster processing. The objective of this stage is to predict the slices with tumor (abnormal MR brain images and the slices without tumor (normal MR brain images). 4.1.
(PDF) An Overview of Brain Tumor
A brain tumor is one of the most malignant tumors in humans. It accounts for. approximately 1.35% of all malignant neoplasm and 29.5% of cancer-related death. [1]. Brain and CNS tumors include ...
Deep learning based brain tumor segmentation: a survey
Brain tumor segmentation is one of the most challenging problems in medical image analysis. The goal of brain tumor segmentation is to generate accurate delineation of brain tumor regions. In recent years, deep learning methods have shown promising performance in solving various computer vision problems, such as image classification, object detection and semantic segmentation. A number of deep ...
PDF Classification and characterization of brain tumor MRI by using gray
This type of tumor which is related to nerv-ous system is called gliomas and glias cells of brain are the building-block. Cancer is a rapid and uncontrollable growth of abnormal tissues which damages the nearby health tissues of brain. Tumor is categorized into Benign [3], Malignant [4] and pre-Malignant [5].
Brain Tumor Detection using Deep Learning and Image Processing
Abstract — Brain Tumor Detection is one of the most. di fficult tasks in medical image processing. The detection task. is diffic ult to perfor m because there is a lot of diversity in the ...
Thesis
This thesis presents a generalized framework for the detection of lesions and segmentation of tumors in brain magnetic resonance imaging (MRI) using fully convolutional neural networks (FCNs). The FCN framework is chosen due to its capacity to model multi-resolution context in the image domain and yield consistent semantic segmentation results ...
PDF Brain Tumor Detection and Classification Using Neural Network
Generally, tumors of the brain or any tumor can be classified into two types of tumor. The first is called benign tumor or non cancerous tumor; whereas the second is very dangerous and cancerous that is said to be malignant tumor. The growth of these two types of tumor inside the skull forces the brain and can be very harmful for life of patient.
A Survey of Brain Tumor Segmentation and Classification Algorithms
1. Introduction. Machine learning has been applied in different sectors, the majority of the studies indicate that it was applied in agriculture [], and health sectors [2,3] for disease detection, prediction, and classifications.In health sectors the most researched areas are breast cancer segmentation and classification [4,5,6,7], brain tumor detection and segmentation [], and lung and colon ...
Thesis
The general brain tumour diagnosis procedure, biopsy, not only causes a great deal of pain to the patient but also raises operational difficulty to the clinician. In this thesis, a non-invasive brain tumour diagnosis system based on MR images is proposed. The first part is image preprocessing applied to original MR images from the hospital.
Shodhganga@INFLIBNET: Analysis of brain tumor detection and
The benign brain tumor does not affect the adjacent normal and healthy tissue but the malignant tumor can affect the neighboring tissues of brain that can lead to the death of person. An early detection of brain tumor can be required to protect the survival of patients. Usually, the brain tumor is detected using MRI scanning method.
PDF Brain Tumor Detection using Convolutional Neural Network
large amount of data present to be processed manually. Brain tumors have diversiﬁed appearance and there is a similarity between tumor and normal tissues and thus the extraction of tumor regions from images becomes complicated. In this thesis work, we developed a model to extract brain tumor from 2D Magnetic Resonance brain Images
Brain Tumor Detection using Convolutional Neural Network
Brain Tumor Detection using Convolutional Neural Network. June 2019. DOI: 10.13140/RG.2.2.15562.52163. Thesis for: Bachelor of Science in Computer Science and Engineering. Authors: Md Abdullah Al ...
PDF Brain Tumor Detection Using Deep Learning
This thesis is based on research work conducted for "Brain Tumor Detection using deep learning techniques". This work would not be possible without two people whose contributions can't be ignored. I consider it an honor to work under my guides Dr. Mehfuza Holia and Prof .Pranay Patel. This thesis is on brain tumor detection.
How I knew I had a brain tumor
This means the survival rate for glioblastoma is low: Data suggest fewer than 7% of patients live past five years, and 1% live past 10 years. "This is one of those cancers that even cancer docs ...
STAT readers respond to "residency research arms race" and more
STAT now publishes selected Letters to the Editor received in response to First Opinion essays to encourage robust, good-faith discussion about difficult issues. Submit a Letter to the Editor here ...
How to Handle Your Finances as a Young Widow or ...
Grief changes your brain chemistry. It changes how you think, how you interact with others, how you work. It literally changes every single thing about your life."

DACBT: deep learning approach for classification of brain tumors using MRI data in IoT healthcare environment

Similar content being viewed by others

Segment anything in medical images

Prediction of tumor origin in cancers of unknown primary origin with cytology-based deep learning

Virtual reality-empowered deep-learning analysis of brain cells

Materials and method

Background of convolutional neural network (CNN) architecture

Convolution neural network for brain tumors classification

Improve CNN model for brain tumors classification

Model cross validation and evaluation criteria

Proposed brain tumors classification model

Experiments

Results and analysis

Results of the proposed CNN model, on original and augmented data sets

CNN model performance evaluation with cross dataset

Results of the transfer learning models (ResNet-50, VGG-16, Inception V3, DenseNet201, Xception, and MobilleNet) on original and augmented data sets

Results of the integrated frameworks (ResNet-50-CNN, VGG-16-CNN, Inception V3-CNN, DenseNet201-CNN, Xception-CNN, and MobilleNet-CNN) on original and augmented data sets

Accuracy comparison of the proposed (ResNet-CNN) model with state of-the-art models

Space and time complexity

Data availibility

Acknowledgements

Author information

Contributions

Corresponding authors

Ethics declarations

Additional information

Rights and permissions

About this article

Share this article

This article is cited by

DDFC: deep learning approach for deep feature extraction and classification of brain tumors using magnetic resonance imaging in E-healthcare system

Revolutionizing heart disease prediction with quantum-enhanced machine learning

An integrative machine learning framework for classifying SEER breast cancer

Segmentation and classification of brain tumors using fuzzy 3D highlighting and machine learning

Quick links

Deep learning based brain tumor segmentation: a survey

Cite this article

Similar content being viewed by others

Brain tumor detection and classification using machine learning: a comprehensive survey

UNet++: A Nested U-Net Architecture for Medical Image Segmentation

Convolutional neural networks: an overview and application in radiology

Introduction

Difference from previous surveys

Scope of this survey

Research challenges

Progress in the past decades

Related problems

Contributions of this survey

Designing effective segmentation networks

Designing specialized modules

Modules for efficient computation

Designing effective architectures

Multi-path architecture

Encoder–decoder architecture

Segmentation under imbalanced condition

Multi-network driven approaches

Network cascade

Network ensemble

Multi-task driven approaches

Customized loss function driven approaches

Utilizing multi modality information

Learning with multiple modalities

Dealing with missing modalities

Future trends

Segmentation with less supervision

Neural architecture search based segmentation

Protect patient’s privacy

Acknowledgements

Author information

Corresponding author

Additional information

Rights and permissions

About this article

Share this article

Brain lesion detection and tumor segmentation in MRI using 3D fully convolutional networks

MRI brain tumour classification using image processing and data mining

Shodhganga : a reservoir of Indian theses @ INFLIBNET

“Here’s How I Knew I Had a Brain Tumor”: One Survivor’s Story After a Common Complaint Became Too Much

How I knew I had a brain tumor

Miracle #1: Getting my “migraine” diagnosed as a brain tumor