This review aims to comprehensively analyse the current state of brain tumour segmentation using deep learning techniques, with a particular focus on 3D U-Net architectures. To ensure a systematic and thorough examination of the literature, a structured search was conducted using electronic databases, including PubMed, IEEE Xplore, Science Direct, and Google Scholar.

Search Strategy: The search terms used included “brain tumour segmentation,” “deep learning,” “3D U-Net,” “MRI,” “BraTS challenge,” “glioma segmentation,” “convolutional neural networks,” and “medical image analysis.” Studies published between 2014 and October 2023 were considered to capture the most recent advancements, especially those related to the BraTS challenges [ 24 ]. Only articles published in English were included.

Inclusion Criteria: The review included peer-reviewed journal articles and conference papers that focused on brain tumour segmentation using deep learning techniques. Research involving 3D U-Net architectures or their variants was prioritised. Papers discussing the BraTS challenges and their contributions and limitations from 2014 to 2023 were also included.

Exclusion Criteria: Studies not related to brain tumour segmentation or those not utilising deep learning methods were excluded. Non-English publications were also excluded.

Data Extraction and Synthesis: Relevant information from the selected studies was extracted, including the proposed methods, datasets used, performance metrics, and key findings. Emphasis was placed on studies that provided critical insights into the advancements, challenges, and future directions of 3D U-Net architectures in brain tumour segmentation.

Image segmentation is a crucial step in medical image analysis, involving the partitioning of an image into meaningful regions to facilitate interpretation and diagnosis. In the context of brain tumour segmentation, the goal is to accurately delineate tumour tissue from healthy brain tissue in MRI scans.

While traditional segmentation methods like edge detection [ 11 , 12 ], region-based segmentation [ 25 ], thresholding [ 26 ], and clustering [ 27 ] have been applied to brain tumour segmentation, they often fall short when confronted with the inherent complexity and heterogeneity of these tumours. These methods rely on identifying specific image features to delineate tumour boundaries, but they face several challenges in the context of brain tumour analysis.

Edge-based methods [ 11 , 12 ] attempt to delineate tumour boundaries by detecting discontinuities in image intensity using operators such as Gradient, Sobel, Prewitt, and Canny. However, the boundaries of brain tumours are often ill-defined and diffuse, making it difficult for these methods to accurately capture the tumour’s true extent. Furthermore, noise and artefacts in magnetic resonance imaging (MRI) scans can lead to the detection of spurious edges, reducing the reliability of these techniques [ 12 ].

Region-based segmentation groups pixels with similar characteristics like intensity, shape, and texture. The underlying assumption of pixel homogeneity within regions can be violated in brain tumours, which often exhibit heterogeneous internal structures. This heterogeneity leads to under-segmentation or over-segmentation, as the algorithm may fail to separate distinct tumour subregions or may erroneously merge tumour regions with surrounding healthy tissue.

Thresholding-based methods [ 25 ], such as Otsu’s method [ 28 ], classify pixels as tumour or non-tumour based on predefined intensity thresholds. However, the intensity distributions of tumour and healthy tissues often overlap significantly, making it challenging to determine a single optimal threshold [ 25 ]. This overlap can result in inaccurate segmentation, especially in cases of low-contrast tumours or those with intensity profiles similar to surrounding structures.

Clustering-based segmentation [ 29 ] employs algorithms like K-means to group pixels into clusters based on feature similarity. While effective in some contexts, K-means clustering assumes spherical and equally sized clusters, which is often not the case for irregularly shaped brain tumours [ 27 ]. Moreover, the presence of noise and the variability in tumour appearance across different patients can compromise the accuracy of clustering-based approaches [ 29 ].

In summary, traditional segmentation methods often struggle to accurately delineate brain tumours due to the latter’s irregular shapes, heterogeneous intensity profiles, and indistinct boundaries. These limitations highlight the need for more sophisticated techniques, such as deep learning, which can learn complex patterns and adapt to the variability inherent in brain tumour imaging data.

Deep learning, a subfield of machine learning based on artificial neural networks with representation learning, has emerged as a powerful tool in medical image analysis [ 30 - 32 ]. Convolutional Neural Networks (CNNs), a specialised deep learning architecture, have been particularly successful in image-related tasks due to their ability to automatically learn hierarchical features directly from raw data [ 33 - 35 ]. This section will focus on CNNs and their application in the context of brain tumour segmentation, highlighting key architectural components, challenges, and evaluation metrics.

CNNs are designed to process data with a grid-like topology, making them particularly well-suited for image analysis. They learn spatial hierarchies of features automatically and adaptively, starting from low-level features like edges and textures to higher-level features representing complex patterns. Figure 2 illustrates a typical CNN architecture used for brain tumour segmentation (see Figure 2 ). The input MRI image undergoes a series of operations, starting with convolutional layers that extract spatial features using learned filters. These features are then downsampled using pooling layers, reducing dimensionality while retaining essential information. Activation functions introduce non-linearity, enabling the network to learn complex patterns. This process is repeated through multiple layers, progressively extracting higher-level features. Finally, fully connected layers integrate these features to produce a segmentation map, delineating tumour regions from healthy tissue.

1) Convolution Operations

Convolutional layers are the core building blocks of CNNs. They apply learned filters to the input image, extracting features such as edges, textures, and patterns (see Figure 3 ). Each filter slides across each channel of the input (e.g., red, green, and blue channels in an RGB image), performing element-wise multiplication and summation to produce a single value in the output feature map. The collection of feature maps generated by different filters represents the learned features at that layer.

2) Challenges in Training Deep Networks

Training deep neural networks presents several challenges, including the vanishing gradient problem, overfitting, high computational cost, and the requirement for large, labelled datasets [ 36 ]. The vanishing gradient problem is particularly relevant in deep architectures (see Figure 4 ). During backpropagation, gradients are calculated using the chain rule, and in deep networks, these gradients can become progressively smaller as they are multiplied through multiple layers. This issue is especially pronounced when using activation functions like sigmoid, whose derivatives are always less than or equal to 0.25. As a result, the gradients in the earlier layers become extremely small, hindering effective weight updates and slowing down or even halting the learning process in those layers [ 37 ]. This challenge can lead to slow convergence, requiring additional techniques such as batch normalization, residual connections, and careful weight initialization to maintain stable learning.

Figure 4 illustrates the training process of a deep neural network using backpropagation. The process begins with a feedforward calculation, where input neurons pass weighted sums through hidden layers to compute outputs. Each neuron applies an activation function to introduce non-linearity, enhancing learning capability. The output layer produces model predictions, which are compared to observed values to compute the error function. If the error is above a predefined tolerance, backpropagation updates the weights using gradient descent. Gradients are calculated using the chain rule to adjust weights and minimize the loss function iteratively. The weight update process continues for multiple epochs until convergence is achieved. The training stops when the error falls below the specified threshold, ensuring optimal model performance. However, factors like learning rate selection, regularization, and the choice of optimizer play a crucial role in stabilizing training and preventing divergence.

3) Activations Functions

Activation functions are crucial components of deep learning models, introducing non-linearity that allows networks to approximate complex, non-linear functions and learn intricate patterns from input data [ 38 - 40 ]. In the context of medical image segmentation, the choice of activation function can significantly influence model performance, convergence speed, and the ability to mitigate issues like the vanishing gradient problem [ 41 - 43 ].

$\text{HardELiSH}\left(z\right)=\{\begin{array}{ll}z\max \left(0,\min \left(1,\left(\frac{z+1}{2}\right)\right)\right)\hfill & \text{if}z\ge 1\hfill \\ \left({\text{e}}^z-1\right)\max \left(0,\min \left(1,\frac{z+1}{2}\right)\right)\hfill & \text{if}z<0\hfill \end{array}$ (1)

The piece-wise linear approximation aims to reduce computational costs without significant performance degradation.

Figure 5 illustrates several commonly used activation functions. Sigmoid maps inputs to a range between 0 and 1. ReLU outputs the input directly if positive, otherwise zero. Leaky ReLU introduces a small slope for negative inputs. Sigmoid maps inputs to a range between 0 and 1. Swish exhibits a smooth, non-monotonic curve. HardELiSH offers a computationally efficient approximation of ELiSH using piece-wise linear functions. These functions play a critical role in shaping the learning dynamics and overall performance of neural networks. The HardELiSH is a computationally efficient approximation of the ELiSH (Exponential Linear Squashing) function, employing piece-wise linear functions to retain much of the performance benefits while reducing computational overhead. HardEliSH activation function has shown superior performance in brain tumour segmentation [ 44 ]. Its ability to balance computational efficiency with strong gradient propagation makes it especially well-suited for segmentation tasks involving high-dimensional medical imaging data, such as those in 3D volumetric datasets like BraTS2020. This superior performance can be attributed to HardELiSH robust handling of non-linearities and its capacity to adapt to the complex patterns present in medical images, enabling better delineation of tumour boundaries and improved segmentation accuracy.

These functions are critical for shaping the learning dynamics and overall performance of neural networks, influencing the convergence speed, gradient flow, and ultimately the model’s accuracy.

These advanced activation functions enable deep learning models to overcome the limitations of traditional functions, such as Sigmoid or Tanh, which suffer from the vanishing gradient problem in deeper networks. In the context of brain tumour segmentation, they provide essential benefits:

Incorporating these advanced activation functions in brain tumour segmentation models can lead to more accurate, robust, and clinically viable results, making them a vital component in modern medical imaging techniques. Continued research into novel activation functions remains crucial for further improving the performance and efficiency of deep learning models in medical image analysis.

4) Fully Connected Layers

Following the convolutional and pooling layers, fully connected layers integrate the extracted features to perform classification or regression tasks (see Figure 6 ). In fully connected layers, each neuron is connected to every neuron in the previous layer, allowing the network to learn global patterns and relationships between features [ 45 ]. In the context of segmentation, these layers combine the learned features to produce the final segmentation map, assigning each pixel to a specific class (e.g., tumour or healthy tissue) [ 46 ].

The U-Net architecture, originally proposed by Ronneberger et al. [ 18 ], has become a cornerstone of medical image segmentation. It features an encoder-decoder structure with skip connections. The encoder path captures contextual information through successive convolutional and pooling layers, while the decoder path recovers spatial information through upsampling and concatenation with corresponding feature maps from the encoder path.

The 3D U-Net architecture extends the original U-Net to process volumetric data, making it particularly suitable for 3D medical imaging modalities like MRI (see Figure 7 ). The network consists of an encoder path that captures context through a series of 3D convolutional and pooling layers and a decoder path that enables precise localisation using 3D up-convolutional layers. Critically, skip connections between the encoder and decoder paths concatenate feature maps of the same resolution, allowing the network to combine high-level semantic information from the decoder path with fine-grained spatial details from the encoder path. This architecture allows for accurate segmentation by effectively capturing 3D spatial context and preserving detailed spatial information across all three dimensions.

The 3D U-Net architecture also benefits from its ability to process entire volumetric patches, reducing the need for slice-by-slice analysis and ensuring more consistent segmentations across contiguous slices. Its use of 3D convolutions enables the extraction of spatial features from the entire volume, capturing intricate patterns and relationships in medical imaging data. Additionally, the 3D U-Net is highly flexible and can be adapted to different scales and resolutions, making it suitable for tasks involving multi-resolution input data.

The ability of 3D U-Net to effectively capture and integrate hierarchical features makes it highly robust in scenarios involving irregularly shaped anatomical structures or small lesions. Moreover, the symmetrical design of the encoder and decoder paths facilitates efficient learning and optimization, while the skip connections help mitigate the risk of information loss during feature downsampling. As a result, 3D U-Net has become a preferred choice for a wide range of 3D segmentation tasks in medical imaging.

While 2D U-Net architectures process volumetric data slice-by-slice, 3D U-Net architectures operate directly on the entire volume [ 47 - 50 ]. This difference has significant implications for segmentation accuracy, computational cost, and the ability to capture spatial context. Table 1 summarises the key differences between 2D and 3D U-Net architectures.

Table 1 summarizes the key differences between 2D U-Net and 3D U-Net architectures, highlighting aspects such as input data, dimensions, layer types (convolution, pooling, up-convolution), spatial context, feature representation, computational complexity, training efficiency, and the format of output segmentation maps.

To thoroughly evaluate the performance of segmentation models, it is crucial to employ a variety of metrics that assess different aspects of accuracy and overlap with the ground truth [ 51 ]. Each metric provides unique insights into the model’s ability to accurately delineate tumour regions from healthy tissue.

1) Equations for Evaluations Metrics

$\text{Accuracy}=\frac{T_P+T_N}{T_P+F_P+F_N+T_N}$ (2)

Accuracy measures the overall correctness of the model’s predictions, representing the proportion of correctly classified pixels (both tumour and non-tumour) out of the total number of pixels However, accuracy can be misleading in cases of class imbalance, where the number of non-tumour pixels significantly exceeds the number of tumour pixels [ 52 ]. In medical imaging, where tumour regions are often small compared to the background, accuracy alone may not adequately reflect the model’s ability to detect pathological regions.

$\text{Precision}=\frac{T_P}{T_P+F_P}$ (3)

Precision quantifies the proportion of true positive predictions among all positive predictions made by the model. It reflects the model’s ability to avoid false positives and is particularly important in medical applications where false positives can lead to unnecessary treatments [ 53 ]. However, high precision alone does not guarantee that the model is effective at identifying all tumour regions, as it does not account for false negatives.

$\text{Recall}=\frac{T_p}{T_P+F_N}$ (4)

Recall, also known as sensitivity or the true positive rate, measures the proportion of actual tumour pixels that are correctly identified by the model. It reflects the model’s ability to detect all positive instances and is crucial in medical diagnosis to minimise missed cases [ 53 ]. However, high recall may come at the cost of increased false positives, which can reduce precision.

$\text{Specificity}=\frac{T_N}{F_P+T_N}$ (5)

Specificity, also known as the true negative rate, measures the proportion of actual non-tumour pixels that are correctly identified by the model. It reflects the model’s ability to correctly identify negative instances [ 52 ]. Which is important for ensuring that healthy regions are not misclassified as pathological. However, specificity alone does not provide information about the model’s ability to detect tumour regions.

Figure 8 provides a graphical representation of sensitivity and specificity. Sensitivity, also known as the true positive rate, measures the proportion of actual positives (tumour pixels) that are correctly identified by the model. Specificity, also known as the true negative rate, measures the proportion of actual negatives (non-tumour pixels) that are correctly identified. These metrics are essential for evaluating the accuracy of segmentation models in detecting the presence and absence of tumours [ 54 ].

$\text{F-score}=\frac{2\times \text{Precision}\times \text{Recall}}{\text{Precision}+\text{Recall}}$ (6)

The F1-score, also known as the Dice Similarity Coefficient (DSC) in the context of image segmentation, is the harmonic mean of precision and recall. It provides a balanced measure of the model’s performance, which is particularly useful when both false positives and false negatives are of concern [ 55 ].

The Dice score is widely used in medical image segmentation due to its ability to handle class imbalance and provide a single metric that reflects both precision and recall. However, it may not fully capture the spatial accuracy of segmentation boundaries, which can be critical in clinical applications.

$\text{Jaccard}\text{Index}=\frac{T_P}{T_P+F_P+F_N}$ (7)

The Jaccard Index, also known as the Intersection over Union (IoU), measures the overlap between the predicted segmentation and the ground truth segmentation. It is similar to the Dice score but is more sensitive to false positives [ 55 ]. It is particularly useful for evaluating segmentation performance in scenarios where precise boundary delineation is critical, such as surgical planning or radiation therapy. However, like the Dice score, it may not fully account for the clinical significance of segmentation errors.

The Hausdorff Distance (HD) measures the maximum distance between the boundaries of the predicted segmentation and the ground truth. It is particularly useful for evaluating the spatial accuracy of segmentation boundaries, which is critical in applications such as radiotherapy planning, where precise tumour localization is essential. However, HD can be sensitive to outliers and may not reflect overall segmentation quality [ 56 ].

$H\left(A,B\right)=\max \left(\underset{a\in A}{\max }\underset{b\in B}{\min }\Vert a-b\Vert ,\underset{b\in B}{\max }\underset{a\in A}{\min }\Vert a-b\Vert \right)$ (8)

Intersection over Union is a key metric in deep learning for evaluating segmentation and object detection models. It measures the overlap between the predicted and ground truth regions. IoU is calculated as:

$\text{IoU}=\frac{\text{Intersection}}{\text{Union}}=\frac{A_{pred}\cap A_{gt}}{A_{pred}\cup A_{gt}}$ (9)

where $A_{pred}$ is the predicted region and $A_{gt}$ is the ground truth. IoU values range from 0 to 1, where 1 indicates perfect overlap. Higher IoU means better model performance. In semantic segmentation, IoU is computed for each class and averaged (Mean IoU). A common threshold (e.g., 0.5) is used in object detection to determine true positives. IoU helps in model optimization by assessing prediction accuracy effectively.

Each of these metrics provides unique insights into the performance of segmentation models, but they also have limitations that must be considered in the context of medical imaging:

In clinical practice, a combination of these metrics is often used to comprehensively evaluate segmentation performance. For example, the Dice score and Hausdorff Distance are commonly used together to assess both overlap and boundary accuracy. Additionally, visual inspection by clinicians remains a critical step in validating segmentation results, as it provides context-specific insights that quantitative metrics may not capture.

Table 2 shows the evaluation metrics commonly used in segmentation tasks, including their formulas and descriptions. These metrics assess various aspects of model performance, such as overall accuracy, precision, recall (sensitivity), specificity, and overlap measures like F1-score (Dice Similarity Coefficient) and Jaccard Index. Additionally, the Hausdorff Distance evaluates the maximum boundary deviation between predictions and ground truth, providing insights into segmentation quality.

Deep learning techniques have revolutionised the field of brain tumour segmentation by enabling models to learn intricate features directly from data, leading to significant improvements in accuracy and efficiency. This section provides an overview of key contributions in this area, highlighting both general studies and notable advancements from the annual Brain Tumour Segmentation (BraTS) challenges. Figure 1 illustrates different MRI modalities and their corresponding segmented brain tumours, showcasing the ability of deep learning models to delineate complex tumour regions.

Several studies have explored diverse deep-learning approaches for brain tumour segmentation, each with its strengths and limitations:

Table 3 summarizes the contributions and limitations of various general studies on brain tumor segmentation using 3D medical imaging. The studies span a range of methodologies, including statistical integration with deep learning, U-Net architectures, preprocessing enhancements, and ensemble models. Contributions include improved segmentation accuracy, robust frameworks, and innovative preprocessing techniques. However, limitations such as computational complexity, dataset-specific challenges, overfitting, slow convergence, and modest survival prediction performance highlight the need for further advancements in these approaches.

The Multimodal Brain Tumour Segmentation (BraTS) challenge has played a crucial role in advancing the field by providing standardised datasets, evaluation metrics, and a common platform for comparing different methodologies [ 24 ]. Each year, the challenge has spurred the development of novel techniques and pushed the boundaries of segmentation accuracy. The following is a summary of notable contributions and achievements from each year of the BraTS challenge, showcasing the evolution of segmentation techniques.

Menze et al. [ 24 ] established the initial BraTS dataset and benchmark, providing a crucial resource for evaluating and comparing brain tumour segmentation models. The dataset included multi-contrast MRI scans (T1, T1ce, T2, FLAIR) with expert annotations of tumour regions (whole tumour, tumour core, enhancing tumour).

The study’s contributions include organizing the BRATS 2012 and 2013 challenges, which created a unique public dataset of MR scans for brain tumour segmentation with multiple expert annotations. The authors established the annotation protocol, acquired and annotated clinical images, generated synthetic datasets, and developed pre-processing methods and evaluation scripts. They also maintained online validation tools for community use. Notably, the study highlights that fusing different segmentation algorithms can significantly improve performance, suggesting that future advancements may stem from exploring various fusion strategies, thereby providing a valuable resource for researchers in medical imaging and brain tumour segmentation.

The study on the BRATS benchmark identifies several limitations, including the high variability in brain tumours, which complicates algorithm rankings and comparisons. Challenges arise from differing test settings across workshop years, making direct comparisons difficult. Additionally, reliance on a single set of annotations for test data may not adequately capture expert variability, potentially skewing results. The complexity of algorithms further obscures the understanding of performance discrepancies, while the choice of evaluation metrics can significantly influence rankings. Furthermore, the lack of longitudinal analysis for automated routines highlights a critical gap in assessing their long-term reliability and effectiveness. These findings underscore the need for future research and benchmarks to address these limitations and advance the field of brain tumour segmentation.

The study identifies several research gaps in brain tumour segmentation, including the high variability of brain tumours, which complicates the evaluation of segmentation algorithms, and the reliance on a single set of annotations that may not fully represent expert opinions. To address these challenges, future improvements could involve expanding datasets to better capture tumour variability, incorporating multiple expert annotations for more reliable comparisons, and conducting longitudinal studies to assess algorithm performance over time. Additionally, enhancing algorithmic transparency through detailed analysis of segmentation pipelines, exploring diverse evaluation metrics for comprehensive assessments, and investigating fusion strategies to combine multiple algorithms for improved performance are recommended. Addressing these areas can significantly enhance the effectiveness and reliability of brain tumour segmentation algorithms.

Kamnitsas et al. [ 65 ] introduced a 3D CNN architecture called DeepMedic for brain tumour segmentation, demonstrating the advantages of using volumetric data over 2D slice-based methods.

The study makes several significant contributions to the field of brain tumour segmentation by enhancing the DeepMedic architecture with residual connections. A key contribution is the demonstration of how residual connections improve the performance of a 3D CNN model, resulting in higher sensitivity and accuracy in segmenting brain tumours. The study includes a comprehensive evaluation of the model on the BRATS 2016 challenge, where it achieves competitive performance compared to other state-of-the-art methods. Furthermore, the research underscores the critical role of data augmentation and normalization techniques in enhancing model robustness, particularly when applied to heterogeneous testing scenarios. These findings provide valuable insights for advancing brain tumour segmentation methodologies.

The study acknowledges several limitations that warrant consideration. A significant limitation is the model’s reliance on high-quality training data, as its performance can degrade when there is a mismatch between the training and testing distributions. While the DeepMedic model demonstrates strong overall performance, it faces challenges in accurately segmenting fine substructures, such as necrosis and non-enhancing tumours. Additionally, the experiments were primarily conducted under conditions where the training and testing distributions were similar, leaving a gap in understanding how the model generalizes to more diverse and heterogeneous datasets. These limitations highlight areas for future research to improve the robustness and applicability of the model in real-world clinical settings.

The study identifies several research gaps and outlines promising future directions. It suggests exploring the optimal amount of external data required to enhance model generalization to new and diverse distributions, as well as investigating the relationship between network capacity and the volume of available training data. Additionally, the authors propose further research into the impact of different architectural choices and training strategies on segmentation performance, particularly for challenging substructures such as necrosis and non-enhancing tumours. Addressing these areas could lead to the development of more robust, efficient, and adaptable models, ultimately improving clinical outcomes in brain tumour diagnosis and treatment.

Kamnitsas et al. [ 66 ] further developed the Ensemble of Multiple Models and Architectures (EMMA), combining DeepMedic, FCN, and U-Net to boost performance and reduce overfitting.

The study makes significant contributions to medical image analysis, particularly in brain tumour segmentation. The authors introduce EMMA, an ensemble method that integrates multiple models and architectures to improve segmentation accuracy. By combining diverse neural network configurations and training strategies, the study demonstrates that ensembling effectively reduces biases and variances inherent in individual models, resulting in enhanced robustness for segmenting both high-grade and low-grade gliomas demonstrating the benefits of ensemble methods in increasing robustness and accuracy. The approach is validated using a comprehensive dataset from the Brain Tumour Segmentation Challenge 2017, underscoring its effectiveness.

The study also acknowledges several limitations. A key challenge is the reliance on the quality and diversity of training data, which can affect the generalizability of the models. Additionally, the complexity of the ensemble method introduces computational overhead, potentially limiting its feasibility for real-time clinical applications. While the approach improves segmentation performance, it does not fully address the issue of model interpretability, which remains a critical concern in medical applications where understanding the decision-making process is essential.

The study identifies several research gaps and future directions. These include exploring unsupervised and semi-supervised learning techniques to enhance model training with limited labeled data, as well as integrating multi-modal imaging data to further improve segmentation accuracy and robustness. Future work could also focus on developing more efficient ensemble methods that reduce computational demands without compromising performance. Finally, advancing model interpretability through explainable AI techniques could facilitate the practical adoption of these methods in clinical environments, bridging the gap between cutting-edge research and real-world applications.

Myronenko [ 67 ] proposed a 3D encoder-decoder architecture with an additional branch for variational autoencoder (VAE) regularisation. This approach aimed to improve segmentation accuracy by incorporating a generative model to learn a latent representation of the input data.

The study presents significant contributions to the field of brain tumour segmentation through the development of an innovative autoencoder-based approach. The authors introduce a semantic segmentation network that utilizes an encoder-decoder architecture, which effectively addresses the challenges of limited training data. By employing test time augmentation and ensembling multiple models, the approach enhances segmentation accuracy across three nested tumour subregions: whole tumour, tumour core, and enhancing tumour. The results achieved in the BraTS 2018 challenge demonstrate the method’s effectiveness, positioning it among the top-performing submissions and showcasing its potential for clinical applications in neuro-oncology.

Despite these advancements, the study acknowledges certain limitations. One notable constraint is the reliance on the availability of high-quality annotated datasets, which can be scarce in medical imaging. Additionally, while the proposed method shows promising results, it may still struggle with variability in tumour presentations and imaging artifacts that can affect segmentation performance. The authors also highlight the computational demands of training and deploying deep learning models, which may limit accessibility for some research and clinical settings.

This proposed study identifies several research gaps and future directions. There is a need for further exploration of unsupervised and semi-supervised learning techniques to mitigate the dependency on annotated data. Additionally, the authors suggest investigating the integration of multi-modal imaging data to improve segmentation robustness and accuracy. Future work could also focus on real-time segmentation applications in clinical environments, as well as the development of user-friendly tools that facilitate the adoption of these advanced techniques by healthcare professionals. Overall, the study lays a foundation for ongoing research in automated brain tumour segmentation, emphasizing the importance of collaboration between machine learning and clinical expertise.

Jiang et al. [ 68 ] introduced a Two-Stage Cascaded U-Net architecture that refined segmentation predictions in a coarse-to-fine manner. The first stage produced a preliminary segmentation, which was then used as input to the second stage for refinement.

The paper makes several significant contributions to the field of brain tumour segmentation. It introduces a novel two-stage cascaded U-Net architecture designed to segment tumour substructures from coarse to fine detail. Trained end-to-end on the BraTS 2019 dataset, which includes diverse high-grade and low-grade gliomas, the model achieved state-of-the-art performance, securing first place in the BraTS 2019 challenge. The method demonstrated exceptional robustness for different tumour regions.

A key limitation is the reliance on the BraTS 2019 dataset, which may not fully capture the variability encountered in real-world clinical scenarios. Additionally, the model’s performance may be influenced by the quality and consistency of input images, and its complex architecture raises concerns about potential overfitting. The authors also note that variability in segmentation performance across different models suggests a need for further refinement and validation to ensure generalizability.

The research gaps and future directions in this study include exploring more diverse datasets to enhance the model’s robustness and applicability across clinical settings. The authors also propose investigating the integration of additional modalities and advanced techniques, such as attention mechanisms or unsupervised learning, to further improve segmentation accuracy. Future work could focus on enabling real-time segmentation capabilities and developing user-friendly tools for clinicians, ultimately aiming to improve patient outcomes through more efficient and accurate tumour analysis and monitoring.

Isensee et al. [ 69 ] proposed the nnU-Net, a self-configuring framework that automatically adapted to new datasets by adjusting various parameters, including network topology, pre-processing, and training details. They replaced instance normalisation with batch normalisation.

The paper highlights significant contributions of the nnU-Net to brain tumour segmentation. The authors demonstrated the generalizability of nnU-Net by achieving high segmentation accuracy without requiring extensive modifications. By implementing specific enhancements tailored for the BraTS 2020 challenge, the model outperformed the baseline on the validation set. The final ensemble model, selected based on the best-performing configurations, achieved impressive Dice scores and HD95 values for whole tumour, tumour core, and enhancing tumour segmentation, ultimately securing first place in the BraTS 2020 competition. This work underscores the effectiveness of nnU-Net in medical image segmentation, particularly for brain tumours.

The study acknowledges notable limitations. The study covers only a limited number of modifications and lacks extensive experimental validation, raising concerns about the robustness and reproducibility of the results. The performance improvements observed may not generalize across different datasets or clinical scenarios. Additionally, reliance on specific validation metrics, such as Dice scores and HD95, may not fully capture the clinical relevance of the segmentation results, potentially limiting the applicability of the findings in real-world settings.

Several research gaps and future directions can be explored from this study. These include investigating more extensive modifications and optimizations of the nnU-Net architecture, such as integrating attention mechanisms or multi-task learning. Comprehensive validation studies across diverse datasets would further enhance understanding of the model’s generalizability and robustness. Additionally, addressing class imbalance in brain tumour segmentation through novel loss functions or data augmentation strategies could improve performance. These efforts have the potential to advance segmentation accuracy and contribute to better clinical outcomes for patients with brain tumours.

Luu and Park [ 70 ] extended the nnU-Net by doubling the filter sizes in the initial layers and implementing group normalisation instead of batch normalisation.

The proposed study makes significant contributions to brain tumour segmentation by extending the nnU-Net framework, a well-established method in medical image analysis. The authors introduced several key modifications, including a larger network architecture, the replacement of batch normalization with group normalization, and the incorporation of axial attention in the decoder. These enhancements led to improved performance in the Brain Tumour Segmentation Challenge (BraTS) 2021, where their models secured first place in the final ranking on unseen test data. Additionally, the authors made their codes, pretrained weights, and Docker image publicly available, promoting transparency and enabling further research in the field.

Despite these advancements, the study acknowledges certain limitations. A notable challenge is the model’s dependency on high-quality MR images, as segmentation performance can degrade with poor image quality or artifacts. This underscores the importance of data integrity in training robust models. Furthermore, while the modifications to nnU-Net yielded incremental improvements over the baseline, the authors suggest that further experimentation is needed to fully explore the potential of these enhancements and their impact on segmentation accuracy across diverse datasets.

The research gap identified in the study highlights the need for more comprehensive datasets that capture a wider variety of tumour characteristics and imaging conditions. Future directions include exploring diverse data acquisition methods and advanced data augmentation techniques to address edge cases where segmentation performance may falter. The authors also emphasize the importance of analysing failure cases to refine model behaviour and improve overall accuracy. These efforts could lead to the development of more resilient algorithms capable of handling the complexities of brain tumour imaging in clinical settings.

Zeineldin et al. [ 71 ] developed an ensemble approach combining multiple frameworks, including DeepSeg, nnU-Net [ 69 ], and DeepSCAN, for automatic glioma segmentation in pre-operative MRIs.

The study makes significant contributions to brain tumour segmentation by developing an ensemble model that integrates multiple state-of-the-art U-Net variants, including DeepSeg, nnU-Net, and DeepSCAN. This ensemble approach achieved exceptional performance in the BraTS-CE 2022 challenge, delivering high Dice Similarity Coefficients (DSC) and low Hausdorff distances (HD95) for the enhancing tumour (ET), tumour core (TC), and whole tumour (WT). The method secured first place in the competition and demonstrated strong generalizability on unseen test datasets, highlighting its potential for clinical applications in automatic glioma boundary detection using pre-operative MRI scans.

The study acknowledges certain limitations, particularly in the context of pediatric brain tumour segmentation. The inherent variability and complexity of pediatric tumours, which often exhibit heterogeneous tissue types, pose significant challenges for accurate segmentation. The model’s performance on the pediatric test dataset was less robust compared to adult datasets, underscoring the need for specialized techniques and larger annotated datasets tailored to paediatric cases. This limitation highlights the importance of further research to address the unique characteristics of paediatric brain tumours.

This study identifies several research gaps and future directions. These include developing more sophisticated models capable of handling the complexities of paediatric brain tumour segmentation, integrating multimodal imaging data, and exploring advanced post-processing techniques to enhance accuracy. Additionally, expanding dataset diversity and size could improve model generalization and robustness across different patient populations. Addressing these gaps could lead to more effective and reliable tools for brain tumour diagnosis and treatment planning, ultimately benefiting both adult and pediatric patients.

Ferreira et al. [ 72 ] employed synthetic data augmentation through Generative Adversarial Networks (GANs) and integrated multiple deep learning models, including nnU-Net and Swin UNETR.

The study makes significant contributions to brain tumour segmentation, particularly in the context of the BraTS 2023 Adult Glioma challenge. The authors introduced an innovative approach combining Enhanced Synthetic Data Augmentation with Model Ensemble techniques. By leveraging generative adversarial networks (GANs) and advanced registration methods, they increased the volume and quality of training data, significantly improving the performance of deep learning models in segmenting gliomas. This work not only advances the state-of-the-art in brain tumour segmentation but also provides a framework for future research in synthetic data generation and model optimization.

The study acknowledges certain limitations. A key challenge is the reliance on high-quality annotated datasets, which are often scarce and time-consuming to obtain. Additionally, while the proposed methods show promising results, they may not generalize well across diverse patient populations or different imaging modalities. The authors also highlight the computational demands of their approach, which could limit its applicability in real-time clinical settings. These limitations suggest that further refinement and validation are needed to ensure robustness and practicality.

The study identifies several research gaps and future directions. These include exploring the generalizability of the proposed methods across diverse demographics and imaging techniques, as well as investigating the integration of multimodal data to further enhance segmentation accuracy. Future work could also focus on developing more efficient algorithms to reduce computational costs while maintaining high performance. Overall, the findings emphasize the importance of continued innovation in synthetic data generation and deep learning methodologies to address the evolving challenges in medical imaging and tumour segmentation.

Table 4 summarizes the methods and results from the BraTS Challenges between 2014 and 2023, showcasing advancements in segmentation techniques for whole tumor (WT), core tumor (CT), and enhanced tumor (ET). Early approaches, such as the 2014 benchmark by Menze et al., established foundational metrics with Dice scores of >0.80 (WT), 0.70 (CT), and 0.60 (ET). Subsequent methods, including 3D CNNs, ensembles, and advanced U-Net architectures, demonstrated significant improvements in segmentation accuracy. Notably, Luu and Park’s enhanced nnU-Net achieved the highest Dice scores in 2021, with 0.938 (WT), 0.923 (CT), and 0.882 (ET). Recent efforts in 2023, incorporating synthetic data augmentation and GANs, further refined performance, emphasizing the growing sophistication of deep learning techniques in medical imaging.

A major challenge consistently identified in the literature is the significant variability in brain tumours’ size, shape, location, and appearance across patients [ 21 , 22 , 24 ]. This heterogeneity complicates model generalisation, as models trained on datasets with limited tumour characteristics may not perform well on the diverse range of tumours encountered in clinical practice. The infiltrative growth patterns and indistinct boundaries of gliomas further exacerbate the challenges of accurate segmentation.

The scarcity of high-quality, annotated medical imaging data remains a significant bottleneck [ 57 , 64 ]. This scarcity stems from privacy concerns, the labour-intensive nature of manual annotation, and the relatively low prevalence of certain tumour types. Moreover, the inherent class imbalance between the vast non-tumour regions and the relatively small tumour regions can bias models towards the majority class, leading to reduced segmentation performance, particularly for smaller tumours or tumour subregion. This is particularly concerning in the medical domain, where false negatives can have severe clinical consequences.

Differences in MRI scanners, acquisition parameters, and imaging protocols across institutions introduce considerable variability that significantly affects model performance and generalisation. Models trained on data from one institution may not perform adequately on data acquired from another due to differences in image quality, contrast settings, and noise levels. Standardisation of imaging protocols is often proposed as a solution, but it faces practical challenges in implementation.

The literature consistently highlights the substantial computational resources and memory required to train deep 3D models, such as the 3D U-Net, due to the high dimensionality of volumetric data [ 58 , 61 ]. This poses a significant challenge for researchers and clinical institutions with limited access to high-performance computing infrastructure, hindering widespread adoption and limiting the development of more complex, potentially more accurate models.

Many studies report challenges related to overfitting, where models trained on limited datasets capture noise and artefacts instead of the underlying patterns of tumour morphology [ 60 , 64 ]. This leads to poor generalisation performance on external, unseen datasets, severely limiting the clinical utility of such models. Ensuring robust generalisation to new data is, therefore, a critical challenge that needs to be addressed for reliable clinical deployment.

The evolution of brain tumour segmentation techniques, particularly evident in the contributions to the BraTS challenges from 2014 to 2023, reflects a clear trend towards increasingly sophisticated models. Early CNN-based methods achieved baseline performance but struggled with generalisation due to limited data diversity [ 22 ]. The subsequent introduction of 3D CNNs capitalised on volumetric data, improving spatial feature representation and leading to significant performance gains. For example, the average Dice scores for whole tumour segmentation in BraTS improved from around 0.8 in the early years to over 0.90 with the adoption of 3D CNNs.

Ensemble methods, which combine multiple architectures to enhance robustness and mitigate individual model weaknesses, have also gained traction. These methods have been shown to further improve segmentation accuracy, often achieving Dice scores of around 0.88 in BraTS challenges. More recently, the nnU-Net framework [ 69 ], which dynamically adapts models to dataset characteristics through normalization techniques and architectural refinements, has emerged as a state-of-the-art approach, achieving Dice scores exceeding 0.9 for certain tumour subregion in recent BraTS challenges.

The incorporation of synthetic data augmentation using Generative Adversarial Networks (GANs) represents another notable trend aimed at addressing data scarcity and diversity issues [ 73 ]. GANs, such as CycleGAN and StyleGAN, have been widely adopted in medical imaging for generating realistic synthetic data [ 74 ]. For instance, CycleGAN has been used for domain adaptation, enabling the translation of images from one modality (e.g., MRI) to another (e.g., CT) while preserving anatomical structures. Similarly, StyleGAN has shown promise in generating high-resolution, diverse tumour images that enhance the generalizability of segmentation models. However, these techniques require careful validation to ensure the generated data are realistic and do not introduce biases that could negatively impact model training. Challenges include mode collapse, where the generator produces limited varieties of images, and the difficulty of ensuring that synthetic images retain clinically relevant features [ 75 ].

In addition to GANs, domain adaptation methods such as unsupervised domain adaptation (UDA) and adversarial domain adaptation have been employed to bridge the gap between source and target domains in medical imaging. For example, UDA techniques like Domain-Adversarial Neural Networks (DANN) have been used to align feature distributions across different imaging modalities, improving model performance on unseen datasets. While these methods reduce the need for extensive labeled data in the target domain, they often struggle with large domain shifts and may require additional fine-tuning for specific applications [ 76 ].

Despite these advancements, the reviewed literature indicates that significant challenges remain in handling the full spectrum of tumour heterogeneity and reducing the computational demands of these models. For instance, GAN-based approaches are computationally intensive and may require significant resources for training and validation [ 77 ]. Furthermore, the lack of standardized evaluation metrics for synthetic data quality poses a barrier to widespread adoption in clinical settings.

The BraTS dataset, while comprehensive, has several limitations that impact its applicability to real-world clinical scenarios. Firstly, the dataset primarily focuses on adult gliomas and may not fully capture the variability seen in paediatric brain tumours or rare tumour types. This limits the generalizability of models trained on BraTS data. Additionally, the reliance on pre-operative MRI scans excludes post-operative or follow-up imaging, which are critical for longitudinal studies and treatment monitoring in clinical practice. Another limitation is the annotation process, which relies on a limited number of experts. This may not fully represent the variability in clinical interpretations, potentially introducing biases in model training and evaluation. Furthermore, the lack of multi-institutional annotations or consensus-based ground truth may restrict the dataset’s applicability to diverse clinical settings. Finally, while metrics like Dice scores and Hausdorff distances are widely used, they may not fully capture the clinical relevance of segmentation results. For example, small errors in critical regions, such as tumour boundaries near vital structures, may have significant clinical implications but are not adequately reflected in these metrics.

To address these limitations and better align the BraTS challenge with real-world clinical needs, we propose several improvements for future iterations.

Despite the impressive performance of models trained on BraTS data, there remains a gap between dataset performance and practical clinical applications. Models may struggle with real-world challenges such as poor image quality, imaging artifacts, or heterogeneous tumour presentations. To bridge this gap, we propose robustness testing using datasets with intentionally degraded images or artifacts, clinical validation studies to assess model performance in real-world settings, and the development of user-friendly tools that facilitate the integration of segmentation models into clinical workflows. By addressing these limitations and incorporating these suggestions, future iterations of the BraTS challenge can better reflect real-world clinical needs and contribute to the development of more effective and reliable tools for brain tumour diagnosis and treatment.

For clinicians, the integration of reliable and accurate segmentation models holds the potential to revolutionise patient care by enabling precise tumour delineation, which is crucial for treatment planning, surgical guidance, and monitoring treatment response. Studies have shown that automated segmentation can reduce the time required for tumour delineation significantly compared to manual segmentation, potentially decreasing clinician workload and minimising the risk of human error. This could lead to faster treatment planning and, potentially, improved patient outcomes. However, several practical hurdles need to be addressed for widespread clinical adoption. These include the need for seamless integration with existing hospital information systems, the development of user-friendly interfaces that do not disrupt established clinical workflows and obtaining necessary regulatory approvals. Moreover, clinicians require models that are not only accurate but also interpretable and trustworthy to confidently integrate them into their decision-making processes.

For researchers, the reviewed literature underscores the need to address the challenges of data scarcity, model generalisation, and computational efficiency. Developing models that perform consistently across diverse patient populations and imaging protocols is essential for facilitating clinical translation. Collaborative efforts between institutions to create larger, more diverse datasets are frequently proposed as a key strategy to improve model robustness and generalizability.

Based on the comprehensive review of the current literature, future research should prioritise the following areas to overcome the identified challenges and accelerate the clinical translation of deep learning-based brain tumour segmentation: