The development of automated segmentation methods has been driven by the need to overcome the limitations of traditional techniques such as region growth and active contour models, which often struggle with the complex and variable nature of tumors [8][9]. Convolutional Neural Networks (CNNs) marked a significant shift, as they learn hierarchical features directly from data, demonstrating superior performance over traditional machine learning algorithms like AdaBoost and Support Vector Machines .

The U-Net architecture has since become a cornerstone in biomedical image segmentation due to its impressive performance with limited data and its symmetric encoder-decoder structure, which is crucial for retaining spatial information [4][11][12]. Its success has inspired numerous variants:

Cascaded and Hybrid Models:Christ et al. proposed a method using cascaded fully convolutional networks (CFCNs) to segment the liver as a region of interest before segmenting lesions, achieving high Dice scores. Similarly, Chlebus et al. used a 2D FCN with an object-based post-processing step to reduce false positives by 85%. Rahman et al. introduced a hybrid ResUNet model, combining ResNet and U-Net, which demonstrated high accuracy and Dice coefficients.

Architectural Enhancements: The SegNet architecture, originally for road scene segmentation, has been modified for liver tumor segmentation, showing promising accuracy but with some false positives . To address U-Net’s limitations, models like UNet++ were introduced with nested, dense skip pathways to reduce the semantic gap between encoder and decoder feature maps . Further innovations include MultiResUNet and Recurrent Residual U-Net (R2U-Net) , which enhanced feature representation and improved training efficiency.

Transformer-BasedModels: The emergence of Vision Transformers has addressed U-Net’s challenge in modeling long-range dependencies. Wang et al. introduced the Mixed Transformer U-Net (MT-UNet), which uses a novel self-attention mechanism to capture both intra- and inter-affinities, outperforming state-of-the-art methods with lower computational complexity.

While advanced architectures have improved performance, their effectiveness is highly sensitive to the choice of hyperparameters. Manual tuning is computationally expensive, leading to suboptimal results . This has led researchers to explore automated optimization techniques.

Studies have leveraged deep learning for the classification of other cancers, often employing metaheuristic algorithms for optimization. For instance, Jamshidi et al. used a VGG-19-based CNN and an ANN with transfer learning for brain tumor classification, achieving high accuracy rates. Similarly, another study by Jamshidi et al. applied a Multi-Layer Perceptron (MLP) with the Dragonfly Algorithm (DA) for lung cancer classification, achieving a remarkable accuracy of 99.82%. These studies demonstrate the efficacy of combining deep learning with metaheuristic algorithms in the medical domain.

The Bat Algorithm (BA), introduced by Yang , is a metaheuristic technique that has shown efficiency in solving various optimization problems. A comprehensive review by Shehab et al. evaluated BA’s effectiveness across diverse fields, including image processing and medical applications, highlighting its potential for hyperparameter optimization in deep learning. Other studies have also explored methods to improve segmentation through transfer learning and attention mechanisms, such as those used by Oktay et al. .

Despite significant advancements, a research gap remains in the comprehensive application of metaheuristic algorithms for hyperparameter optimization of 3D deep learning architectures for liver tumor segmentation. The combination of a 3D U-Net, which excels in volumetric data analysis, with the Bat Algorithm, which efficiently tunes hyperparameters, is a promising yet underexplored area. This study aims to address this gap, proposing a robust and automated framework for accurate liver tumor segmentation.

We used the 3D Liver and Liver Tumor Segmentation Dataset, which includes 123 3D CT scans with tumor masks, downloaded from Kaggle. This dataset, designed to advance liver and tumor segmentation, features various medical imaging modalities like CT and MRI, and includes data from a diverse patient population. Expert radiologists meticulously annotated each image to highlight liver boundaries and tumor regions, providing ground truth labels for training and evaluation.

The dataset supports 3D segmentation tasks, reflecting the complexity of liver anatomy and pathology, and includes a wide range of pathological variations, such as benign and malignant tumors. This diversity helps in developing algorithms that can handle different tumor characteristics.

Privacy and ethical guidelines were strictly followed, with patient identifiers anonymized. Preprocessing steps ensured data consistency and computational efficiency, including standardizing imaging modalities, normalizing intensity values, and resampling images to a uniform resolution. Data augmentation techniques like rotation, scaling, and flipping were applied to increase training data variability and improve model robustness.

Overall, this well-curated, diverse, and ethically sourced dataset aims to facilitate advancements in medical image analysis, contributing to better diagnosis, treatment planning, and patient care in hepatology.

All images were preprocessed before being fed into the proposed model through the following three steps:

Step 1 Normalization: Each CT scan was normalized to the range [0, 1] by dividing the pixel intensities by the maximum value within each scan. Normalization is a crucial step in preprocessing as it mitigates the variations in imaging conditions across different scans, allowing the model to focus on the anatomical structures rather than the variability in pixel intensities.

Step 2 Resizing: The 3D volumes were resized to a fixed dimension of 64 × 64 × 32 using bilinear interpolation. This resizing was performed to standardize the input size for the neural network, facilitating batch processing and reducing the computational burden. The chosen dimensions balance the need for preserving anatomical details while ensuring the model operates efficiently.

Step 3 Data Augmentation: To further enhance the robustness of the model, data augmentation techniques such as random rotations, flips, and shifts were applied to the training dataset. Augmentation helps artificially increase the diversity of the training data, enabling the model to generalize better to unseen data.

The proposed 3D U-Net model is specifically designed to capture the intricate details of liver tumors within volumetric CT data. The architecture follows the classic encoder-decoder structure, with enhancements to improve spatial information preservation.

Encoder (Contraction Path): The encoder consists of a series of 3D convolutional layers, each followed by a ReLU activation function and 3D max-pooling operations. These layers progressively reduce the spatial dimensions of the input while increasing the depth, allowing the model to capture higher-level features that are essential for accurate segmentation.

Bottleneck: At the bottleneck, the network reaches its deepest layer, where the most abstract and high-level features are extracted. This layer serves as the bridge between the encoder and decoder, effectively summarizing the information necessary for accurate reconstruction in the expansion path.

Decoder (Expansion Path): The decoder mirrors the encoder, using 3D up-sampling and skip connections to reconstruct the spatial dimensions of the original image. The skip connections from corresponding encoder layers help in retaining spatial information lost during the down-sampling process, which is critical for the precise localization of the tumor regions.

Final Output Layer: The final layer of the network applies a 3D convolution with a sigmoid activation function to produce the segmentation map. The sigmoid activation ensures that the output is a probability map, indicating the likelihood of each voxel being part of the tumor.

Key Components of the 3D U-Net Algorithm

The 3D U-Net is a specialized convolutional neural network (CNN) designed for volumetric image segmentation, particularly prevalent in medical imaging. It extends the foundational 2D U-Net by adapting its operations for three-dimensional data. Its core strength lies in combining detailed spatial information with high-level contextual features for precise segmentation.

The key components of the 3D U-Net model are detailed in .

Table 1. Key components of the 3D U-Net algorithm.

To optimize the performance of the 3D U-Net model, the Bat Algorithm was employed to fine-tune critical hyperparameters such as the learning rate and batch size. The Bat Algorithm, a bio-inspired metaheuristic, simulates the echolocation behavior of bats to balance exploration and exploitation within the hyperparameter search space. The key components and their tunings are summarized in .

Table 2. Key components and their tunings used in the Bat Algorithm (BA).

Initialization: The optimization process begins with a population of bats, each representing a candidate solution with randomly initialized values for the learning rate and batch size. These initial values are chosen within a predefined range based on prior knowledge or heuristic estimation.

Fitness Evaluation: The fitness of each bat is evaluated based on the validation loss after training the model for a small number of epochs. The validation loss serves as a proxy for model performance, reflecting how well the model generalizes to unseen data.

Optimization Process: During each iteration, the bats update their velocities and positions based on the best solutions found so far. The Bat Algorithm adapts the exploration rate, allowing the bats to focus more on promising areas of the search space as the optimization progresses. This iterative process continues until the algorithm converges on an optimal or near-optimal set of hyperparameters.

Convergence Criteria: The algorithm concludes when it reaches a predefined number of iterations or when improvements in fitness become negligible, indicating that the optimal hyperparameters have likely been found.

The experiments leveraged its computational capabilities to train the 3D U-Net model efficiently. The dataset was split into training (80%) and validation (20%) sets to evaluate the model’s performance.

4.1.1. Training Protocol

The model was trained for 10 epochs using the hyperparameters optimized by the Bat Algorithm. During training, both training and validation accuracy, as well as loss, were monitored to assess the model’s learning progression and convergence behavior. A relatively small number of epochs was used to quickly evaluate the effectiveness of the hyperparameters, with the option to further fine-tune based on the initial results.

4.1.2. Evaluation Metrics

The performance of the model was evaluated using standard segmentation metrics: accuracy, precision, recall, and F1-score. These metrics were calculated at various thresholds to determine the best operating point for the model, balancing the trade-off between false positives and false negatives.

Accuracy: This fundamental metric quantifies the proportion of true results, including both true positives (TP) and true negatives (TN), within the entire dataset. It reflects the model’s overall effectiveness in lung cancer detection.

Precision: Also known as the positive predictive value, precision measures the ratio of true positives to the total number of true positives and false positives. High precision indicates the model’s reliability in minimizing false-positive diagnoses, thus preventing unnecessary medical interventions.

Recall (Sensitivity): Recall calculates the model’s ability to correctly identify all actual cases of lung cancer, represented by the ratio of true positives to the sum of true positives and false negatives. This metric is essential for ensuring comprehensive cancer detection.

F1 Score: The F1 score balances precision and recall, providing a single measure that harmonizes both metrics. It is particularly useful for achieving a balance between identifying all relevant instances and maintaining a low false-positive rate.

The metrics were calculated using a separate test dataset, distinct from the training data, to ensure an unbiased evaluation of the model’s performance.

The training process demonstrated a consistent improvement in both accuracy and loss metrics, indicating effective learning and convergence. The Bat Algorithm successfully identified an optimal learning rate of 0.001 and a batch size of 3, leading to improved validation performance.

To visually illustrate the learning behavior of the model and the impact of the optimized hyperparameters, presents the loss and accuracy of training and validation recorded over 10 epochs. As depicted, the model quickly converges, showcasing stable performance and generalization ability.

Figure 1. The figure displays the model’s training and validation performance over 10 epochs. Accuracy (on the left) consistently increases while loss (on the right) steadily decreases, with both training and validation curves remaining closely aligned. This indicates effective learning without overfitting.

Validation Accuracy: The final model achieved a high validation accuracy of 98.74%, with a corresponding validation loss of 0.0484. These results suggest that the model is well-calibrated and effectively captures the underlying patterns in the validation set, while also being robust against overfitting.

Optimization Outcome: The optimized hyperparameters resulted in a well-balanced model that generalizes effectively across the validation data, avoiding common pitfalls such as underfitting or overfitting. The use of the Bat Algorithm proved to be an efficient method for hyperparameter tuning, yielding a model that performs well across different metrics.

The model’s performance was evaluated across different probability thresholds to identify the optimal operating point. At a threshold of 0.3, the model achieved the best balance between precision (0.6323) and recall (0.5303), resulting in an F1-score of 0.5768. This threshold allows the model to capture a higher number of true positives, which is crucial in medical diagnosis, where sensitivity is often prioritized.

As the threshold increased, precision improved slightly, but this came at the expense of recall. Beyond a threshold of 0.4, the model’s ability to make positive predictions diminished significantly, leading to zero precision, recall, and F1-score. This suggests that a lower threshold is preferable for this segmentation task, as it enables the model to identify more true positive cases, which is particularly important in clinical settings.

To provide a more detailed view of the model’s performance, the confusion matrix at the optimal threshold of 0.3 is presented in . The overall trade-off between sensitivity and specificity across all possible thresholds is summarized by the ROC curve in . Furthermore, the statistical distribution of the F1-scores across the entire test set is visualized using a box plot in .

Figure 2. The confusion matrix shows strong model performance, with 3,100,000 correct “No Tumor” and 99,344 correct “Tumor” predictions. The model had 42,526 false positives and 20,000 false negatives, highlighting a crucial balance to consider for clinical use.

Figure 3. The ROC curve shows excellent model performance with an AUC of 0.99, indicating a strong ability to distinguish between cases. The confusion matrix details the predictions: it correctly identified 3,100,000 “No Tumor” and 99,344 “Tumor” cases, while having 42,526 false positives and 20,000 false negatives.

Figure 4. The scattered box plot shows consistent model performance, with a median Dice score of approximately 0.76. The majority of patient scores fall between 0.72 and 0.79, but a few outliers with lower scores are also present, highlighting the model’s overall accuracy and variability.

To further evaluate the model’s performance, the segmentation results were visualized on several validation samples. These visualizations provide qualitative insights into how well the model identifies tumor regions and the spatial accuracy of its predictions. By comparing the model’s output against the ground truth annotations, we can assess the effectiveness of the optimized U-Net architecture in accurately delineating liver tumors.

For a qualitative assessment of the model’s performance, presents representative examples. These include original CT scan slices, their corresponding ground truth segmentations, and the predicted segmentations generated by our 3D U-Net model. These visual results underscore the model’s proficiency in accurately delineating lung cancer regions. Additionally, a heatmap of the prediction probabilities for a representative sample is shown in , providing a continuous measure of the model’s confidence across the segmented region.

(a)

(b)

(c)

Figure 5. Visual comparison of original CT slices, corresponding ground truth segmentations, and predicted segmentations generated by the 3D U-Net model. Each row illustrates a distinct sample, with columns depicting the original image (a), ground truth (b), and model prediction (c).

Figure 6. The figure visualizes the model’s liver tumor segmentation performance using a three-panel display: the original CT image, a prediction heatmap, and the ground truth mask. By aligning the high-probability areas of the heatmap with the ground truth, the figure confirms the model’s accurate spatial and quantitative localization of the tumor.

To comprehensively evaluate the efficacy of our proposed models for lung cancer segmentation, their performance was rigorously benchmarked against a diverse set of existing methodologies reported in the literature. These comparative studies, detailed in , encompass various deep learning architectures and traditional machine learning approaches.

As evidenced by , our Proposed Optimized 3D U-Net with Bat Algorithm demonstrates superior accuracy, achieving 98.74%. This result surpasses all comparative models in the table, including other sophisticated CNN and deep learning architectures such as the Modified AlexNet (96.80% by ) and the CNN AlexNet (97.25% by ). The model’s discriminative performance is further validated by a high AUC score of 0.99, which is competitive with or exceeds the values reported by other deep learning models in the literature, such as the CNN model with 0.98 .

Table 3. Performance comparison of different models.

The table also includes our Proposed MASE (Multi-Att. Ens.) model, which achieves a high accuracy of 98.09% and a near-perfect AUC of 0.9961. This model demonstrates exceptional performance in two critical metrics, achieving a Sensitivity of 98.73% and a Specificity of 98.96%. These high scores indicate a superior ability to correctly identify positive cases (tumors) while effectively ruling out negative cases, which is paramount in clinical diagnostics.

While the table presents varied metrics across different studies, making direct F1-Score, Precision, and Recall comparisons challenging for all entries due to their absence in some prior works, our Optimized 3D U-Net model provides a complete set of these critical segmentation metrics. With an F1-score of 0.5768, a Precision of 0.6323, and a Recall of 0.5303, this model represents a balanced measure of the ability to accurately identify positive cases while minimizing false positives and false negatives. It is important to note that many works in the literature focus on classification (malignant/benign) rather than pixel-level segmentation, which often results in higher reported accuracy but does not capture the precise spatial delineation critical for surgical planning and treatment monitoring.

The enhanced performance of our proposed Optimized 3D U-Net model can be primarily attributed to two key factors:

1) The inherent capability of the 3D U-Net architecture to effectively capture volumetric contextual information and hierarchical features from CT scan data, which is crucial for precise 3D medical image segmentation.

2) The sophisticated hyperparameter optimization performed by the Bat Algorithm. As discussed in Section 3.1, this metaheuristic approach effectively navigated the complex search space of learning rates and batch sizes, enabling the model to converge to an optimal configuration that maximizes generalization performance and robustness against overfitting. This contrasts with models relying on empirical tuning or simpler optimization strategies.

These findings collectively affirm that the synergistic combination of a robust 3D deep learning architecture with an intelligent metaheuristic optimization strategy yields a highly effective and competitive solution for automated lung cancer segmentation in CT imaging, demonstrating significant potential for clinical application.