Architectural Overview of DDPVT-Net

As illustrated in Figure 1, the core of DDPVT-Net employs a Pyramid Vision Transformer (PVT) [12] encoder to extract multi-scale feature maps across four hierarchical stages from input images. These feature maps establish long-range dependencies through self-attention mechanisms, forming the foundation for subsequent processing. Features from the latter three encoder stages are channeled into a Multi-scale Feature Aggregation (MFA) module, which integrates cross-stage contextual information through adaptive channel weighting and spatial attention, ultimately generating a unified global feature map. This aggregated feature map subsequently guides both the CNN decoder and reverse attention module to concentrate computational resources on diagnostically relevant regions, thereby enhancing segmentation precision.

The CNN decoder progressively reconstructs spatial resolution through a

cascade of upsampling operations. During this process, it synthesizes high-level semantic features from the reverse attention module with localized texture details delivered via skip connections, employing element-wise summation for multi-source feature fusion. Two residual blocks within the decoder further refine these hybrid features through depthwise separable convolutions, enhancing the model’s capacity to capture intricate mucosal patterns and microvascular structures characteristic of colorectal polyps.

Complementing this pathway, the MLP decoder operates as a parallel processing stream specializing in abstract pattern recognition. Through sequential fully connected layers, it transforms the global feature map into discriminative class-specific embeddings, focusing on inter-region differentiation rather than spatial preservation. This design enables efficient identification of subtle intensity variations between polyp tissue and surrounding mucosa, particularly effective in low-contrast endoscopic imaging scenarios.

The final segmentation output is generated through weighted fusion of predictions from both decoders. This synergistic integration capitalizes on the CNN decoder’s proficiency in spatial detail reconstruction and the MLP decoder’s strength in semantic boundary refinement. The combined approach ensures comprehensive utilization of both local textural cues and global contextual awareness, achieving superior performance in delineating polyps with irregular morphologies and indistinct margins.

DDPVT-Net addresses critical challenges in colorectal polyp segmentation, including morphological heterogeneity, scale variance, class imbalance, and boundary ambiguity. The PVT encoder’s multi-scale feature hierarchy combined with the MFA module’s cross-stage fusion strategy effectively amplifies underrepresented class features. Simultaneously, the CNN decoder’s residual refinement blocks enhance boundary precision, while the MLP decoder’s nonlinear transformations ensure robustness against imaging artifacts and illumination variations. This dual-decoder architecture establishes a new paradigm for gastrointestinal lesion analysis, demonstrating significant advancements in computational endoscopic diagnostics.

To address the challenges of colorectal polyp segmentation―including morphological diversity, size variability, class imbalance, and boundary ambiguity―we employ the Pyramid Vision Transformer (PVT) [13] as the backbone network, particularly effective in handling endoscopic imaging artifacts such as motion blur, rotational distortions, and specular reflections. Recent studies [14]-[16] demonstrate that Vision Transformers surpass conventional CNNs in both performance and robustness. As an optimized pyramid-structured variant, PVT achieves effective multi-scale structural perception by preserving global contextual information while capturing fine-grained image details. Its hierarchical architecture not only maintains compatibility with convolutional layers but also outperforms standard Vision Transformers (ViT) in local-global feature fusion, making it uniquely suited for modeling objects with diverse shapes and sizes.

This work adopts PVTv2 [17], an enhanced version of PVT, which significantly improves feature extraction capabilities to support precise polyp segmentation. The PVT family, through its global-local feature fusion mechanisms, seamless CNN compatibility, and advanced modeling of complex lesion morphologies, establishes an optimal framework for colorectal polyp analysis. These improvements collectively enhance segmentation accuracy and robustness, as evidenced by comprehensive experimental validations. The architectural details of PVT and its advanced variant PVTv2 are illustrated in Figure 2.

DDPVT-Net incorporates a reverse attention mechanism to augment local feature discrimination. While the Pyramid Vision Transformer (PVT) excels in global context modeling, it struggles with small-sized and irregularly shaped polyps under noisy endoscopic conditions (e.g., motion blur, boundary ambiguity). The reverse attention module selectively amplifies subtle transitional regions neglected by PVT through spatial-channel suppression of high-confidence areas, thereby

enhancing sensitivity to fine-grained morphological variations.

By synergizing PVT’s global semantic extraction with reverse attention’s boundary-aware refinement, DDPVT-Net achieves precise and robust polyp segmentation. This dual-stream architecture not only facilitates early colorectal cancer detection by resolving submillimeter lesion ambiguities but also mitigates class imbalance through adaptive region reweighting. The mathematical formulation of the reverse attention mechanism is defined as:

RA i = f i ⊙ A i (1)

A i = Θ ( σ ( U P ( g i + 1 ) ) ) (2)

where A i represents the weight of the inverse attention, ⊙ represents the multiplication of the corresponding elements of the feature map, U P represents the upsampling operation, σ represents the Sigmoid function, and Θ represents the inverse operator subtracting the input from the full 1 matrix.

The MFA module addresses the limitations of single-scale feature representation by hierarchically integrating multi-stage features from the Pyramid Vision Transformer (PVT) encoder. Through cascaded fusion of fine-grained details from shallow layers and coarse-grained semantic contexts from deep layers, MFA constructs comprehensive feature representations that are particularly effective in segmenting colorectal polyps with morphological heterogeneity and boundary ambiguity (Figures 3-4).

By aggregating feature maps from the last three encoder stages via cross-stage concatenation and adaptive channel reweighting, MFA provides the CNN decoder with enriched texture details for boundary refinement while supplying the MLP decoder with discriminative patterns for precise classification. This dual-stream

enhancement leverages both local continuity and global dependencies, enabling the model to dynamically prioritize diagnostically critical regions. The mathematical formulation of MFA is defined as follows:

f 21 = Conv 3 × 3 ( Up ( f 4 ) ) ⊙ f 3 (1)

f 22 = Cat ( Conv 3 × 3 ( Up ( f 4 ) ) , f 21 ) (2)

f 31 = Conv 3 × 3 ( Up ( Up ( f 4 ) ) ) ⊙ Conv 3 × 3 ( Up ( f 4 ) ) ⊙ f 2 (3)

f 32 = Cat ( f 31 , Conv 3 × 3 ( Up ( Conv 3 × 3 ( f 22 ) ) ) ) (4)

MFAResult = Conv 1 × 1 ( Conv 3 × 3 ( f 32 ) ) (5)

where ⊙ represents the multiplication of feature maps Cat() represents the concatenation of feature maps along the channel dimension, Up() represents the upsampling operation, and Conv_(1 × 1) and Conv_(3 × 3) represent convolution operations with kernels of 1 and 3, respectively.

The DDPVT-Net architecture integrates an MLP decoder as the secondary processing stream to analyze fused feature maps, prioritizing class-discriminative attributes over spatial localization. This branch enhances inter-class separability through hierarchical nonlinear transformations, generating domain-invariant representations that improve generalization across heterogeneous endoscopic imaging conditions. In contrast to the CNN decoder’s convolutional operations for pixel-wise boundary reconstruction, the MLP decoder employs fully connected layers to model global anatomical semantics and topological consistency, effectively resolving ambiguous classification regions through long-range dependency learning. The synergistic fusion of both decoders’ outputs achieves dual optimization: the CNN pathway preserves submillimeter structural details via residual skip connections, while the MLP pathway establishes diagnostic-relevant feature manifolds to mitigate class imbalance. This complementary paradigm attains state-of-the-art performance (93.2% Dice, 89.7% F-score) by harmonizing local texture fidelity with global semantic coherence in complex colorectal polyp segmentation tasks (Figure 5).

To comprehensively evaluate the performance of DDPVT-Net in medical image segmentation, we conduct experiments on five public datasets, including Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, SETIS-LaribPolypDB, and CVC-300. The experimental setup followed the criteria of PraNet to ensure the consistency and fairness of the evaluation. In the Kvasir-SEG and CVC-ClinicDB datasets, 1612 images were used, divided into training, validation and test sets by 8:1:1 to optimize the model and perform performance evaluation. To verify the generalization ability of the model, it is also tested on 636 images of CVC-ColonDB, ETIS-LaribPolypDB and CVC-300. This method evaluates the performance of DDPVT-Net on specific data sets, shows its adaptability and stability in processing diverse medical images, and highlights the practical application potential of the model.

The experimental framework was implemented on a Windows 64-bit platform using PyTorch 2.1.2 with CUDA 11.8 acceleration and Python 3.8. Training utilized an NVIDIA RTX 4060Ti GPU (16GB VRAM) with Adam optimization [18] (initial learning rate 1e−4), combining binary cross-entropy and Dice loss under a batch size of 8 for 200 epochs, integrated with early stopping to prevent over-optimization. Comprehensive data augmentation included geometric transformations (random flips, center crops, elastic deformations) and intensity variations (Gaussian noise, channel permutation) to enhance cross-domain generalization. Validation strictly applied intensity standardization to maintain clinical workflow authenticity while ensuring computational efficiency. The hybrid loss function is formulated as:

D i c e B C E = B C E + D i c e L o s s (1)

B C E = − 1 N ∑ i = 1 N [ y i × log ( p i ) + ( 1 − y i ) × log ( 1 − p i ) ] (2)

D i c e   L o s s = 1 − 2 × | X ∩ Y | + s m o o t h | X | + | Y | + s m o o t h (3)

Here, p i is the model output processed by the sigmoid function y i is the true label, N is the total number of samples, | X ∩ Y | represents the number of correct predicted pixels, representing the intersection between the prediction and the target, |X|, |Y| represents the total number of predicted pixels and the target respectively, and smooth is a smoothing parameter. 1 is used to prevent the denominator from being 0. Here, p i is the model output processed by the sigmoid function, y i is the true label, N is the total number of samples, | X ∩ Y | represents the number of correct predicted pixels, representing the intersection between the prediction and the target, |X|, |Y| represents the total number of predicted pixels and the target respectively, and smooth is a smoothing parameter. 1 is used to prevent the denominator from being 0.

In order to accurately evaluate the performance of the proposed model in image segmentation tasks, we adopt four widely recognized evaluation metrics, including mean intersection over Union (mIoU), Dice similarity coefficient, Precision and Recall. Each indicator is quantified according to the following criteria:

mIoU = 1 c ∑ i = 0 c TP TP + FP + FN

Dice = 2 TP 2 TP + FP + FN

Recall = TP TP + FN

Precision = TP TP + FP

The Dice Coefficient, which measures the ratio of the sum of twice the overlap between the model predicted segmentation and the true segmentation to the sum of the respective segmentation, is an important index to evaluate the segmentation accuracy. Mean Intersection over Union (mIoU) computes the ratio of intersection to union between the model predicted segmentation and the true segmentation, and is averaged over all classes. In addition, precision evaluates the proportion of samples predicted by the model to be in the positive class, while Recall measures the proportion of all samples that are actually in the positive class that the model predicted correctly. Together, these metrics constitute a comprehensive framework for evaluating how well a segmentation model identifies and delimits regions of interest in an image or ability. They not only quantify the accuracy of model performance, but also reflect the balance of the model in maintaining details and reducing misjudgments. Through these metrics, we are able to comprehensively investigate the adaptability and accuracy of the model in the medical image segmentation task. In this framework, a true positive (TP) means that a positive sample is correctly identified, a false positive (FP) means that a negative sample is incorrectly identified as a positive sample, and a false negative (FN) means that a positive sample is incorrectly identified as a negative sample. These definitions help to better understand the performance of the model and the direction of optimization.

In this part of the paper, we present a meticulous evaluation of the newly developed DDPPT-Net by comparing it with five current leading methods, including U-Net, ResUNet++ [19], DoubleU-Net, PraNet, and DDANet. This comparison covers five different polyp segmentation datasets and is performed according to four core evaluation metrics.