This study is anchored on the theory of Feature Learning (Representation Learning) and the Transformer Attention Mechanism, which collectively explain the effectiveness of hybrid CNN-ViT architectures in medical image classification.

Feature learning enables deep neural networks to automatically discover discriminative representations directly from raw image data without manual feature engineering [14]. Through hierarchical learning, lower network layers capture simple visual patterns such as edges, textures, and color gradients, while deeper layers learn more complex and abstract lesion characteristics. In skin disease classification, feature learning facilitates the extraction of clinically relevant visual cues necessary for differentiating among dermatological conditions.

The Transformer Attention Mechanism introduced by [10] provides the theoretical foundation for Vision Transformers. Unlike conventional CNNs that focus primarily on local receptive fields, self-attention enables the model to capture long-range spatial dependencies and contextual relationships across the entire image. This capability is particularly important in dermatology, where lesion symmetry, shape irregularity, and global structural characteristics are critical diagnostic indicators.

The integration of CNN and ViT architectures combines localized feature extraction with global contextual understanding, resulting in a more comprehensive representation of skin lesions and improved classification performance.

The integration of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) represents a promising advancement in optimizing skin disease classification. Several studies have explored hybrid models combining these two architectures to enhance diagnostic accuracy by leveraging CNNs’ local feature extraction capabilities and ViTs’ ability to capture global dependencies. Below is a focused review of the most relevant works contributing to this optimization.

[15] developed a multi-task CNN for skin lesion classification, particularly melanoma detection. The model integrated both segmentation and classification tasks, achieving high performance in delineating lesion boundaries and classifying melanoma. However, the model was limited by the lack of diverse skin types in the training dataset, which restricted its generalizability. The study highlighted that hybrid models, incorporating both CNNs and ViTs, could improve both local feature extraction and global contextual understanding, which are crucial for accurate melanoma detection.

[15] developed a deep CNN for skin cancer detection, achieving dermatologist-level performance in melanoma and basal cell carcinoma diagnoses. While the CNN model demonstrated high accuracy, it was limited by the absence of clinical context such as patient history and lesion evolution. Esteva et al.’s findings suggest that a hybrid CNN-ViT model would address this limitation by combining CNNs’ ability to extract local features with ViTs’ capacity to model global relationships, enabling the integration of non-visual clinical data for improved diagnostic performance.

[3] evaluated the effectiveness of CNNs in detecting skin diseases like melanoma and psoriasis. The study found that while CNNs outperformed traditional methods, they struggled with modeling global dependencies and integrating clinical data. These limitations emphasize the potential of CNN-ViT hybrid models, which can incorporate both image features and clinical metadata, improving diagnostic accuracy and generalizability across various dermatological conditions.

[8] compared a CNN’s diagnostic performance in melanoma recognition to that of 58 dermatologists, achieving 95% sensitivity. Despite this, the model’s performance was limited by the lack of clinical metadata, underscoring the importance of integrating both image data and patient-specific information. The study advocates for hybrid CNN-ViT models, which combine local feature extraction from CNNs with global context modeling and clinical context integration from ViTs, to improve diagnostic consistency and applicability in real-world settings.

[12] proposed TransUNet, a hybrid model combining CNNs and ViTs for skin lesion segmentation. The model achieved superior performance in segmentation accuracy, capturing both local and global features. The study demonstrated that hybrid CNN-ViT models could significantly enhance skin disease classification by capturing fine-grained local features from CNNs and modeling long-range dependencies through ViTs. However, the computational complexity of the hybrid architecture remains a challenge, and future work needs to focus on optimizing this trade-off for clinical deployment.

[11] compared Vision Transformers (ViTs) with CNNs for skin lesion classification, finding that ViTs outperformed CNNs, especially in identifying malignant lesions. While ViTs showed promise, performance inconsistencies across different lesion types were noted. The study advocates for combining CNNs’ localized feature learning with ViTs’ global attention mechanisms to improve generalization and performance across diverse skin types and lesion presentations, supporting the development of hybrid CNN-ViT models for more robust skin disease classification.

[16] developed a CNN-based real-time skin disease diagnostic system that achieved high accuracy but was limited by its reliance solely on image data and the absence of clinical metadata. Their study suggests that hybrid models, integrating CNNs with other techniques such as ViTs, could improve contextual understanding and enhance system robustness in real-world clinical applications. The combination of CNN and ViT can overcome the limitations of image-only models by integrating global attention and contextual analysis.

[17] developed a multimodal CNN-based model that integrated clinical data with image analysis, improving skin cancer diagnosis. The hybrid model outperformed image-only CNNs, achieving higher sensitivity and diagnostic accuracy. The authors emphasize the need for hybrid models that combine visual and non-visual data to improve real-world clinical applicability. A CNN-ViT hybrid model could provide a better approach by capturing both local and global features and integrating clinical metadata for a more holistic and accurate diagnosis.

[15] developed a CNN model for psoriasis classification with 87% accuracy but noted limited generalizability due to dataset diversity. The study suggests that hybrid CNN-ViT models could improve generalizability by combining CNNs’ efficiency in feature extraction with ViTs’ ability to model global dependencies and contextual relationships, making the model more applicable to diverse patient populations and various dermatological conditions. [6] proposed a hybrid CNN-SVM model to improve skin lesion classification. While the model achieved an accuracy of 85%, it was computationally complex and lacked clinical data integration. The study highlighted the need for models that balance performance with computational efficiency, suggesting that a CNN-ViT hybrid model could improve both performance and interpretability by combining CNNs’ ability to extract local features with ViTs’ capacity for global attention mechanisms. [18] compared a CNN-based AI system with experienced human dermatologists in detecting malignant and benign skin lesions. The CNN model achieved promising results, but its dependency on high-quality curated datasets limited its robustness in real-world scenarios. The study suggests that hybrid CNN-ViT models could enhance resilience to inconsistencies in visual data, improving accuracy across varied clinical conditions by incorporating global contextual understanding from ViTs.

The development process adhered to the DSDM framework, structured into several key stages to guide the implementation of the CNN-ViT skin disease classification system.

This initial phase involved confirming the suitability of the dataset (HAM10000), verifying the computational resources required for training the model, and assessing the model’s potential for deployment in clinical settings. A clear understanding of user needs, clinical relevance, and project constraints was established to guide the design and ensure the system meets healthcare requirements. The iterative process began with the implementation of the preprocessing pipeline, CNN-ViT hybrid modules, and the fusion of features from CNN and ViT for classification. These were refined across multiple experimental cycles to enhance performance. The final deployment involved integrating the model with a user-friendly interface, validating its performance through metrics such as accuracy, precision, and recall, and providing explainability outputs for clinical interpretation.

The HAM10000 dataset was utilized for this study. The dataset contains 10,015 dermoscopic images representing seven diagnostic categories: Actinic Keratoses (AKIEC), Basal Cell Carcinoma (BCC), Benign Keratosis-like Lesions (BKL), Dermatofibroma (DF), Melanoma (MEL), Melanocytic Nevi (NV), and Vascular Lesions (VASC). To prevent data leakage, dataset partitioning was performed prior to augmentation. Images associated with the same lesion were restricted to a single partition. The dataset was divided into training (70%), validation (15%), and testing (15%) subsets. To address class imbalance and improve model generalization, augmentation techniques including horizontal flipping, vertical flipping, random rotation, zooming, brightness adjustment, and image translation were applied exclusively to the training subset. The balanced class distributions presented in Table 1 represent the post-augmentation dataset used during model training.

This study proposes a Hybrid CNN-ViT Model to enhance skin disease classification by leveraging both local feature extraction (via CNNs) and global context modeling (via ViTs). The model is a multi-stage pipeline consisting of: Input Layer, CNN Feature Extractor, Transformer Encoder (ViTs), Fusion Module, and Softmax Classifier.

The operational flow of the skin disease classification system is outlined as follows:

Input Module: Users upload images, which are then preprocessed for standardization (resizing, normalization, and augmentation).

Feature Extraction:

CNN Backbone: Extracts low-level features such as textures and edges.

ViTEncoder: Divides the image into patches, processes them with self-attention to capture global relationships, and outputs high-level semantic features.

Fusion Module: Combines the local CNN features and global ViT features to enhance feature representation.

Classification: The fused features are passed through a softmax classifier to predict the skin disease category.

Training: The model is trained on 70% of the dataset (HAM10000), with augmentation applied to improve generalization. Training proceeds through multiple epochs with backpropagation and gradient descent, and early stopping is applied to prevent overfitting.

Validation: A separate 15% of the dataset is used to validate the model, providing insights into generalization. Hyperparameters are adjusted based on validation loss to improve model performance.

Testing: The final 15% of the dataset is used to test the model’s performance on unseen data. Performance metrics (accuracy, precision, recall, F1-score) are computed to evaluate the model’s effectiveness.

Feature Extraction

CNN Module: Extracts fine-grained local features such as edges, textures, and color variations, essential for differentiating skin diseases.

ViTModule: Models global dependencies across the entire image, capturing contextual relationships that CNNs may miss.

Fusion Layer

The fusion layer merges the outputs of the CNN and ViT modules, creating a more comprehensive feature vector that incorporates both local and global features.

SoftmaxClassifier

The final decision-making stage uses the fused feature vector to predict the skin disease class, ensuring that the model outputs probabilities for each class and provides interpretable results.

Performance Evaluation

The model’s performance is evaluated using standard classification metrics (accuracy, precision, recall, F1-score) on the test set. The system’s ability to handle environmental variability, such as lighting changes and image quality, is also assessed, with results compared against traditional CNN models.

Figure 1.The framework in the methodology.

The proposed model combines EfficientNet-B0 and Vision Transformer (ViT-Base/16) architectures. The CNN branch employs EfficientNet-B0 for extracting local image features such as texture, color variation, lesion borders, and structural patterns. The ViT branch processes image patches through self-attention mechanisms to capture long-range spatial relationships and global contextual information.

Model configuration parameters include

The extracted CNN and ViT features were concatenated through a fusion layer and subsequently passed through fully connected layers and a Softmax classifier for final disease prediction.

2.6.1. Convolutional Layer: Extracting Local Features

The CNN module starts with a convolutional layer, which applies a set of trainable filters (kernels) to detect patterns such as edges, textures, and shapes from the input image. The convolution operation between an input image I ( x , y ) and a filter K ( x , y ) (kernel) is mathematically expressed as:

1) I ( i + m , j + n ) is the input image at position ( i + m , j + n ) .

2) K ( m , n ) is the 3x3 Kernel (Filter).

3) C = bias.

4) The summation runs over the 3 × 3 kernel window.

Therefore, for multiple filters (F_m), the convolution is expressed thus;

where

K_k = kernel for filter k and.

C_k = bias term.

Therefore, the feature extraction within the CNN layer is handled using Equation (3)

where

X = Input Image.

Flatten converts the spatial feature into a 1D vector using Equation (3).

2.6.2. Transformer Encoder Module: Global Feature Learning

Given an input image X (with height H, width C, and channels C), ViT splits the image into patches of size P × P, giving N patches. This can be mathematically expressed thus;

where;

N = total number of patches.

P= patch size.

H = Image Height.

W = Image Width.

Each patch is flattened into a vector and mapped to a fixed-size embedding using a linear transformation:

where;

X = Flattened Patch pixels.

W_e = Learnable weight matrix.

b_e = Bias term.

Z = Embedded feature representation of the patch.

Self-Attention Mechanism:Each transfer Encoder layer applies Multi-Head Self-Attention (MHSA)

where:

Q = W_QZ, K = W_KZ, V= W_VZ are the Query(Q), Key(K), and Value(V) Matrices.

F_d = feature dimension.

The softmax function ensures that the values sum to 1.

This step allows the Transformer Encoder module to learn global relationships across the image. Furthermore, the output from the self-attention layer is passed through a feedforward network. Therefore, the feature extraction within the Transformer Encoder is handled using Equation (7).

where:

X_patch = Image divided into patches

F_ViT = Transformer Encoder layer.

E = Patch embedding (Projection Matrix).

MHSA = Multi-Head Self-Attention, capturing global relationships.

MLP = Multi-Layer Perceptron, refining extracted features.

The Fusion Module of the model is responsible for integrating feature representations extracted from both the CNN and Vision Transformer (ViT) modules to enhance skin disease classification. Therefore, given an input image X, CNN processes the image and extract the local features and it passed into the transformer Encoder Module which processes the input patches.

Hence, the Skin Disease Classification Model (SDCM) is obtained by concatenating CNN features (Equation (1)) and ViT features (Equation (7))

where

W_FC and b_FC are the trainable weights and biases.

F_CNN = CNN Feature Layer.

F_ViT = Tranformer Encoder Layer.

|| = Concatenation parameter.

Performance evaluation was conducted using Accuracy, Precision, Recall, and F1-score. For the seven-class classification problem, Precision, Recall, and F1-score were computed using weighted averaging to account for class imbalance.

Accuracy measures the proportion of correctly classified samples, while Precision evaluates the proportion of correctly predicted positive samples. Recall measures the ability of the model to identify actual positive samples, and F1-score provides a harmonic balance between Precision and Recall. Table 2 aptly highlights the performance of the SDCS.

Table 2.Performance evaluation of SDCS.

Confusion Matrix: The confusion matrix for SDCS (Figure 2) shows strong classification performance for most diseases, with a few instances of misclassification, particularly between diseases with similar visual patterns (e.g., melanoma and melanocytic nevi). This indicates the need for further refinement in differentiating these diseases, which may require additional data augmentation or model tuning.

Figure 2. Confusion matrix of SDCS.

Result 3:

A comparative performance evaluation between the hybrid SDCS model and the CNN-only model developed by Agarwal and Godavarthi (2023) showed that the SDCS outperforms the CNN model across all metrics. The SDCS achieved a +14.35% improvement in precision, +12.68% improvement in accuracy, and +17.16% improvement in recall compared to the CNN-only model.

This comparison highlights the effectiveness of integrating CNN and ViT models, combining the strengths of both local feature extraction (CNN) and global context modeling (ViT) for improved classification of skin diseases. Table 3 shows the performance comparison of the study’s model (SDCS) against the Agarwal and Godavarthi model. Graphical highlights in Figure 3.

Table 3.Proposed SDCS performance comparison against the existing model.

Figure 3. Comparative analysis chart of the CNN model with SDCS.