Cervical cancer is a major problem to the health of women in the world as it is the number one killer among women when it comes to cancer-related deaths in 37 countries and there is a disturbing trend towards an increase in young people contracting the disease [1][2]. Timely screening and proper diagnosis of cervical cancer is very important to enhance prognosis since it can be cured entirely when diagnosed and treated early enough [3]. The prevention and control of cervical cancer mostly depends on pathological screening. Papanicolaou (Pap) smear and ThinPrep Cytologic Test (TCT) are critical tools in cervical cancer screening and they are aimed at determining and categorizing squamous intraepithelial lesions, allowing detection and treatment of pre-invasive stages of the disease before it advances to invasive phases [4]. Precision in the classification of cervical cytopathology images is an important step in the procedure. To ascertain the level of abnormality and evaluate whether the cancerous process is present, pathologists examine smear specimens using optical microscopy, which requires great skill and scrupulous examination to identify differences between normal and abnormal cells [5].

Classifying cells is a basic procedure in diagnosing diseases and biomedical studies especially during pathological evaluation on such diseases as cervical cancer, where correct characterization of malignancy grade of the cells is paramount. Cervical cell grading can be used to diagnose samples using morphological evaluation of cells present in samples to diagnose them in different grades including benign and atypical squamous cells through high-grade squamous intraepithelial lesions. Precise classifying is an essential basis that will be used to direct clinicians in coming up with proper treatment measures: lesions of low grade usually require no more than usual surveillance, but lesions of high grade should be subjected to urgent colposcopy or aggressive medical management. The accurate cell classification will allow high-risk patients to be treated in time and avoid unnecessary treatment of low-risk patients, which greatly contributes to the effective distribution of public health resources.

Conventional cell classification strategies have been based largely on manual identification of morphological properties combined with traditional machine learning methods like the support vector machines (SVM), decision trees, etc., to diagnose. Nonetheless, they require strong reliance on the quality of feature engineering and are prone to being unable to learn more intricate, nonlinear features specific to cellular morphology. The last few years have seen an explosion in the development of deep learning technologies and computer vision, which allows convolutional neural network-based models to make substantial progress in medical image analysis. These developments provide a powerful tool in the automated screening of cervical cancer that promises to be much more efficient and objective in clinical diagnosis [6].

Although the current breakthrough in deep learning has been impressive, cervical cells in clinical samples are usually presented as densely packed and with substantial overlap [7]. At present, the majority of the work related to overlapping cell classification in the field of deep learning is divided into two major technical directions. The one that has gained a leading position is segmentation-based single-cell classification. Current studies and procedures to train models generally use non-overlapping images of individual cells taken out of high quality or sparse distributions of samples, allowing models to learn different morphological properties of individual cells efficiently. Although this allows the model to learn clear properties of the morphology, it does not take into account the fact that there are many overlapping cells in the real diagnostic setting, and how these can confuse classifiers. The pixel level features contained in overlapping areas are a composite of several cell instances, which makes it difficult to assign the features correctly to a certain cell with the help of classifiers. Therefore, important morphological properties of malignant cells can be hidden, which makes it more difficult to determine and identify such cells accurately and finally reduces the accuracy of their classification. The restriction considerably undermines the utility of deep learning models in clinical practice.

The purpose of this paper is to explore the influence of overlapping cells on cell classification performance in order to show that cell overlap presents a major problem to existing cell classification techniques. The research applied an advanced classification model, namely, YOLO11l-cls. Training inputs were single-cell images that had gone through segmentation and preprocessing to enable the model to learn how to correctly identify characteristics of typical single-cell lesions. In order to measure how much cell overlap impacts the effect, two different test sets were used to assess the performance of the model:

Construction of comparison datasets: To obtain the actual cervical pathological images, two test sets were created, one with overlapping cells and the other one with clear, standardized non-overlapping cells that were extracted out of overlapping populations using accurate segmentation methods.Quantitative evaluation and conclusion: The pre-trained YOLO11l-cls classifier was optimized and then evaluated using the two image sets. The negative effect of cell overlap on classification accuracy was shown by comparing the important performance measures, such as area under the curve (AUC), accuracy and precision.

When applying the well-performing classifier in the practical clinical conditions, we have noticed a significant fact that when the model is used on a dataset with overlapping cells its performance drops significantly. Nevertheless, after eliminating overlapping cell cases in the dataset, the classification accuracy and precision recover immediately and substantially.

The current paper will be organized as follows: In Section 2, I review the literature concerning the automated classification of cervical cells and its solutions to cope with cell overlap. In Section 3, I focus on the origin of the dataset, the segmentation procedure, and how the two comparative test sets were created. The architecture of the core classifier, YOLO11l-cls, and the overall evaluation framework are described in Section 4. The experimental setup, training hyperparameters, performance metrics and quantitative analysis of results are discussed in Section 5. Conclusively, Section 6 provides a summary of the negative effect of cell overlap on classification performance and concludes the study.

The primary objective of automated analysis in cervical cancer cytopathology imaging is to achieve accurate classification of cells as either malignant or benign. Early computer-aided diagnosis (CAD) systems predominantly relied on machine learning approaches, including SVM [8], k-nearest neighbors (KNN) [9][10], and random forests [11]. However, these methods exhibit a high dependence on handcrafted feature extraction and demonstrate limited generalization capability.

A significant proportion of high-performance cervical cell classification studies concentrate on achieving optimal diagnostic accuracy using ideal, feature-complete single-cell datasets, typically circumventing the challenges posed by overlapping morphological features through meticulous manual annotation. Xu et al. [12] developed a deep learning framework based on an enhanced Faster R-CNN architecture integrated with a shallow feature enhancement network and a generative adversarial network for cervical cell image classification. Their model achieved a maximum accuracy of 99.81% and an overall average accuracy of 89.4% across the SIPaKMeD and Herlev datasets. Rahaman et al. [13] introduced DeepCervix, an automated classification framework leveraging Hybrid Deep Feature Fusion (HDFF) technology, which demonstrated high classification accuracy in 2-class, 3-class, 5-class, and 7-class configurations on two publicly available datasets. Chen et al. [14] proposed an automated cervical cell screening system designed to support subsequent clinical diagnosis, with the CompactVGG classification model serving as its core component. The system achieved classification accuracies of 97.8% and 94.81% on the SIPaKMeD and Herlev datasets, respectively. Maurya et al. [15] presented a hybrid deep learning framework that employed transfer learning with Long Short-Term Memory (LSTM) and CNN models, as well as an ensemble approach combining Vision Transformer and CNN, trained and evaluated on the SIPaKMeD dataset, yielding high classification performance. Zhang et al. [16] proposed A2SDNet121, a deep convolutional neural network aimed at addressing the issue of low accuracy in cervical cell classification. The model was applied to binary and multi-class classification tasks on the Herlev and SIPaKMeD datasets, achieving an accuracy of up to 99%. Deepa et al. [17] adopted a two-stage approach involving segmentation followed by classification, utilizing a CNN model to classify overlapping cervical smear cells. Overlapping cells were segmented using the midpoint algorithm, and the cropped images of individual cells were then fed into the model for distinguishing between normal and abnormal cells, resulting in a classification accuracy of 96%.

To address the challenges posed by cell clustering and overlapping, researchers have primarily employed object detection or instance segmentation techniques. Furthermore, only a limited number of studies have attempted to classify overlapping cells directly without relying on prior segmentation. Mulmule et al. [18] introduced a fully automated transfer learning framework that extracts 128 × 128 pixel patches containing multiple cervical cells from high-resolution images, enabling direct classification of overlapping cells under microscopic examination with a reported accuracy of 99.86%. Nguyen et al. [19] proposed a method for segmenting and classifying overlapping cervical cells based on semi-supervised learning, utilizing the SemiFixMatch model and achieving competitive performance in this task. In another study, Mulmule et al. [20] fine-tuned and retrained AlexNet along with pre-trained models from ImageNet and Places365 on the CERVIX93 dataset using cervical cancer enhancement data, evaluating model performance across several metrics. Their results demonstrated that the AlexNet model achieved the highest predictive performance in cervical cancer prediction, with an accuracy rate of 99.03%.

The main related work of cervical cell image classification are summarized in Table 1.

Table 1. Work related to cervical cell classification.

This study was conducted using 2504 overlapping cervical cell images obtained from the Fourth Central Hospital of Baoding, and received formal approval from the hospital’s Institutional Review Board prior to commencement.

The pathological images of cervical cancer cells used in the current research have been obtained based on the data set reported in the paper by Yu and Feng et al. [21] which originated at the Fourth Central Hospital of Baoding, China. This dataset consists of 2504 overlapping cervical cell images. In order to make the classification model learn correct morphologic characteristics of cellular lesions the study uses high-quality and non-overlapping cell instances as inputs to the classifier when training. To achieve this, we use the developed Cellpose-SAM [22] deep learning based instance segmentation model as the main method in constructing the datasets. Cellpose-SAM is a segmentation model that combines the accurate instance separation approach of Cellpose [23] with the strong general feature learning properties of SAM, which has shown better generalization results and segmentation quality when morphologically complex and densely packed cells are processed. It allows extracting individual cell instances of the original images reliably, thus forming a sound basis of model training.

Figure 1 and Figure 2 show the areas of overlap between normal and abnormal cervical tissues in this dataset as well as non-overlapping cell maps produced by instance segmentation, respectively. The normal cervical cells have a central nucleus and a cytoplasm which can be easily differentiated against the background, where the nuclear region is less than the cytoplasmic region. Conversely, the cancerous cervical cells tend to have fibrous cytoplasm and enlarged nuclei.

From an initial dataset of 2504 images, 433 images exhibiting cell overlap were selected. These images were segmented using the CellPose-SAM model, generating 4624 non-overlapping cell samples. The segmented samples were classified based on their cellular content: a sample was labeled as abnormal if it contained any abnormal cells, and normal otherwise. Consequently, 3130 normal and 1494 abnormal samples were obtained. The detailed dataset composition is summarized in Table 2.

Figure 1. Images of normal cervical cells. (a) represents the original image; (b) displays the ground truth derived from manual segmentation; (c) presents the instance map containing six individually cropped cells based on segmentation results.

Figure 2. Images of abnormal cervical cells. (a) represents the original image; (b) displays the ground truth derived from manual segmentation; (c) presents the instance map containing six individually cropped cells based on segmentation results.

Table 2. Summary of data characteristics in this study.

In this study, the YOLO11l-cls model was employed to systematically investigate and quantify the adverse effects of cellular overlap on cervical cytology classification through comparative experiments. Trained exclusively on segmented, morphologically distinct single-cell data and evaluated on an ideal baseline of non-overlapping cell instances (Test Set 1), the model demonstrated near-optimal theoretical diagnostic accuracy. This was evidenced by an AUC of 0.9959, an F1-score of 0.9255, and a remarkably high precision of 0.9941, underscoring its robust feature extraction capabilities and high diagnostic consistency.

However, when the trained classifier was applied to a mixture of overlapping and non-overlapping cells (Test Set 2), significant degradations across all evaluation metrics were observed. Compared to the baseline, the AUC decreased to 0.8816, representing an absolute reduction of 0.1143 (an approximate 11.5% drop). Concurrently, precision fell by 0.2573 (from 0.9941 to 0.7368), recall decreased by 0.1740 (from 0.9108 to 0.7368), and the F1-score dropped by 0.1446 (from 0.9255 to 0.7809). Crucially, the overall classification accuracy was reduced to 0.8305.

This performance deterioration became most acute in Test Set 3, which consists exclusively of overlapping cell images. Under these extreme conditions, the model’s diagnostic capacity suffered a severe bottleneck, with the AUC plummeting to 0.6602, accompanied by a sharp decline in accuracy (0.6064), precision (0.6420), recall (0.5361), and an F1-score of 0.5843.

This stepwise performance decay robustly confirms that cellular overlap poses a critical barrier to automated models achieving their theoretical performance limits. The pronounced drops in precision and recall reveal that overlapping cell structures severely impair the network’s ability to recognize and interpret pivotal diagnostic features, particularly nuclear morphology. Automated cervical cytology classification systems remain heavily reliant on high-quality, morphologically intact cellular inputs. Therefore, it is imperative to address the challenges of instance segmentation and feature extraction in the presence of overlapping cells to enhance the real-world applicability and clinical utility of these models.

This research was supported by the S&T Program of Hebei under project No. 22377774D. This grant ensured the smooth implementation of this research, and we would like to express our heartfelt gratitude.

*These authors contributed equally to this work.

^#Corresponding author.