Abstract

Background: Oral squamous cell carcinoma (OSCC) often arises from precancerous lesions and is frequently asymptomatic in its early stages. Oral exfoliative cytology is a minimally invasive screening tool for early detection, but diagnostic accuracy remains suboptimal due to the subjective interpretation of cytological findings. Objective and quantitative evaluation of nuclear morphology may enhance diagnostic precision. Methods: A total of 9593 cytological images of squamous epithelial cells were analyzed from 85 cases, categorized into five diagnostic groups: negative for intraepithelial lesion or malignancy (NILM) without inflammation (NILM [inf−]), NILM with inflammation (NILM [inf+]), oral low-grade squamous intraepithelial lesion (OLSIL), oral high-grade squamous intraepithelial lesion (OHSIL), and OSCC. Nuclear area and hyperchromasia were quantified using image analysis software. Statistical analyses and kernel density estimation were performed to evaluate intergroup differences and distributions. Results: The OSCC group exhibited the largest nuclear area and the greatest hyperchromasia, with the broadest distribution of nuclear features. The OHSIL group also showed significant nuclear enlargement and heterogeneity. Positive correlations between nuclear area and hyperchromasia were observed in non-neoplastic groups, but they were weakened in the OHSIL and OSCC groups, reflecting tumor-associated nuclear pleomorphism. Most pairwise comparisons showed significant differences in nuclear features between groups. Conclusion: Quantitative assessment of nuclear area and hyperchromasia distinguishes between diagnostic categories in oral cytology. These findings suggest that morphometric image analysis may improve diagnostic accuracy and aid in the early detection and risk stratification of OSCC.

Keywords

Mouth Neoplasms, Exfoliative Cytology, Morphometry Cell Nucleus, Hyperchromasia, Image Processing

Share and Cite:

1. Introduction

Oral squamous cell carcinoma (OSCC) is a malignant epithelial tumor that can develop from precancerous epithelial dysplasia and frequently remains asymptomatic in its early stages [1]. In general, early detection is critical for the effective management of malignant tumors. Consequently, oral exfoliative cytology, valued for its minimal invasiveness, rapid turnaround, and ease of application, is widely used as a screening tool for oral mucosal lesions, including OSCC and epithelial dysplasia [1]-[3].

However, cytology alone does not provide a definitive diagnosis equivalent to that of tissue biopsy [4]. Previous studies investigating the diagnostic accuracy of oral cytology have reported an overall accuracy of approximately 70% - 80% [5] [6]. Therefore, quality control in cytological diagnosis is essential, and efforts such as evaluating diagnostic accuracy rates and analyzing discrepancies between cytological and histopathological findings have been undertaken [6] [7]. Some authors have argued that cytology is insufficient as a substitute for biopsy [8]. Thus, obtaining further evidence regarding oral cytology is indispensable.

The cytological evaluation of oral exfoliative samples is based on a variety of cellular findings, among which nuclear characteristics are considered particularly important for improving diagnostic accuracy [9]. Therefore, to improve diagnostic accuracy, it is necessary to clarify the nuclear characteristics according to cytological diagnostic categories; however, current knowledge regarding cytological evaluation of the oral mucosa remains limited.

In this study, a design was adopted to reflect the overall cytomorphological tendencies of the specimens as encountered in routine cytological screening, rather than to highlight only the features of diagnostically evident or atypical cells commonly analyzed in previous studies. The aim of this study was to clarify the nuclear features across different cytological diagnostic categories based on a whole-sample random sampling approach.

2. Materials and Methods

2.1. Subjects

This study included 85 cases in which oral exfoliative cytology of the tongue mucosa was performed at the Nihon University School of Dentistry at Matsudo Hospital. The cases were classified into the following groups: negative for intraepithelial lesion or malignancy (NILM) without inflammation (NILM [inf−]), 19 cases (cases 1 - 19); NILM with inflammation (NILM [inf+]), 19 cases (cases 20 - 38); oral low-grade squamous intraepithelial lesion or oral low-grade dysplasia (OLSIL), 13 cases (cases 39 - 51); oral high-grade squamous intraepithelial lesion or oral high-grade dysplasia (OHSIL), 15 cases (cases 52 - 66); and OSCC, 19 cases (cases 67 - 85).

2.2. Cytological Images for Analysis

A total of 9593 cytological images of squamous epithelial cells, randomly captured at ×40 objective magnification from Papanicolaou-stained slides, were used for analysis (OOPID-v1, Open Oral Pathological Images Dataset, URL: https://www.oopid.jp/). The distribution of images was as follows: NILM [inf−], 1939 images; NILM [inf+], 2148 images; OLSIL, 1467 images; OHSIL, 1731 images; and OSCC, 2308 images. Representative cytological features are shown in Figure 1.

Figure 1. Representative cytological features of analysis. (a) Negative for intraepithelial lesion or malignancy without inflammation, Pap stain, 400× magnification. (b) Negative for intraepithelial lesion or malignancy with inflammation, Pap stain, 400× magnification. (c) Oral low-grade squamous intraepithelial lesion or oral low-grade dysplasia, Pap stain, 400× magnification. (d) Oral high-grade squamous intraepithelial lesion or oral high-grade dysplasia, Pap stain, 400× magnification. (e) Oral squamous cell carcinoma, Pap stain, 400× magnification.

2.3. Image Selection Criteria

For nuclear analysis, a board certified oral cytopathologist(certified by the Japanese Society of Clinical Cytology) selected images based on the following exclusion criteria:

1) Images lacking nuclei.

2) Images with overlapping nuclei and bacterial colonies.

3) Images showing overlapping nuclei.

4) Images with overlapping nuclei and cytoplasmic granules.

5) Images with marked degenerative changes.

6) Images with indistinct or poorly defined nuclear borders.

2.4. Region of Interest (ROI) Setting

Using image editing software (Photoshop 2024, Adobe, Tokyo, Japan), the region of interest (ROI) was defined for the nuclei of target cells using the Quick Selection Tool (brush size: 9), and corresponding binarized images were subsequently created.

2.5. Image Analysis

For each cytological image, the nucleus of the centrally located cell was selected as the target ROI. Thus, one nucleus was measured per image. Image analysis software (ImageJ 1.54j, NIH, Bethesda, MD, USA) was used to calculate nuclear area and hyperchromasia from the binary and original images. In this study, nuclear area was defined as the ROI pixel count converted to μm². In this study, hyperchromasia was defined as nuclear staining intensity, representing the degree of nuclear chromatin coloration. Hyperchromasia was quantified as the mean gray value of all pixels within the ROI on an 8-bit grayscale (0 - 255) in ImageJ, where 0 corresponds to black (strongest staining) and 255 to white (no staining). Accordingly, lower mean gray values indicate higher degrees of nuclear hyperchromasia (darker staining), reflecting stronger hematoxylin affinity.

2.6. Grouping of Analysis Results

The analysis results were compared across five groups: NILM [inf−], NILM [inf+], OLSIL, OHSIL, and OSCC.

2.7. Statistical Analysis and Data Visualization

All statistical analyses were conducted using R software (version 4.3.1, R Core Team, 2023). Normality was assessed for each cytological category using the Shapiro-Wilk test. Spearman’s rank correlation coefficients were calculated to evaluate the relationship between nuclear area and hyperchromasia within each diagnostic category. Two-dimensional kernel density estimation was performed to visualize the distribution of nuclear area and hyperchromasia, and the results are presented as contour plots indicating relative densities (100%, 90%, 70%, 50%, 30%, and 10%). The kernel density estimation was performed using the MASS package in R (function kde2d), and contour levels were defined as proportions of the maximum density. Homogeneity of variance among the five groups was assessed using the Fligner-Killeen test. When both non-normality and heterogeneity of variance were present, pairwise permutation tests were conducted for multiple comparisons using the rcompanion package (pairwisePermutationTest, method = “holm”). Effect size (Cliff’s δ) and its 95% confidence interval (C.I.) were calculated. The 95% confidence intervals were estimated by nonparametric bootstrapping (2000 resamples). A significance level of 0.05 was applied to all statistical tests.

2.8. Ethical Considerations

This study was conducted with the approval of the Ethics Committee of the Nihon University School of Dentistry at Matsudo (Approval Numbers: EC21-19-035-1 and EC21-009), in accordance with the Declaration of Helsinki. All data were anonymized, and personal information was protected.

2.9. Computational Environment

All image processing and statistical analyses were performed on a personal computer equipped with an AMD Ryzen 5 3600 processor, 32 GB of RAM, and an NVIDIA GeForce RTX 4060 graphics card (8 GB VRAM).

3. Results

The results by cytological diagnosis category are summarized in Table 1. The number and selection rate of selected cells relative to the total number of target cells was as follows: NILM [inf−], 1766/1939 (91.1%); NILM [inf+], 1967/2148 (91.6%); OLSIL, 1272/1467 (86.7%); OHSIL, 1529/1731 (88.3%); and OSCC, 1984/2308 (86.0%).

The median nuclear area [interquartile range] for each group was as follows: NILM [inf−], 51.0 [28.3] μm²; NILM [inf+], 52.6 [26.3] μm²; OLSIL, 47.6 [27.1] μm²; OHSIL, 60.7 [35.0] μm²; OSCC, 67.2 [54.0] μm².

The median hyperchromasia [interquartile range] for each group was as follows: NILM [inf−], 98.3 [39.1], NILM [inf+], 84.7 [40.9], OLSIL, 74.1 [42.1], OHSIL, 85.5 [46.1], and OSCC, 72.5 [39.0].

The results of the kernel density estimation are shown in Figure 2. Compared with NILM [inf−], all other categories (NILM [inf+], OLSIL, OHSIL, OSCC) exhibited distinct distribution patterns, with OSCC showing the most pronounced difference.

The comparative distribution patterns among the diagnostic groups relative to NILM [inf−] are shown in Figure 2. The contour plots generated by kernel density estimation show a progressive shift and broadening of the nuclear feature distributions from NILM [inf+] through OLSIL and OHSIL to OSCC. Notably, the OSCC group displayed a markedly wider dispersion along both the nuclear area and hyperchromasia axes, reflecting the increased heterogeneity associated with malignant transformation.

Spearman’s correlation coefficients between nuclear area and hyperchromasia were as follows: NILM [inf−], 0.546; NILM [inf+], 0.561; OLSIL, 0.535; OHSIL, 0.437; and OSCC, 0.293.

Shapiro–Wilk normality tests for each cytological group indicated that both nuclear area and hyperchromasia deviated from normal distributions (nuclear area: W = 0.769 - 0.960, p < 0.001; hyperchromasia: W = 0.971 - 0.997, p < 0.001). The Fligner-Killeen test for homogeneity of variance among the five groups indicated that homogeneity of variance was not present for either nuclear area (χ² = 763.92, p < 0.001) or hyperchromasia (χ² = 53.49, p < 0.001).

For nuclear area, significant differences were observed between all groups except between NILM [inf−] and OLSIL (Table 2). For hyperchromasia, most pairwise comparisons between groups showed significant differences, with the exceptions of NILM [inf+] vs. OHSIL and OLSIL vs. OSCC, for which no significant differences were observed (Table 2).

Table 1. Summary of nuclear analysis across cytological diagnostic categories.

Abbreviations: NILM, negative for intraepithelial lesion or malignancy; inf, inflammation; OLSIL, oral low-grade squamous intraepithelial lesion or oral low-grade dysplasia; OHSIL, oral high-grade squamous intraepithelial lesion or oral high-grade dysplasia; OSCC, oral squamous cell carcinoma; IQR, interquartile range.

Figure 2. Contour plots of two-dimensional kernel density estimation for nuclear area and hyperchromasia. The dotted line always represents NILM [inf−]. (a) NILM [inf+]. (solid orange line) vs NILM [inf−]. (dotted line) (b) OLSIL. (solid green line) vs NILM [inf−]. (dotted line) (c) OHSIL. (solid blue line) vs NILM [inf−]. (dotted line) (d) OSCC. (solid yellow line) vs NILM [inf−]. (dotted line).

Table 2. Pairwise statistical comparisons of nuclear area and hyperchromasia across cytological diagnostic categories.

Abbreviations: NILM, negative for intraepithelial lesion or malignancy; inf, inflammation; OLSIL, oral low-grade squamous intraepithelial lesion or oral low-grade dysplasia; OHSIL, oral high-grade squamous intraepithelial lesion or oral high-grade dysplasia; OSCC, oral squamous cell carcinoma; CI, confidence interval; Cliff’s δ, effect size.

4. Discussion

Nuclear findings in squamous epithelial cells observed on oral cytology can be detected in cells that have not undergone nuclear loss due to processes such as keratinization [10]. According to Zafer et al., nuclear features serve as important diagnostic indicators that reflect pathological conditions such as inflammation, epithelial dysplasia, and squamous cell carcinoma [9]. In practice, microscopic cytological examination demonstrates not only normal and atypical cells, but also diagnostically ambiguous and degenerative cells. In fact, the smears prepared on glass slides often contain a diverse population of cells, including reactive and regenerative cells, as well as cells exhibiting varying degrees of atypia from mild to severe. Therefore, in this study, to capture the overall picture of such diverse cytological presentations reflective of screening practice, nuclear characteristics were analyzed using a randomly sampled cytological image dataset.

In this study, nuclear area and hyperchromasia were quantified through image analysis, and their correlation coefficients were evaluated. In the NILM [inf−], NILM [inf+], and OLSIL groups, the correlation coefficients exceeded 0.5, indicating a moderately strong positive correlation. Assuming a relatively constant chromatin content within the nucleus, larger nuclei may exhibit lower chromatin density per unit area, resulting in a lighter nuclear appearance in the images. Previous studies have suggested that chromatin condensation plays a role in controlling nuclear size, although the underlying mechanisms remain unclear and warrant further investigation [11]. In contrast, OHSIL exhibited a lower correlation coefficient (below 0.5), and OSCC demonstrated an even lower value. Nuclear pleomorphism has been reported in oral cancers [12] [13]. Therefore, the findings in OSCC and OHSIL may reflect heterogeneity resulting from tumor-associated nuclear pleomorphism.

Two-dimensional kernel density estimation was used to visualize the distribution pattern of nuclear area and hyperchromasia. Kernel density estimation has been applied to various biomedical analyses, including the spatial visualization of T-cell distributions around cancer cells [14], and the identification of cell clusters in multiparameter flow cytometry [15] and hotspot mapping of brain metastases in breast cancer [16]. Given its ability to provide a smooth and continuous representation of density differences in multidimensional data, KDE is regarded as a particularly effective visualization method for comparative analyses across diagnostic categories and may also align with the broader trend of digital pathology and AI-based cytological analysis that increasingly leverages quantitative morphometric data [17] [18].

Previous studies have reported that nuclei in inflammatory specimens generally tend to exhibit hyperchromasia [19], and this overall tendency was also observed in our analysis. In inflammatory conditions, reparative cells with enlarged nucleus with pale staining chromatin are often observed [20]; however, reactive atypical cells associated with inflammation can also appear, and these cells tend to show hyperchromatic nuclei [21]. Therefore, the presence of such reactive atypical cells may have contributed to the difference in hyperchromasia between the NILM [inf−] and NILM [inf+] groups.

In the OLSIL group, compared with the NILM [inf−] group, hyperchromasia tended to be higher, whereas nuclear area did not show a significant difference. The findings regarding nuclear area were not consistent with those reported in previous studies [19]. This may reflect the present diagnostic approach, which places emphasis on hyperchromasia in the cytological assessment of OLSIL, and it may also be attributable in part to the fact that OSIL was subdivided into OLSIL and OHSIL in the present study. In contrast, the OHSIL group exhibited a significantly larger nuclear area than both NILM [inf−] and OLSIL groups. Furthermore, the OHSIL group demonstrated a broader distribution of nuclear features, which can be interpreted as a reflection of dysplasia-associated cellular pleomorphism, as discussed above. Pairwise statistical comparisons (Table 2) further supported these findings, showing that nuclear area significantly differentiated OHSIL from both NILM [inf−] and OLSIL, whereas hyperchromasia provided significant separation between NILM and OLSIL but did not distinguish OLSIL from OSCC. These results suggest that nuclear enlargement is a more consistent marker of progression, whereas hyperchromasia alone may be insufficient to differentiate dysplasia from carcinoma. The findings suggest that nuclear area may be an important factor that should be given greater weight in cytological judgment.

Previous studies utilizing morphometric computer-assisted image analysis have quantitatively confirmed nuclear enlargement in tumor cells of OSCC, thereby supporting the histopathological recognition of increased nuclear size as a key diagnostic indicator [22] [23]. Consistent with these findings, the present study demonstrated that the OSCC group exhibited not only the most extensive variation in nuclear parameters among all groups but also clear evidence of nuclear enlargement and hyperchromasia. These results are in line with those of Seto et al., who reported a significant increase in both nuclear size and chromatin density in OSCC using quantitative cytomorphometric analysis [19]. However, previous studies have not incorporated visual assessments of the distribution of nuclear size or hyperchromasia, such as those provided by kernel density estimation, which may offer a more comprehensive understanding of cellular heterogeneity. The present findings obtained through kernel density-based comparison further underscore the significance of both nuclear enlargement and hyperchromasia as critical cytological features for diagnosing OSCC. Moreover, the pronounced heterogeneity in these nuclear characteristics, as reflected in the broad distribution of measured values, is considered to reflect the morphological diversity typical of malignancy and may serve as an adjunctive diagnostic feature.

The present study has several limitations. The ROI selection was performed by a single observer, which may have introduced observer bias. In addition, multiple cells were analyzed per patient, and because the number of patients was limited, the statistical independence of observations could not be fully ensured, raising the possibility of pseudo-replication. The study focused exclusively on tongue specimens, as the tongue is the most common site of squamous cell carcinoma; however, this design limits the generalizability of the findings to other oral sites. Because hyperchromasia was analyzed, variability in slide preparation quality could potentially influence the measured results, although all specimens were stained under standardized conditions and overall quality was well maintained. The exclusion of images that did not meet predefined quality criteria may have introduced selection bias, possibly affecting the representativeness of the analyzed dataset. Furthermore, cytoplasmic features were not analyzed, even though cytoplasmic characteristics may influence the apparent nuclear color tone or staining intensity, which could have affected the interpretation of nuclear hyperchromasia. In addition, future studies will need to consider the quantification of the nuclear-to-cytoplasmic ratio, a classic cytological marker of malignancy. Finally, all analyses were conducted using specimens from a single dataset without an external validation cohort, and thus the generalizability of the findings to other populations or institutions remains uncertain. Future studies incorporating independent validation datasets, stratified sampling, and cytoplasmic feature quantification are warranted to enhance the robustness and clinical applicability of nuclear morphometric analysis.

In addition, the present findings should be considered as foundational data that could be embedded into practical diagnostic workflows. For example, nuclear area and hyperchromasia metrics may serve as objective adjuncts to conventional cytological interpretation, supporting cytotechnologists and pathologists in reducing subjectivity when assessing borderline or ambiguous cases. Furthermore, these quantitative features, along with various distributional characteristics, may be integrated into machine learning pipelines, where nuclear morphometry findings could potentially function as explanatory variables for training AI models aimed at the pre-screening of oral cytology specimens. Such integration would not only enhance diagnostic reproducibility, but also contribute to the development of AI-assisted systems that align with real-world screening practice. Indeed, surveys of current practice confirm that digital cytology and AI-based workflows are rapidly expanding, though challenges in standardization, interoperability, and whole-slide imaging quality remain important considerations for implementation in routine diagnostics [24] [25].

5. Conclusion

Significant differences were observed in the distribution patterns of nuclear area and hyperchromasia in squamous epithelial cells obtained from oral cytology specimens. These findings indicate that both nuclear area and hyperchromasia are key parameters in cytological evaluation, and that their distributional characteristics may further improve diagnostic accuracy. These findings suggest that evaluating nuclear area and hyperchromasia in oral exfoliative cytology could improve the early detection and risk stratification of OSCC.

Acknowledgments

This work was supported by the Japan Society for the Promotion of Science (JSPS) KAKENHI Grant Number JP21K11907.

Conflicts of Interest

The authors declare no conflict of interest.

References