In general, topic models are commonly used to extract topic information from text corpora. Traditional topic models use word cooccurrence information to identify topics, such as LSA [9] , PLSA [10] , etc. Especially LDA [11] , which has good generalization ability and extensibility, is widely used to discover hidden topics from text corpora. However, this approach is more suitable for long texts with rich topic matter [12] . Texts from the Internet are usually very short with limited contexts, and most of the above models and their variants do not take into account the specificity of short texts, and thus face serious data sparsity problems when applied to short texts.

A common solution to this problem is to extend short texts and aggregate them into lengthy pseudo-documents to enrich the representations of short texts, e.g., based on user groups [13] , based on synonyms [14] , based on the time of text publication [15] [16] , etc. Alternatively, it is extended using external knowledge, such as WordNet [17] , transfer learning with the help of long texts [18] , and so on. Other methods are modified in mathematical theory for short texts, such as the bi-term topic model BTM [19] , which uses bi-term co-occurrence information to model the corpus generation process.

GSDMM is an unsupervised topic model. It is solved in a similar way to LDA, except that it assumes that each document contains only one potential topic. Such an assumption is more in line with the actual situation of short texts such as comments, headlines, and other types of short texts. The model is based on the Dirichlet Multinominal Mixture Model (DMM), which generates documents and approximates the solution using collapsed Gibbs sampling. GSDMM implements clustering based on the two principles of completeness and consistency, and its performance is significantly better than that of K-means, HAC, and DMAFP. The model assumes that documents are generated according to a mixed polynomial distribution and that topics and documents correspond to one another.

However, these approaches are based on statistical knowledge and cannot solve the ambiguity problem. That is, multiple expressions of a certain meaning may be classified under different topics. Comprehension bias affects the accuracy of the results, so language models are gradually being used to explore more effective solution strategies.

Effective embeddings can help recognize information at the syntactic and semantic levels [20] and enhance the semantic consistency of topics in short texts, so embedding-based techniques have been gradually used to improve topic modeling in recent years. For example, Fan et al. [21] classified short texts based on word embeddings with bi-directional GRU, WETM [22] and JEA-LDA [23] discovered structural information about topics and words based on word embeddings, and Jia et al. [24] used sentence embeddings and CNN to cluster textual features. In addition, embedding information can be combined with methods for extending text. For example, VAETM [25] inferred potential representations of topic distributions of short texts by combining word embeddings with external knowledge; ETM [26] inferred topics by aggregating short texts into pseudo-documents using word embeddings and applying Markov Random Field Regularization models.

Language models facilitate the development of topic modeling [27] . BERTopic [28] is a topic model that performs clustering based on embeddings. It transforms documents into dense vector representations using the pre-trained language model SentenceBERT [29] , dimensionality reduction using the UMAP algorithm [30] , clustering with HDBSCAN [31] , and finally, applying a class-based variant of TF-IDF to extract the topic representation of each cluster. However, when dealing with a large number of short texts, sparse features introduce bias to clusters exclusively by the similarity of sentence embeddings, which makes it difficult for the language model in BERTopic to fully understand semantics.

Some studies introduce topic knowledge into advanced language models, where learning multi-dimensional features contributes to a comprehensive text understanding of the complex properties. Nguyen et al. [6] used LDA to obtain probabilistic topic vectors for comment texts and integrated sentence embeddings from SentenceBERT to identify topics; Agarwal et al. [7] used GA algorithm to improve WGSDMM for web service clustering; Peinelt et al. [8] proposed tBERT model to improve the performance of BERT in semantic similarity detection using topic models such as LDA and GSDMM. Liu et al. [32] combined GSDMM with SBERT to effectively extract topics and overall semantic features of short texts. Performs targeted classification tasks that require manual labeling and training. Although the proposed method in this paper also introduces pre-clustered topic knowledge, the splicing is different and it performs entirely in an unsupervised manner without prior labeled data.

The purpose of topic modeling is to cluster texts based on their semantic relationships. Given a document collection $\stackrel{\to }{d}$ containing D short texts, perform clustering to get the result $Z=\left\{z_1,\cdots ,z_D\right\}$, where $z_d\in \left\{1,2,\cdots ,K\right\}$ is the topic to which document d belongs and K is the number of topics. Finally, the topic-word distribution representation is constructed based on the importance of the word w in topic z. There are some problems with clustering using text similarity in the clustering process. This paper aims to improve the vector representation used for similarity computation. We use pre-clustered knowledge to reduce the intra-cluster distance and increase the inter-cluster distance, to improve the objective, the optimization problem of clustering accuracy, which is measured by Macro-F1.

Specifically, short texts have more low-frequency words, and their vectors are characterized by large l2 norms, making the distribution sparse. Therefore, in the automatic clustering phase of BERTopic, the density-based HDBSCAN algorithm can discover more and finer-grained clusters. When the number of automatic clusters is larger, it needs to be mapped to a pre-specified number of clusters, a process that requires cosine similarity. However, the characteristics of low-frequency words make the vector space present a narrow cone, showing anisotropy in the spatial distribution, which interferes with the calculation of cosine similarity. The short text has more low-frequency words, which leads to bias in topic vectors due to the uneven distribution of word frequencies. This affects the accuracy of the mapping results based on cosine similarity, so there is room for improvement of the native BERTopic. Moreover, it is challenging to mitigate the effect of anisotropy while maintaining semantic relevance.

Cosine similarity is a common method for calculating semantic similarity. For embeddings A and B, the cosine similarity is calculated as shown in Equation (1). The denominator represents the modulus of two embeddings, which can also be regarded as the l2 norm, i.e., the distance from the origin.

${\text{sim}}_{\text{cos}}=\frac{A\cdot B}{\Vert A\Vert \Vert B\Vert }$ (1)

Specifically, according to Li et al. [33] , language models can use word co-occurrence as a proxy for semantic similarity. Given the context c, the modeling process of language models can be simplified as:

$p\left(x|c\right)=\frac{\exp \left(h_c^{\text{T}}{W}_x\right)}{{{\displaystyle \sum }}_{x^{\prime }}{h}_c^{\text{T}}{W}_{x^{\prime }}}$ (2)

The similarity of sentence embeddings can be regarded as the similarity of context embeddings $h_c^{\text{T}}{h}_{c^{\prime }}$, and the dot product of context embeddings $h_c$ and word embeddings $w_x$ is used as a proxy in Equation (2). According to Li et al. [33] , in a well-trained language model, $h_c^{\text{T}}{W}_x$ can be approximately decomposed as:

$h_c^{\text{T}}{W}_x=\text{PMI}\left(x,c\right)+\text{logp}\left(x\right)+{\lambda }_c$ (3)

where PMI(x,c) denotes pointwise mutual information between x and c. This metric is semantically meaningful, so context vectors imply semantically relevant interactions due to higher-order word co-occurrence relations. Thus the cosine similarity can represent the semantic relations of sentences in the vector space. However, logp(x) represents the probability of the occurrence of word x. Low-frequency words have a larger l2 norm and are sparsely distributed, making the shape of vectors in the embedding space a narrow cone, characterized by anisotropy. In short texts, the occurrence probability of low-frequency words is larger, so it will cause bias in the semantic information captured by context vectors in higher-order space because of the distributional characteristics of word frequency.

Considering fewer short text features, this paper introduces pre-clustered knowledge to weaken the influence of word frequency through interaction between different features. In the pre-clustering phase, several potential problems may arise when the external corpus is used. On the one hand, pre-clustering is an unsupervised process and cannot adjust the learning direction of the model in contrast to supervised learning with predefined labels. Using an external corpus may introduce noise and data matching between two different data sources can complicate simple problems. On the other hand, the original corpus and external corpus may have large differences in feature space, which is not comparable in feature representation, and it is difficult to judge algorithmic effects with l2 paradigm. Based on the above considerations, this paper uses the original corpus as the object of pre-clustering to maintain simplicity and effectiveness. It is used as an improvement strategy to combine the topic knowledge from classical short-text clustering algorithms with semantic embeddings, but this also destroys the original semantic space and leads to inconsistency. The feature matrix being too large will also lead to the problem of large computation and long model fitting time, so the dimensionality reduction process is necessary before fusion to reduce the noise in data and prevent overfitting. The specific steps are as follows:

1) Get topic vectors by a short-text clustering algorithm, and reduce dimension;

2) Get semantic embeddings by a language model, and reduce dimension;

3) Splice embeddings, and compute similarity using cosine.

Taking vectors incorporating pre-clustered knowledge as the final representation of each document, the l2 paradigm of vectors under the same topic is reduced, and the spatial distribution is denser which alleviates anisotropy. In this way, the cosine similarity measure is more reasonable, which can improve the accuracy of the clustering results.

In brief, get topic vectors by a classical short-text clustering algorithm, and splice them and semantic embeddings in the cluster remapping phase of BERTopic, so that short texts on the same topic are more likely to be assigned into the same cluster. Figure 1 is the overall structure of BERTopic with pre-clustered knowledge.

When it is desired to cluster documents into class K, the process is shown below:

1) Get Embeddings

Use a classical short-text topic model to obtain the vector matrix of the topic ¹ and denote it as $g\in R^{D\times K}$. The topic vectors are generally sparse, so to minimize redundancy, use an auto-encoder to map it to lower dimensionality, and denote it as $g^r\in R^{D\times \left(K/2\right)}$：

$g^r=\text{AutoEncoder}\left(g\right)$ (4)

Sentences are fed into a pre-trained model SentenceBERT to generate semantic embeddings e. Use UMAP to reduce dimension, preserving the global structure while retaining local semantic features, and denote it as $e^r\in R^{D\times 128}$:

$e=\text{SentenceBERT}\left(\stackrel{\to }{d}\right)$ (5)

$e^r=UMAP\left(e\right)$ (6)

2) Cluster

The optimal clustering results are automatically generated using the HDBSCAN algorithm on semantic embeddings $e^r$, where $T\in R^{D\times 1}$ denotes corresponding labels after automatic clustering.

$T=\text{HDBSCAN}\left(e^r\right)$ (7)

When $\left|T\right|>K$, clusters mapping is required. Splice dimensionality reduced semantic embeddings with topic vectors as $e^{\prime }$, and takes the mean value under the same topic as the representation $t_k$ for each topic:

$e^{\prime }=\text{concat}\left(e^r,g^r\right)$ (8)

$e^r=\text{UMAP}\left(e\right)$ (9)

The distance matrix is computed through cosine similarity among topic representations and fed into Agglomerative Clustering algorithm for further clustering, to get new cluster labels $T^{\prime }$:

$\text{distance}=1-{\text{sim}}_{\text{cos}}\left(t\right)$ (10)

$e^r=\text{UMAP}\left(e\right)$ (11)

where the diagonal of distance is filled using zeros. Mapping the labels yields the final clustering result Z. Update the representation of each topic.

3) Topic Words

Generalize the classical TF-IDF as c-TF-IDF for the clustering granularity. It treats sentences under the same cluster as a whole and extracts important words to construct the topic-word representation. The importance score is shown in the following equation:

$W_{w,z}=t{f}_{w,z}\cdot \log \left(1+\frac{A}{t{f}_w}\right)$ (12)

where $t{f}_{w,z}$ denotes the co-occurrence frequency of word w in topic z, and A denotes the average number of words across all topics.

Dataset: StackOverflow is a well-known community for programmers to learn and share. The dataset collects post-question titles on different topics. For each topic, 100 pieces of data are extracted, satisfying lengths of 5 to 40. After conversion to lowercase, stemming extraction and lexical reduction, the basic attributes are shown in Table 1 .

Baseline: To evaluate the effectiveness of G-BERTopic, the following baselines are selected for comparison in this paper.

The model proposed in this paper consists of two parts: cluster generation and topic representation, so the evaluation metrics are selected for these two aspects. For cluster quality, it includes internal and external evaluation metrics, where internal quality evaluates the cluster structure and external quality relies on the accuracy of topic assignment accuracy [34] . Topic representation is evaluated by choosing semantic coherence in this paper.

In this paper, the following evaluation metrics were chosen and the results are the mean values of 10 experiments.

Macro-F1 is an external evaluation metric for datasets that are not affected by data imbalance. We take the same number of samples for each category of the StackOverflow dataset. It is defined as:

${\text{F1}}_{\text{macro}}=\text{2}\cdot \frac{{\text{Precision}}_{\text{macro}}\cdot {\text{Recall}}_{\text{macro}}}{{\text{Precision}}_{\text{macro}}+{\text{Recall}}_{\text{macro}}}$ (12)

where ${\text{Precision}}_{\text{macro}}$ and ${\text{Recall}}_{\text{macro}}$ are the average values of ${\text{Precision}}_i$ and ${\text{Recall}}_i$ for each category. ${\text{F1}}_{\text{macro}}$ takes the value in the range of $\left[0,1\right]$. The higher the score, the better the clustering quality.

NMI(Normalized Mutual Information) is a normalization of MI used to measure the degree of consistency of two clusters, and belongs to external evaluation metrics. It is defined as:

$\text{NMI}\left(X,Y\right)=\frac{\text{MI}\left(X,Y\right)}{F\left(H\left(X\right),H\left(Y\right)\right)}$ (13)

where $X=\left\{x_1,x_2,\cdots ,x_k\right\}$ is assigned topics, $Y=\left\{y_1,y_2,\cdots ,y_k\right\}$ is true topics, and $\text{MI}\left(X,Y\right)$ is the mutual information of X and Y, reflecting the degree of correlation of the two. $H(\cdot )$ the entrop and $F(\cdot )$ is a normalization function. NMI takes values in the range of $\left[0,1\right]$. The higher the score, the better the clustering quality.

DB (Davies-Bouldin) is an internal evaluation index, which measures differences between clusters and closeness within clusters. It is defined as:

$\text{DB}\left(k\right)=\frac{{{\displaystyle \sum }}_{i=1}^k\underset{j.j\ne i}{\max }\left\{\frac{S_i+S_j}{d_{ij}}\right\}}{k}$ (14)

where $S_i=\left(1/n\right){\displaystyle {\sum }_{x\in C_i}\Vert x-z_i\Vert }$ measures tightness in cluster $C_i$, $d_{ij}=\Vert z_i-z_j\Vert$ measures the dispersion between $C_i$ and $C_j$. $n_i$ denotes the number of samples in $C_i$, $z_i$ is the center of mass of $C_i$, and $\Vert \cdots \Vert$ denotes Euclidean distance. The smaller the value, the better the quality of clustering.

Coherence measures the coherence of the topic. In this paper, we use the idea proposed by Röder et al. [35] , which utilizes the topic representation and reference corpus to determine the contextual features of each topic word. The higher the value, the better the topic modeling results.

The comparison results are shown in Table 2 . Note that the external evaluation metrics need to compare the original topics with the predicted topics, so we use the Hungarian Algorithm for matching. Experiments show that G-BERTopic has superior metrics on the StackOverflow dataset, with Macro-F1 of 0.661, which is 0.16 better than the original BERTopic and maintains the best or second best in other metrics. The results show that the incorporation of pre-clustered topic knowledge improves accuracy while ensuring semantic coherence, proving effectiveness of G-BERTopic.

Table 2. Comparison of indicators.

Experiments were conducted by incorporating pre-clustered topic knowledge at different phases of BERTopic, and the results are shown in Table 3 . The result shows that the highest accuracy rate is after clustering, which justifies the algorithm, that is, the first clustering phase should not be destructed. The dimension reduction phase is a key step that affects the clustering results by performing a low-dimensional dense vectorized representation of documents. Therefore, adding topic information in the first two cases, whether before dimension reduction or after dimension reduction before clustering, affects the intuition of clustering based on semantic similarity, i.e., adding other features instead blurs the semantic understanding. The purpose of generating topic vectors after the first clustering is to obtain a comprehensive representation, so splicing topic-related vectors from GSDMM in the second clustering phase can add more effective knowledge to improve the accuracy of clustering.

Since word embeddings and sentence embeddings share the same high-dimensional space and short texts are characterized by more low-frequency words, word embeddings are used here instead of sentence embeddings to verify whether the addition of pre-clustered knowledge improves the anisotropy. In this paper, l2 norm of different word frequencies is calculated to compare the distance from the origin before and after the improvement, as shown in Table 4 . From the results, it can be seen that word frequency brings bias to the semantic space, because low-frequency words are distributed farther than high-frequency words. After splicing the topic knowledge, l2 norms are all reduced to different degrees, indicating that this approach makes the vector distribution more tightly, which alleviates that bias to some extent.

Table 3. The effect of incorporating pre-clustered knowledge at different stages.

For different word frequencies, different components are grouped into 3 clusters by k-means, and SSE is calculated as the degree of closeness in the vector space. From the results, high-frequency words are denser than low-frequency words. So, the sparse distribution of low-frequency words will lead to some gaps in the semantic space, making the similarity of embeddings not accurately represent the semantics of sentences. The method in this paper reduces the sum of squares of the distances of word embeddings to cluster centroids in different degrees, and the lower the frequency, the better the effect. This proves that the method proposed in this paper can alleviate this sparsity and make the vector representation semantically smoother, which in turn improves the accuracy of clustering.

In conclusion, the experiment proves that adding pre-clustered knowledge in the process of secondary clustering can improve the clustering accuracy of BERTopic.

BERTopic separates clustering and topic representation, where the language model can be replaced with the state-of-the-art. Similarly, G-BERTopic can combine BERTopic with different classical topic models. The results in Table 5 show that Macro-F1 is significantly improved after the combination compared to the native algorithm, and it remains comparable in NMI and Coherence. For the internal metric DBI, it is worse because the addition of features affects semantic space, but it has been shown in Section 4.3 that adding new features helps mitigate anisotropy and can improve clustering accuracy, so such a loss is worthwhile. It proves that incorporating pre-clustered knowledge in BERTopic can improve the clustering effect. The reason for choosing GSDMM is that the two are more consistent in their assumptions, i.e., they both assume that a document contains only one topic. In addition, due to portability, the effect of topic assignment can be further improved by replacing the self-aggregating topic model with a more advanced one.

Table 5. Comparison of BERTopic combined with different pre-clustered models.