The non-negative matrix factorisation (NMF) method, which was first used in the paper [1] , is a technique that allows a non-negative matrix $X$ to be decomposed into two non-negative factors given by $W$ and $H$, where

$X\approx W\times H,$

where $W$ is an $m\times r$ matrix with elements in ${ℝ}^{+}$, and $H$ is an $r\times n$ matrix with nonnegative elements. Here, $m$ represents the number of features, $n$ represents the number of observations or samples, while $r$ denotes the rank or dimensionality of the feature subspace of $X$.

Let $\left\{x_1,x_2,\cdots ,x_n\right\}\in {ℝ}^m$ be a set of data. For any column vector $x_j$ in $X$, we can express it as

$x_j\approx \underset{k=1}{\overset{r}{{\displaystyle \sum }}}{w}_k{h}_{kj},$

indicating that $x_j$ can be approximated by a linear combination of the basis vectors (i.e., all column vectors of $W$), with the components of $H$ acting as the weight coefficients. Thus, $H$ is also known as the coefficient matrix, allowing the feature vector to be computed as a linear combination of the basis vectors.

Originally proposed for feature extraction, the NMF nonnegative nature makes it suitable for various non-negative processing tasks, underpinned by strong theoretical foundations and interpretability [2] . In pattern recognition, dimensionality reduction is crucial for efficiently managing high-dimensional data and improving recognition accuracy. While the K-means algorithm is frequently used for clustering [3] , its performance often degrades in high-dimensional spaces, underscoring the necessity for effective dimensionality reduction techniques.

In addition to pattern recognition, NMF has applications across various fields, including dimension reduction [4] , image processing [5] , speech processing [6] [7] , spectral analysis [8] [9] , DNA expression analysis [10] [11] , microRNA-disease analysis [12] [13] , social network analysis [14] , text analysis [15] , hyperspectral signal unmixing [16] , and blind spectral unmixing [17] [18] . As data size increases, so does the demand for processing and storage. This has led researchers to develop new algorithms for linear dimensionality reduction (LDR), a data analysis method that reduces the dimensionality of information with minimal loss of information before analysis. The transformation of data is an effective means to improve the linear representation. The idea is to transform the data via a function $f$ to be learned, and use $f\left(X\right)=\left[f\left({\theta }_1\right),f\left({\theta }_2\right),\cdots ,f\left({\theta }_n\right)\right]$ and apply the factorization to $f\left(X\right)$ instead of $X$, i.e. $f\left(X\right)\approx W\times H$.

We assume that each data point is associated with a label $y_1,\cdots ,y_n$. In the most basic case, they have labels $y_i$ that are binary, i.e., they can take two values representing two classes. For multi-class cases, the labels can be represented using one-hot encoding [19] . A prominent example of a non-linear transformation applied to count data in document classification is the term frequency-inverse document frequency (TF-IDF) transformation [20] [21] . In this context, $X\left(i,j\right)$ denotes the frequency of word $i$ appearing in document $j$, and the TF-IDF transformation maps $X$ to $f\left(X\right)$ where

$X\left(i,j\right)=f\left(X\left(i,j\right)\right).$

In [22] [23] , document classification is achieved through NMF and KL-NMF. Additionally, [24] demonstrates the application of non-linear transformations to enhance the effectiveness of these classification methods. This motivates our exploration of improved transformations, which we aim to apply to various document datasets. This motivates our exploration of some nonlinear transformations, which we intend to apply to various document datasets to enhance the results obtained in [25] .

Another effective method is the probabilistic IDF [26] , which modifies the basic IDF calculation to incorporate probabilities, ensuring that terms that do not appear in a document are still assigned meaningful weights. Jelinek-Mercer smoothing [27] [28] , initially developed for language modeling, can also be adapted for TF-IDF by interpolating maximum likelihood estimates with background distributions.

Moreover, Katz [29] and Witten-Bell [30] smoothing techniques provide further enhancements by allowing for interpolation between observed and unobserved frequencies, improving robustness in document classification tasks at language analysis. It is important to emphasize the use of these smoothing techniques to improve the performance of TF-IDF so that the weights assigned to terms indicate the general importance of the terms in the overall corpus.

Outline and Contribution of the Paper

This paper proposes a new method for document classification. As detailed in Section 2, KL-ONMF serves as the reference for our method. We will smooth the TF-IDF and incorporate parameters (see Section 3), which we will optimize using the fminsearch function in MATLAB with a machine learning approach. We apply non-linear transformations in order to capture non-linear characteristics better. We will show that this new approach is much better than KL-ONMF on the original data for document classification (see Section 4).

In this section, we focus on Alternating Optimization for Orthogonal Nonnegative Matrix Factorization (ONMF) with the Kullback-Leibler (KL) Divergence. The goal is to minimize the KL Divergence between the given matrix $X$ and the product of two matrices $W$ and $H$, while enforcing specific constraints on $H$.

The optimization problem can be formulated as follows:

$\underset{W\in {ℝ}^{m\times r},H\in {ℝ}^{r\times n}}{\text{min}}{D}_{\text{KL}}\left(X,WH\right)\text{subject}\text{to}H\ge 0\text{and}H{H}^{\top }=I_r.$

For ONMF utilizing the KL Divergence, it is crucial that $X$ is component-wise nonnegative, reflecting the inherent properties of KL Divergence, which is only defined for non-negative matrices. This non-negativity is essential, as it ensures that the divergence measurement remains valid and interpretable in the context of probability distributions.

In contrast, ONMF using the Frobenius norm does not impose such restrictions, allowing for broader applications but potentially leading to less interpretable results in probabilistic terms. The choice of KL Divergence in this context emphasizes our focus on modeling non-negative data, making it particularly suitable for applications like document classification or image processing, where the underlying data naturally adheres to non-negativity constraints.

Moreover, the constraint $H{H}^{\top }=I_r$ ensures that the matrix $H$ retains orthogonality, which is vital for preserving the interpretability of the factors derived from the decomposition. This orthogonality condition aids in distinctly separating the components, allowing for clearer insights into the latent structures represented by the factors in $W$ and $H$.

This formulation highlights the methodological rigor behind ONMF with KL Divergence, ensuring that the results are both mathematically sound and practically applicable to real-world datasets. The update rules for $W$ and $H$ in ONMF with the KL Divergence highlight the different optimization strategies compared to Frobenius ONMF, making KL-ONMF less sensitive to outliers and more focused on data points with smaller norms.

Algorithm 1 summarizes the alternating optimization scheme, known as KL-ONMF.

To address the limitations of raw term frequency counts and ensure a more balanced representation of term importance, we introduce smoothing techniques in the calculation of both term frequency (TF) and inverse document frequency (IDF). The term frequency is recalculated with a smoothing term to prevent zero counts from disproportionately affecting the results. Similarly, the inverse document frequency is adjusted to ensure that terms common across many documents do not dominate the weighting.

The term frequency $\text{TF}\left(i,j\right)$ is computed as:

$\text{TF}\left(i,j\right)=\frac{f_{i,j}+\text{smoothing}}{{\displaystyle \underset{i^{\prime }\in j}{\sum }{f}_{i^{\prime },j}}+\text{smoothing}},$(3)

where smooth = 1 for Laplace smoothing or add-1 smoothing [33] . The inverse document frequency $\text{IDF}\left(i\right)$ is calculated to measure the importance of the term across the document corpus:

$\text{IDF}\left(i\right)=\text{log}\left(\frac{N+\text{smoothing}}{\text{DF}\left(i\right)+\text{smoothing}}\right).$(4)

where $\text{DF}\left(i\right)$ represents the number of documents containing the term $i$. This formulation helps to mitigate the impact of terms that appear in a large number of documents, ensuring that the IDF value remains informative. The TF-IDF weighting of a term is therefore the product of its TF term frequency Equation (3) and its IDF document inverse frequency Equation (4). It is defined by Equation (5):

$X\left(i,j\right)=\text{TF}\left(i,j\right)\times \text{IDF}\left(i\right).$(5)

In the following, we start with a matrix $X$ representing word occurrences, where each element $X\left(i,j\right)$ indicates how many times word $i$ appears in document $j$. The algorithm computes the Term Frequency (TF) for each word in a document, reflecting its relative importance, and the Inverse Document Frequency (IDF) measures how common or rare a word is across all documents. The resulting TF-IDF scores highlight the significance of words in the context of the documents. The detailed algorithm is described below.

In this section, we use nonlinear functions for document clustering, and we compare the performance of KL-ONMF (Algorithm developed in [25] ) applied to the original data with that of KL-ONMF applied to the transformed data using (Algorithm). All experiments were run on a LAPTOP 11^th Gen Intel^® Core^TM i5-1135G7 @ 2.40GHz × 8 16,0 Go RAM.

Initialization:

Similar to k-means, ONMF algorithms can be initialized in a variety of ways. We take the approach suggested in [35] to simplify the presentation. The successive nonnegative projection algorithm (SNPA) [36] is used in this method to initialize $W$. It finds a subset of $r$ columns from $X$ that accurately represent the dataset’s evenly distributed data points.

To refine the parameters during the optimization phase, we utilize the Nelder-Mead optimization algorithm [37] . For this algorithm, we initialize the parameters by generating random values. The Nelder-Mead algorithm iteratively refines these parameters based on the objective function (Equation (10)), which in our case is designed to minimize the difference between the transformed data and the original data, while also incorporating a regularization term. By combining the SNPA for the initialization of $W$ and the Nelder-Mead method for parameter optimization, we aim to achieve robust and efficient convergence in the ONMF framework.

Parameterization of Functions:

To minimize the error rate of the objective functions and enhance accuracy, we begin by setting a fixed initial parameter value for $a$ at 0.0001. This choice ensures consistency across our optimization process, allowing us to focus on adjusting the other parameters, $b$ and $c$. The values for $b$ and $c$ are randomly initialized (see Section 3.6), specifically rounded to three decimal places to maintain precision. This randomness introduces variability in the initial conditions, which can help the optimization algorithm explore different configurations effectively. To find the optimal values of $b$ and $c$, we employ the fminsearch algorithm, which utilizes the Nelder-Mead method. This optimization technique is particularly beneficial for multidimensional problems where derivatives may not be easily computable. It iteratively adjusts the parameters based on the evaluation of the objective function, seeking to minimize the error rate. By systematically refining the values of $b$ and $c$, the algorithm converges towards a local parameterization that improves the model’s accuracy in its predictions. Overall, this approach ensures a robust framework for optimizing the model’s performance based on the given dataset.

Objective Function:

The objective function aims to minimize the sum of the squared deviations between the calculated and observed outflows. This approach ensures that the model’s predictions closely align with the actual data. Let $\Theta$ be the set of parameters defined as:

$\Theta =\left\{{\theta }_1,{\theta }_2\right\}$

where ${\theta }_1$ represents the first coefficient influencing the model’s predictions, and ${\theta }_2$ represents the second coefficient. The parameters in $\Theta$ are crucial for minimizing $J\left({\theta }_1,{\theta }_2\right)$ while adhering to the constraints that require both ${\theta }_1$ and ${\theta }_2$ to be non-negative. This ensures that the model parameters remain within a physically meaningful range, ultimately leading to a more accurate representation of the relationship between the inputs and outputs.

Mathematically, the objective function can be expressed as:

$\min J\left({\theta }_1,{\theta }_2\right)={\Vert X_{\text{train}}-Y\Vert }_F^2+\{\begin{array}{ll}1\times {10}^6\hfill & \text{if}{\theta }_1<0\text{or}{\theta }_2<0\hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}+\lambda \left({\theta }_1^2+{\theta }_2^2\right)$(10)

where $J\left({\theta }_1,{\theta }_2\right)$ measures how well the model’s predictions (calculated outflows) match the actual observed outflows. The goal is to find the parameters ${\theta }_1$ and ${\theta }_2$ that minimize this function. The Frobenius norm ${\Vert .\Vert }_F^2$ quantifies the difference between the predicted and actual values. Here, $X_{\text{train}}$ represents the observed outflows, while $Y$ denotes the calculated outflows based on the model and parameters ${\theta }_1$ and ${\theta }_2$. The calculated outflows $Y$ can be expressed as:

$Y=W\cdot H$(11)

where $W$ is the matrix of features and $H$ is the matrix of coefficients derived from the parameters ${\theta }_1$ and ${\theta }_2$. The Frobenius norm computes the squared difference between these two sets of values, aggregating the errors across all observations. To discourage invalid parameter values, a penalty term is added, ensuring that only non-negative parameters are considered in the optimization process. To prevent overfitting by penalizing large values of the parameters, we add a regularization coefficient $\lambda$ that controls the strength of this penalty, balancing the fit of the model to the data against the complexity of the model.

Document Data Sets:

We cluster the 14 document data sets from [38] using KL-ONMF. Table 1 provides not only the names of the data sets but also their dimensions (where $m$ represents the number of words and $n$ the number of documents) and the number of clusters $r$.

Accuracy:

Accuracy is defined as the proportion of correctly classified data points relative to the total number of data points. In the setting of document clustering, the accuracy of a computed disjoint clustering ${\left\{{\tilde{C}}_i\right\}}_{i=1}^r$ compared to the true disjoint clusters $C_i\subset \left\{1,2,\cdots ,n\right\}$ for $1\le i\le r$ is defined as follows:

$\text{accuracy}\left({\left\{{\tilde{C}}_i\right\}}_{i=1}^r\right)=\underset{\pi \in \left[1,2,\cdots ,r\right]}{\text{min}}\frac{1}{n}\underset{i=1}{\overset{r}{{\displaystyle \sum }}}\left|C_i\cap {\tilde{C}}_{\pi \left(i\right)}\right|.$(12)

The correctly classified data points are those that have been assigned to the same cluster as their original data. The clustering process was carried out using the K-means algorithm.

Running time:

Table 2 summarizes the execution times associated with the objective function evaluated through the Nelder-Mead optimization algorithm across various datasets.

The results indicate substantial variability in processing times, with the “classic” dataset exhibiting the highest execution times, ranging from 172.11 to 215.82 seconds. This suggests that this dataset may present more complex optimization challenges or larger data sizes. Conversely, datasets like “ng3sim” and “hitech” show significantly lower mean execution times of 1.84 and 1.65 seconds, respectively, indicating efficient convergence during optimization. The “la2” dataset, with a maximum time of 55.91 seconds, points to potential outliers that may require further analysis to understand the underlying causes of increased processing time. Meanwhile, datasets such as “tr11,” “tr23,” “tr41,” and “tr45” demonstrate consistent execution times, which may reflect similar optimization landscapes. Overall, these execution times underscore the impact of dataset characteristics on the performance of the Nelder-Mead optimization algorithm, highlighting opportunities for further refinement in handling more computationally intensive datasets.

Optimal Parameters

In this section, we present the optimal parameters identified for each dataset through cross-validation. The table below summarizes these parameters, which play a crucial role in enhancing the performance of our model. The first column of the table lists the datasets. To ensure stability and prevent excessive fluctuations during optimization, we fixed the parameter $a$ at 0.0001 rather than allowing it to be optimized alongside $b$ and $c$. This choice is guided by the observation that large variations in $a$ can lead to significant distortions in the TF-IDF scores, potentially overshadowing the contributions of the other parameters. By maintaining $a$ at a low constant value, we provide a baseline influence that stabilizes the model while allowing $b$ and $c$ to be fine-tuned for better performance.

In a sensitivity analysis, we examined the effects of varying $a$ around its fixed value. The results indicated that small deviations from 0.0001 did not substantially alter the overall performance of the clustering algorithms. However, larger adjustments (e.g., setting $a$ to 0.01 or higher) resulted in noticeable degradation in clustering quality, highlighting the importance of keeping $a$ within a narrow range. This analysis supports our decision to fix $a$ at 0.0001, ensuring that the model remains robust while still allowing $b$ and $c$ the flexibility needed to adapt to specific dataset characteristics. The remaining columns represent the optimized parameters $b$ and $c$, which vary for each dataset. The parameter $b$ influences the model’s responsiveness to changes in the data, while $c$ affects the overall scaling of the model’s output. Finally, the “Best Smooth” column indicates the optimal smoothing value achieved during cross-validation, which reflects the model’s performance in terms of clustering accuracy. The varying values in this column highlight how different datasets require tailored parameter settings to achieve the best results.

Table 3 presents parameter values for three different nonlinear transformation: logarithmic transformation (Equation (7)), square root transformation (Equation (8)), and hyperbolic tangent transformation (Equation (9)). Even though these methods use different mathematical functions, the values for parameters $b$ and $c$ are very similar across all datasets. For the logarithmic transformation, $b$ usually ranges from 0.068 to 3.180, and $c$ ranges from 1.091 to 27.062. The square root transformation also shows similar patterns, with $b$ values in the same range and $c$ values that suggest these two transformations might work well with similar types of data. By analyzing the ranges of the parameter of the “Best Smooth”, we observe interesting trends: for the logarithmic transformation, the values of $b$ range from 0.068 to 3.180, while those for $c$ span from 1.091 to 27.062. The “Best Smooth” values in this transformation vary from 0.10 to 66.67. For the square root transformation, the $b$ parameters remain within a similar range, from 0.068 to 3.374, and the values of $c$ range from 1.091 to 27.057. The “Best Smooth” values for this function is between 0.10 and 46.67. For the hyperbolic tangent transformation, the $b$ values also fall within the same interval, from 0.068 to 3.636, with $c$ values ranging from 1.091 to 27.064. The “Best Smooth” values in this approach vary from 0.10 to 83.33. These intervals demonstrate that, although the transformations employ different mathematical nonlinear functions, the parameters $b$ and $c$ exhibit significant similarities, suggesting that these transformations may be well-suited for similar types of data. The choice of a fixed starting value for $a$ allows for a clearer analysis of the impact of $b$ and $c$ on the results, minimizing the effects of variations in $a$.

Performance Evaluation:

We present the accuracy achieved by KL-ONMF on both the original and transformed data across different datasets. The accuracy metrics are denoted as follows: acc-KL for overall accuracy, BestAcKL-ONMF (Train) for the accuracy on the training dataset, and BestAcKL-ONMF (Validation) for the accuracy on the validation dataset. The distinction between “training” and “validation” phases is critical in our approach, particularly given the nature of the KL-ONMF method and the transformations applied to the data. During the training phase, the model focuses on learning complex patterns and relationships within the feature space that is generated from the transformed TF-IDF scores. This phase heavily relies on the training data, which plays a significant role in adjusting the model’s parameters. Conversely, the validation phase serves an essential purpose by preventing the model from overfitting to the training data. Evaluating the model on a separate validation set allows us to gauge its ability to generalize to new, unseen data—an aspect that is vital for practical applications. In our methodology, where TF-IDF scores are subjected to various transformations, validating performance on data that the model has not previously encountered becomes particularly important. This validation helps us assess the effectiveness of the smoothing and non-linear transformations applied, ensuring they genuinely enhance the model’s clustering and classification capabilities rather than simply memorizing the training data. Thus, the clear separation of training and validation phases in our approach not only bolsters the reliability of the model’s performance but also underscores its robustness in real-world scenarios.

Table 4 compares the performance of the AcKL-ONMF method applied to both untransformed data and smoothed, non-linear transformations. The metrics include baseline accuracy, Best AcKL-ONMF accuracy for training, and validation sets across various datasets.

Notably, the Best AcKL-ONMF(Train) values consistently outperform the baseline accuracy, indicating that the method effectively learns features from the transformed data. For instance, Dataset 1 shows a significant increase from 71.45% to 74.61%, demonstrating enhanced model performance through the application of smoothing and non-linear transformations.

When examining the Best AcKL-ONMF(Validation) results, there is a clear trend of improvements in accuracy on unseen data. Datasets such as Dataset 2 show a rise from 85.38% to 91.99% in training, with a validation accuracy of 85.49%, illustrating the model’s ability to generalize effectively.

The average accuracies further validate our approach. The training set achieves an average of 60.61%, while the validation set maintains competitive performance at 59.01%. This consistency across both training and validation sets suggests that the AcKL-ONMF method not only optimizes the model parameters well but also ensures robust performance on unseen data.

Table 5 compares the performance of the AcKL-ONMF method applied to both untransformed data and smoothed, non-linear transformations. The metrics include baseline accuracy, Best AcKL-ONMF accuracy for training, and validation sets across various datasets.

Notably, the Best AcKL-ONMF(Train) values consistently outperform the baseline accuracy, indicating that the method effectively learns features from the transformed data. For instance, Dataset 1 shows a significant increase from 71.45% to 73.85%, demonstrating enhanced model performance through the application of smoothing and non-linear transformations.

When examining the Best AcKL-ONMF(Validation) results, there is a clear trend of improvements in accuracy on unseen data. Datasets such as Dataset 2 show a rise from 85.38% to 92.92% in training, with a validation accuracy of 92.46%, illustrating the model’s ability to generalize effectively.

The average accuracies further validate our approach. The training set achieves an average of 59.38%, while the validation set maintains competitive performance at 58.48%. This consistency across both training and validation sets suggests that the AcKL-ONMF method not only optimizes the model parameters well but also ensures robust performance on unseen data.

Table 6 compares the performance of the AcKL-ONMF method applied to both untransformed and smoothed, non-linear transformed data. The metrics include baseline accuracy, Best AcKL-ONMF accuracy for training, and validation sets across various datasets.

The Best AcKL-ONMF(Train) values consistently exceed the baseline accuracy, demonstrating the method’s effectiveness in extracting features from the transformed data. For example, Dataset 1 shows an increase from 71.45% to 73.86%, indicating a substantial improvement in model performance due to the application of smoothing and non-linear transformations.

In terms of Best AcKL-ONMF(Validation), the results reveal a positive trend in accuracy for unseen data. Dataset 2 exemplifies this, with the accuracy rising from 85.38% to 91.58% in training, while validation accuracy stands at 82.24%. This illustrates the model’s capacity to generalize effectively to new data.

Average accuracies further support our findings, with the training set achieving an average of 61.39%, compared to a baseline of 58.98%. The validation set maintains a competitive average of 59.06%, reinforcing the consistency of the AcKL-ONMF method across both training and validation contexts.

In this paper, we developed an innovative clustering approach for non-negative data, named Optimal Predictive Modeling of Nonlinear Transformations using the Kullback-Leibler Divergence “OPMNT-KL-ONMF”. Our methodology begins by applying smoothing to TF-IDF scores, followed by the integration of parameters optimized using the Nelder-Mead method. We then employ nonlinear transformations to extract nonlinear features. Comparative results between KL-ONMF applied to original data and data transformed by OPMNT demonstrate a significant performance improvement. Our OPMNT-KL-ONMF model represents a promising advancement in the field of clustering non-negative data, such as document datasets, and paves the way for future research and applications across various domains of data analysis.

Sincere thanks to the members of OJAppS for their exemplary professionalism, and a special acknowledgment to Managing Editor Emily Lee for her exceptional commitment to high-quality standards.