The US has long been recognized as a powerful tool used in the diagnosis and evaluation of different clinical examinations. Speckle is most often considered a dominant source of noise in US imaging [1]. In medical ultrasound imaging, the speckle noise masks small differences in grey level and spoils its visual observation quality and diagnostic procedure. Also, the presence of speckle noise limits the accuracy of computer-assisted methods such as feature extraction, analysis, recognition, and quantitative measurements problematic and unreliable [2]. Ultrasound image despeckling is a very challenging task in US images, therefore considerable effort has been spent over the last few decades in developing techniques to reduce the noise [3]. The speckle reduction aims to remove speckle noise as well as enhance image features and the detectability of small and low-contrast objects.

One of the state-of-the-art denoising methods is the NLM algorithm proposed by Buades [4]. The core idea of NLM filter to restore the noisy image is based on performing a weighted average of the most similar pixels to the pixel to be denoised are given the largest weights incorporating the non-local information [5]. Practically, the similarity is computed between equally sized neighborhoods around the pixels in consideration. These patches are more capable of capturing the image structures (e.g., texture) [6]. However, in the NLM filter as well as many denoising techniques, the suppression of the noise starts to degrade when the noise level increase. The presence of noise generally increases the dissimilarity between image patches and hence degrades the matching performance. This eventually leads to decreases in the efficiency of the NLM filter.

This work proposed modifications to the NLM method to handle the high speckle noise level in medical ultrasound imaging. The solution to this challenge is to design a robust and efficient NLM filter robust to presence of large noise. The question now becomes: how to determine the similarity weights in an optimal sense? Our key element is to design a high-performance NLM filter by improving the similarity computation between the image pixels, which is the core of the NLM algorithm.

To address this problem, one of the approaches to deal with noisy data is to learn the representation of the free noise and the most informative structures of the data matrix. Luckily, one of the promising techniques of machine learning is the kernel PCA, which can provide a solution to this problem. The kernel PCA can come up with a robust representation of the data in the high dimensional space. This work concerns the kernel PCA technique and can provide the ability to find the best image patch feature containing small noise, then calculate the similarities between patches around the pixels in that subspace. In this new space, the hidden information of the image patches due to the presence of the noise can be obvious. In sense of NLM, representing the image patch more concisely can help with their matching. These theoretical advantages of the kernel method motivated us to calculate similarity measurement in high dimensional space to improve the efficiency of the NLM in the presence of large noise. This will be a super useful method for medical ultrasound images, due to the narrow dynamic range of the gray levels and the typically high level of noise which is introduced at all stages of image acquisition, especially the speckle noise [7]. To our knowledge, no research exists addressing the kernel PCA-NLM framework to derive a NLM filter for ultrasound despeckling.

The initial formulation of the NLM filter relies on the assumption of an additive white Gaussian noise model (AWGN). However, the noise which degrades ultrasound images is signal-dependent and more complex. The distribution of noise in ultrasound images has been largely studied in the literature and many models have been proposed. A more general speckle noise model was first introduced for ultrasound image denoising by Loupas et al. [8]. This model has been considered in many studies. Accordingly, given this model, in this paper we introduce a novel restoration scheme for US images, using the kernel PCA motivation for the NLM filter. The remainder of this paper is structured as follows. Section 2 covers the basic idea of the NLM in more detail, in addition to briefly reviewing different approaches to optimize the parameters of the NLM to improve its quality or speed in the author’s own terms. Section 3 gives a brief overview of the main speckle reducing categories, providing a short description of the principle behind each family. Section 4 introduces the common speckle noise models. The theory underlying the OBNLM is presented in details in Section 5, which adapted to speckle noise and archived best despeckling results among the others.

Section 6 explains the idea behind the learning representation theory. Section 7 describes the implementation of the kernel PCA-NLM speckle reduction method. In addition to applications of the proposed de-speckle information about artificial phantom data and real ultrasound images and definitions of image quality metrics are presented in details. Section 8 shows the experimental promising results of the proposed method. The theoretical interpretations of the findings are due to the superiority of the proposed discussed in Section 9. Finally, a conclusion and further works are needed to bring an improvement to performance of our speckle filter in Section 10.

Appendix A contains the important martial, that we do not put in the body of the paper which is necessary to understand the data analysis. Appendix A contains all the mathematical background of linear PCA, and procedure to compute the PCA. Lastly, the kernel PCA method which has been developed as a non-linear version of the PCA to extract feature in higher dimensional space is explained.

Given a discrete noisy image v = { v ( i ) | i ∈ Ω } the NLM scans the huge space of the image to enhance the redundancy, instead of update the pixel value as an average of its limited local four or eight neighborhood as local image filter. The contribution of the all others pixels is weighted according to their similarities. Given these weights the estimated value NL(v)(i) is computed as a weighted average of all pixels in the image [9]. The noise goes down with square root of the number of times of averaging [10].

N L ( v ) ( i ) = ∑ j ∈ Ω w ( i , j ) v ( j ) (1)

NLM looks for similar pixels around and average them for denoising the current pixel. To find the similar pixels, the Euclidean distance (d(i, j)) between pixelsat location i, j is computed. The d(i, j) does not depend only on the pixels v(i), and v(j) but also on its surrounding windows x_i and x_j, respectively. By using patch-wise representation the pixel is no longer is a single feature, but a more informative high dimensional data point. This window size should be large enough to be robust to the noise and at the same time to be able to preserve fine details. Indeed, the patch size reflects the scale of the “noise” compared to the image resolution [11].

Based on theoretical grounding nonlocal philosophy, the search window (V) of radius (t) should be large enough to be robust to noise and at the same time to be able to take care of the details and fine structure [11]. In [12], the authors have noticed that, choosing locally the best search windows can control the filtering results. Smaller search windows are not robust enough to noise and in case of strongly textured images [13]. As search window grows, there are slightly differences among the root mean square error (RMSE), but sharply increasing in the operation period. Thus, restricting the search of patches to a window is the common practice that, besides the speed-up, the result is visually better [5]. With a smaller search window size, the small features will be preserved [14].

In the original NLM the Euclidean distance using a Gaussian weighted function, is given by:

d ( i , j ) = ∫ Ω G a ( τ ) | v ( i + τ ) − v ( j + τ ) | 2 d τ (2)

Normally the dissimilarity of the patches is not a binary thing, and the dissimilarity of the patches is transformed into weighting function, that argues which patches are considered as a similar or a dissimilar. To achieve this weighting a Gaussian weighting function is used, which is defined as:

w ( i , j ) = 1 Z ( i ) exp ( − d ( i , j ) h 2 ) (3)

where the parameter h controls the decay of the weighting function and Z(i) is a normalizing term defined by:

Z ( i ) = ∑ j exp ( − d ( i , j ) h 2 ) (4)

In the natural images there are some points which are more distinct and unique than the others, and do not appear anywhere. On the other hand there patches that have more close patches than others. So the normalizing term is needed to count and guarantee the following conditions:

0 ≤ w ( i , j ) ≤ 1 ∑ j w ( i , j ) = 1 (5)

The exponential weighting function can be considered as a decreasing function depending on the similarity of the patches. If a particular local difference has a large magnitude, then the value of the w(i, j) will be small and therefore, that measurement will have little effect on the output image. The NLM uses exponential weighting function of the Euclidean distance between their arguments, which still assigns positive weights to dissimilar neighbourhoods. When weights are very small, the estimated pixel intensities can be severely biased due to many small contributions [15].

The parameter h, namely the width of the exponential function, quantifies how fast the weights decay with increasing dissimilarity of respective patches. The larger the parameter the smoother the output, and the edges will be blurred, and on the other hand, choosing a very small h leads to noisy results identical to the input [11]. The h parameter is typically optimized manually in the NLM algorithm [16]. The simplest and most common one is to set a single h for the whole image. The h parameter is an important parameter, and best value of h is roughly proportional to noise standard deviation [4].

Several different despeckle filtering methods are used based upon different mathematical models of the phenomenon [17]. Speckle is considered to be a deterministic process (not random) because when an object is imaged under the same operating conditions no changes in the speckle pattern occur. For these reasons, speckle cannot be reduced by signal averaging over time. Unlike the additive white Gaussian noise model adopted in most denoising methods, ultrasound imaging requires specific filters due to the signal dependent nature of the speckle intensity. In what follows, we give a brief overview on the main speckle reducing methods, providing a short description of the principle behind each family and some of its limitations are discussed. Filters that are used widely in both SAR and ultrasound imaging include the following categories: adaptive filters, homomorphic, anisotropic diffusion, and wavelet filtering [18]. A number of adaptive speckle filters have been proposed, and they are widely used in US image restoration because they are easy to implement and control [19]. The most cited and applied filters in this category include the Lee [20] [21] [22], Frost [23], Kuan [24], and Gamma maximum a posterior (MAP) filters [25]. Many improvements of these classical filters have been proposed since [26]. The Lee and Frost filters have the same structure, whereas the Kuan filter is a generalization of the Lee filter. Both filters form the output image by computing the central pixel intensity inside a filter-moving window, which is calculated from the average intensity values of the pixels and a coefficient of variation inside the moving window [1]. In comparison with non-adaptive speckle filters (The best-known non-adaptive filters are those based on the use of the mean or the median), adaptive speckle filters are more successful in preserving subtle image information. Adaptive filters use weights that are dependent on the degree of speckle in the image, whereas non-adaptive filters use the same set of weights over the entire image [19]. A homomorphic filtering is used in ultrasound to sharpen features and flatten speckle variations in an image [1]. This form of filtering performs image despeckling by computing the fast Fourier transform (FFT) of the logarithmic compressed image, applying a denoising homomorphic filter function, and then performing the inverse FFT of the image. The homomorphic filter function may be constructed either using a band-pass Butterworth or a high-boost Butterworth filter [27]. Speckle reduction filtering in the wavelet domain is based on the idea of the Daubechies and Symlet wavelet and on soft-thresholding denoising [1]. Wavelet filtering exploits the decomposition of the image into the wavelet basis where only the useful wavelet coefficients are utilized and zeroes out the other wavelet coefficients to despeckle the image. Different wavelet thresholding approaches can be used [28]. Diffusion filters filter perform contrast enhancement and remove noise from an image by modifying the image via solving a PDE. The diffusion coefficient in these filters serves as the edge detector, producing high values at features and low values in homogeneous regions. So in the region where the edge is high, the diffusion will be suppressed and vice versa. Speckle reducing anisotropic diffusion (SRAD) in is formulated as an efficient anisotropic diffusion despeckling technique. The SRAD not only preserves edges but also enhances edges by inhibiting diffusion across edges and allowing diffusion on either side of the edge. SRAD is adaptive and does not utilize hard thresholds to alter performance in homogeneous regions or in regions near edges and small features. This technique was compared with the Frost filter Lee filter and homomorphic filtering and documented that anisotropic diffusion performed better [29]. Unlike the adaptive speckle filters, all the considered PDE-based approaches produce a family of resulting images based on an iterative diffusion process. However, it has no rational criteria to select the optimal stop criteria. So it may has a limitation in retaining subtle features such as small cysts and lesions in ultrasound images Nevertheless, meaningful structural details are unfortunately removed during a large number of iterations [11] [19]. The previously mentioned approaches for speckle reduction are based on the so-called locally adaptive recovery paradigm [30]. Compared with them, the NLM algorithm relies on a L2-norm between two image patches instead of pixel comparison and the pattern redundancy is not restricted to be local. This strategy leads to competitive results when compared to most of the state-of-the-art methods. However, the noise of ultrasound images cannot be considered as an AWGN. To address this problem, considering the Bayesian formulation [31] and the Loupas noise model [8] the OBNLM algorithm for speckle reduction in ultrasound images was proposed by Coupé et al. [32]. This formulation improve the denoising performances of the NLM filter for the speckle noise removing while preserving meaningful edges in compared to the compared the original NLM filter, adaptive filter, and SRAD filter where no stop criterion is needed [32] [33]. Among the many existing despeckle methods to tackle image despeckle, we may quote OBNLM technique, which has provided satisfactory results among them.

The distribution of ultrasonic speckle noise has been largely studied for many years. A realistic modeling of noise statistics of ultrasound images cannot be easily exhibited, considering the complex image formation process. Speckle is described as one of the more complex image noise models; unlike thermal and readout noise, it is non-Gaussian, object dependent, with its variance being proportional to the local field intensity [34]. In the literature there are many models have been proposed, what follows is a briefly review of the main three models for the amplitude distribution of the backscattered ultrasound.

In this section, the OBNLM is presented. In [31] [53], a Bayesian formulation of the NLM filter is used to derive a new speckle filter. The Kervrann [31] generalized NLM filter is called Bayesian NLM mean (BNL) filter. In [14], an algorithm based on maximum likelihood estimation (MLE) which allows one to deal with noises other than Gaussian, e.g., speckle noise was introduced. The Bayesian NLM framework is based on the probabilistic intensity similarity measure, in contrast to the conventional NLM relies on L₂ norm distance metrics which have a fixed structure [31] [53]. This metrics is formulated as the likelihood that the noise distributions of the two intensity observations are the same and correspond to the same scene radiance value. It considers how intensities fluctuate in the imaging system by embedding the effect of noise. This approach considered the effect the image noise distributions as a source of useful information, rather than attempting to remove the noise. Such that the similarity is high when the two intensities are both well within the noise distributions of certain true intensities, and become significantly lower otherwise. The optimal Bayesian estimator for the v(i) can be written as

u ⌢ o p t v ( i ) = ∑ j = 1 m v ( j ) p ( v ( i ) | v ( j ) ) p ( v ( j ) ) ∑ j = 1 m p ( v ( i ) | v ( j ) ) p ( v ( j ) ) (15)

Using the Bayes’s and marginalization rules, and p(v(i)|v(j)) and p(v(i)) respectively denote the distribution of v(i)|v(j) and prior distribution of v(i).

Compared to the classical NLM formulation, the OBNLM introduced the blockwise implementation and a new statistical distance for patch comparison (Pearson distance) for weight computation [32]. The restoration of a block v(B_i) based on a Bayesian NLM scheme is given by

N L ( v ) ( B i ) = 1 | V i | ∑ j = 1 | V i | p ( v ( B i ) | v ( B j ) ) p ( v ( B j ) ) v ( B j ) 1 | V i | ∑ j = 1 | V i | p ( v ( B i ) | v ( B j ) ) p ( v ( B j ) ) (16)

where:

V_i is the search window centerd on the pixel i.

B_i is the block centered on the pixel i.

v(B_i) is the vector containing the intensities of the block B_i.

NL(v)(B_i) is the vector containing the restored value of B_i.

p(v(B_i)|u(B_j)) and p(v(B_j)) respectively denote denotes the distribution of v(B_i) conditionally to v(B_j) and the prior distribution of v(B_j).

In the case of a white Gaussian noise, the likelihood p(u(B_i)|u(B_j)) is proportional to

p ( v ( B i ) | v ( B j ) ) ∝ exp ( − ‖ v ( B i k ) − v ( B j ) ‖ h 2 ) (17)

The likelihood can be factorized for a block considers the noise model (7) is obtained by:

p ( v ( B i ) | v ( B j ) ) ∝ exp ( − ‖ v ( B i ) − v ( B j ) ‖ v ( B j ) h 2 ) (18)

So instead of the usual L₂-norm, the Pearson distance defined as

d P ( v ( B i ) − v ( B j ) ) 2 = ( v ( B i ) − v ( B j ) ) 2 v ( B j ) (19)

This similarity measure is better for US images than the L₂-norm because it takes into account the impact of the nature of the speckle noise in an ultrasound image.

The performance of machine learning methods is heavily influenced by the different forms of data representation on which they are applied [54] [55]. There are many applications of representation learning algorithms such as speech recognition, object recognition, visualization, data compression and natural language processing. We refer the reader to [54] for extensive details about these learning methods. PCA is the most widely used technique for dimension reduction. it is an unsupervised linear method used to learn a lower representation of the data by a linear projection of the input data features onto a set of new features which are a linear function of all of the original features [56] [57] [58]. PCA is useful for data visualization tasks, by projecting a very large dimensional data (for example gene expression) onto a lower dimensional space that can be more readily and quickly visualize how these data are related to each other and get better sense of what data looks like [59] [60]. The PCA transform can be used for image compression without losing important properties of the data [61] [62]. Lower dimensional framework is used in face recognition to transform the image into what is called Eigen-faces. Using fewer Eigen images is computational efficiency and provides better ability to learn and recognize new faces [63]. The image (e.g., face photo), can be represented as a linear combination of basis images. The number of basis images can be much smaller than the original collection. By using few PC’s, we can capture more than 50% of the variance in the image. This presentation is very useful in face recognition algorithm [63].

In 1965, G. Golub and W. Kahan introduced the singular value decomposition (SVD) as a decomposition technique for calculating the singular values, pseudo-inverse and rank of a matrix [64]. PCA can be efficiently computed via the SVD of the data matrix. In case of having large matrix, the SVD is an efficient, numerically stable and fast technique to solve the eigenvalue problem by matrix factorization [65]. Frequently use low-rank approximation of the data matrix is considered as noise reduction process. By discarding the later PCs which are treated as noise factors and projecting back the dataset into the original observation space will remove the noise. The latent structure of the data might be masked by noisy dimensions. In image denoising, it may be an efficient way to transform the local image patches in different representation coefficients. The idea behind this strategy is to decompose the local image patches, select the clean coefficients, and then reconstruct them. Several authors have shown that to decompose an image in wavelet base is superior for image denoising [66] [67] [68] [69] [70]. The independent components (ICs) was applied on the on local image patches to derive locally basis set [71]. The free noise structures of input data can be obtained by preserving the most informative features and removing the outliers which are associated with the least important dimensions [72]. The PCA can be used to find the less noise pre-image of the image patch in the (Eigen) space, and then calculate similarity measurement of the NLM filter in lower-dimensional space. It has shown in [73], that PCA combined with patches was an efficient filter for images corrupted by Poisson noise and outperformed the Poisson-NLM method. Based on PCA-NLM, two authors [16] [74] have proposed simultaneously similar idea for Gaussian noise reduction. They suggested firstly projecting the noisy patch to the most important PC’s first and then computing the similarity. Their hypothesis has significantly improved the denoising effect and the computation cost. Furthermore, computing distances in few dimensions and eliminating fraction of patches around the pixel under processing using the L₂ of the rank-1 approximations has accelerated the NLM algorithm [16]. However, the PCA is not always good enough for learning representation of the data. The PCA is purely second order representation and processing of the data, but it is observed that more information in the images is in the higher order of the data. In other words, the Eigen space only utilizes the gray scale information variance that may weakly related to the properties of the natural data. However, presence of noise and distortion affects the calculated principal components and hence the overall performance of SVD [75] [76]. Our work [77] [78] have showed, that the SVD dimension reduction is powerful only when images are contaminated by small amounts of noise. However, when the noise level is particularly high, the treatment relying on linear PCA transform is no longer relevant. The kernel method is one of the promising methods, which have attracted significant attention in the area of the machine learning. It is like opposite way to the dimension reduction approach, that instead of small number of PCs the kernel method can get benefit from the more features. It comes up with a different idea for similarity measurement in high dimensional space, helps in keeping the similar thing together and dissimilar thing apart. Appendix A details the theory of the kernel method. It is first appeared in the form of support vector machine (SVM) which is one of the powerful binary classification algorithms. For non-linearly separable data, the kernel method helps researchers to build an efficient SVM linear classifier in high-dimensional feature space [79]. The data points become linearly separable in the higher dimensional space, which is not the case in the lower-dimensional space. In the original space, the higher-dimensional data points are not visualized, but they are projected down to the lower dimensional space. The kernel technique is not restricted only to SVM. However various algorithms in machine learning can be enhanced with the use of the kernel method. In face recognition, the kernel technique is usually used to transform the image into what is called Eigen-faces in order to enhance the matching process and improving the ability to learn to recognize new faces [80]. Kernel PCA is a nonlinear version for computing singular vectors in high dimensional space [81]. That, the kernel PCA can come with a better encoding of the information. Furthermore it is more robust to the presence of the noise than the linear PCA [77] [82] [83] [84]. So, the kernel method representation can produce a robust subspace to the presence of the noise as well a rich informative encoding of the image patches. As it will be shown in the following sections, this approach will highlight similarities between different patches, which are the core of the NLM algorithm. Therefore, increase the performance of NLM algorithm.

This section presents a detailed description of the implementation of the proposed speckle noise reduction scheme. The core of the proposed method concentrates on finding the right representation of the patches which will highlight the matching process of the noisy image patches. Thus, better denoising performance can be obtained when similarity between pixels is computed in free noise space. The nonlinear projective approach produced by kernel PCA can remove the difficulties faces the NLM in the presence of large noise by using an appropriate kernel function and represent image data to a relatively high dimensional space. As it will be shown in the result section, this approach removes the speckle noise and enhanced the contrast of the input US images. The information about the US images used in the experiment, and the quality image metrics is presented below. Our main methodology involves the comparison of results produced by the OBNLM and kernel PCA-NLM.

In this section, we evaluate the performance of our speckle reduction and the OBNLM filters on the data set, and the reasons for these findings are presented. All the experiments and visualization of the results are implemented using MATLAB software (Math works).

The above experimental results have proved that the two filters were an efficient despeckle way, while it is not surprising, the kernel PCA-NLM essentially achieved the best despeckling effect. It was able to reduce noise in a homogenous region and enhanced the US the image contrast. The kernel PCA obtained better visual inspection as well as higher ENL and CNR values surpassing the OBNLM method which it still damaged image details, especially, the tiny boundary. In this section, the causes of the finding of all images used in experiments are discussed.

The OBNLM despeckle algorithm was considered as one of the state-of-the-art despeckling methods based on Bayesian estimator framework to derive the NLM method. The OBNLM filter is based on clever idea which takes the noise properties into account to calculate the similarity between the image patches using the Bayesian frame. The probabilistic similarity measure implemented in the in the OBNLM filter better indicate the confidence of a match than the L₂ norm, in a meaningful way, that leads to improve performance of NLM filter. Furthermore, it employed a new statistical distance measure which considers the impact of the noise as a relevant criterion for patch comparison. The main drawback of the NLM filter that it determines the pixel similarity based on the noisy image patches thereby leading to inaccurate filtering. This drawback existed in OBNLM filter which made a strong assumption that noisy observation itself provides a good approximation on the true intensity signal [93]. Since, the weights are computed directly based on the observed noisy image patches, which can deviate much from the true value, As a result, the weights become sensitive to the noise and lessens the efficiency of the OBNLM.

It is not surprising the kernel PCA-NLM achieved the best despeckling performance, as we have mentioned previously, that minimizing the L₂ cost improves NLM algorithm performance. The similarity computation operation in the NLM filters is enhanced with the more cleanly and informative representation of the image patch in the higher dimensional space. The kernel SVD can extract a larger number of principal components that can exceed the data dimensionality [86] [87]. In term of NLM, the high-dimensional feature induced by kernel PCA, the number of dimensions is equal to the number of patches in the neighbourhood, which can be quite large than the original patch size. This higher dimensional representation helps in keeping the similar thing together and dissimilar thing apart in the nonlinear subspaces of produced by the kernel PCA. This is similarly to what we mention in section 6 that the kernel method enhances the classification performance of the SVM algorithm by creating the kernel based SVM classifier [79].

The higher representation produced by the kernel PCA is robustness to the presence of noise. When adding more PCs using the kernel PCA, more reliable feature is reconstructed without getting the noise yet. Since, there are many structures choosing from many PCs [94].

The higher dimensional encoding provided by the kernel concept gives the potential of getting clean features and removing some of the eigenvectors where the noisy part of data resides [83].

Since, the kernel space produces more features sitting on their correct positions and provides less noisy features. This higher-dimensional space provided by the kernel PCA is similar to what is called super resolution idea [95], where the smear structures like edges or corners within the low resolution image become sharper in the high-resolution image. That, the kernel method scales up of the features dimension and lead to produce a high-resolution image from a blurred one [86]. This is another interpretation of why the higher dimensional feature space since lead to a better denoising. The kernel method is capable of capturing part of the higher-order statistics which are particularly important for encoding image structure [86] [96] [97]. The polynomial kernel is another popular kernel [98], and the high dimensional space by this kernel may contain all possible interactions of among the pixels of different orders as it is shown in (A-24). In other words,, the kernel representation also can take into accounts higher-order statistics of the pixels like that the relation between the more than two pixels in edges or curves [99]. All that we have mentioned before are our interpretations of why the kernel PCA-NLM have provided a better speckle noise reduction and enhanced identification of the edges in compare to the OBNLM method.

In this paper, an extension of the NLM is proposed for US images degraded by speckle noise. Our novel restoration scheme for US images is guided by learning a good encode of the image patches using the kernel PCA-NLM adapted to the statistics of speckle noise. The similarity computation operation in the NLM filters is enhanced with the informative representation of the image using Kernel PCA. Experiments were carried out on phantom data and real US images. In comparison to the OBNLM technique, the experimental results have shown that the proposed filter brought significant improvement in terms of ENL, and CNR and visual inspection.

Although the proposed method performs better than the OBNLM techniques, the scope for improvement still exists. What follows are some suggestions to improve the quality of this work. Capturing the higher-order statistics for the image processing by the kernel method is more efficient and accurate but has computational complexity. Since kernel PCA requires storing and manipulating the kernel matrix the size of which is the square of the number of training patches in the search window. So in future work, modifications may be incorporated to reduce the computation time. To make the algorithm faster methods such as the Kernel Hebbian Algorithm (KHA) can be used [86] [100]. Enriching our understanding of higher-order statistics, using the kernel method, can help in learning a better expressive image representation. The same study can be also conducted by investigating different kernel representations which can give a better representation of the pattern of the images, and hence improve the performance of the proposed denoising method.

The authors declare no conflicts of interest.

Salih, M.E., Zhang, X.M. and Ding, M.Y. (2022) Kernel PCA Based Non-Local Means Method for Speckle Reduction in Medical Ultrasound Images. Open Access Library Journal, 9: e8618. https://doi.org/10.4236/oalib.1108618

The PCA is the main linear technique for dimension reduction, which is used in various areas applications such as data visualization and image compression, noise reduction [101]. The philosophy of data reduction, given data points in n-dimensional space is to project the data point into lower dimensional space while preserving as much information as possible. The SVD is an efficient algorithm to perform the PCA. It is different from eigenvector calculating thing, and it is available in Matlab. Here, we detail the principles of SVD. Then, we give the definition of the kernel method and how to extract the significant dimensions of the data set. Finally, we describe how to calculate the kernel SVD high dimensional feature using the kernel theory.

In this section, steps of how to calculate the SVD are presented. Each step is expressed mathematically, in addition to the corresponding Matlab command.

Given a database X = { x 1 , ⋯ , x m } , consist of m x i features, (for example, a concatenation of

Given matrix X ∈ ℝ m × n , SVD consist of the product of three matrices Σ, U, and V:

X = U Σ V T (A-1)

where:

Σ ∈ ℝ n × n is a diagonal matrix with positive singular values σ i in the diagonal, regard that singular values go form largest to smallest: σ 1 > σ 2 > ⋯ > σ n .

U ∈ ℝ m × n is the left eigenvector matrix, and it has an orthonormal columns.

V T ∈ ℝ m × n is the right eigenvector matrix, and has an orthonormal rows.

X^TX and XX^T are symmetric matrices by definition

( X X T ) T = X T X T T = X T X (A-2)

Computing the eigenvector of XX^T and X^TX and is the way to estimate U, and V matrices using the definition of SVD in, respectively. While Σ is constructed by estimating the positive square roots of the of the nonzero eigenvalues of X^TX, or XX^T

X T X = ( U Σ V T ) T U Σ V T = ( V T T Σ T U T ) U Σ V T = V Σ U T U Σ V T = V Σ 2 V T (A-3)

X X T = U Σ V T ( U Σ V T ) T = U Σ V T ( V T T Σ T U T ) = U Σ V T V Σ U T = U Σ 2 U T (A-4)

We have m singular vectors (instead of eigenvector). The left singular vectors have size m, and the right singular vector have size n.

One of the SVD properties is to produce orthogonal matrices. V is orthogonal, i.e. V T = V − 1 , V V T = I , where I is the identity matrix. But, it is not true that U U T = I , and so we do not called U an orthogonal matrix. Some code adds some extra artificial columns to U matrix and zeroes Σ matrix for dimension match. This it does by a process called (Gram?Schmidt process), then U will have orthonormal columns, and U^TU is just the identity matrix. Σ is a diagonal matrix, thus it has zeroes off diagonal everywhere. It is n × n (where n is the original dimension of the data). Each value in Σ is an eigenvalue for one of the eigenvectors.

In summary, the SVD essentially give the PC’s in V^T, and it gives you each of their eigenvalues in Σ Most of implementation of SVD actually goes to the extra step of sorting the rows of the V^T and Σ, so the eigenvectors values goes form largest to smallest.

U, Σ, and V matrices are generated in Matlab by:

[ U , Σ , V ] = svd ( X ) .

The statistical property of the SVD is based on representation of data in subspaces of significant [16]. This is core property to derive the less noise image features. The lower dimension representation of the data can be obtained by using a truncated SVD to get a compressed version of the data matrix. This truncation can do by keeping the first d singular values. Using form I, the reduction of X is given as

X d = U d ∑ d V d T (A-5)

where X^d is referred to as the rank d approximation of X, or the “Reduced SVD” of X. for example, If we eliminate dimensions by keeping the three largest singular values, this is a rank 3 approximation [102]. Now are going to look how to use SVD to do some cool stuff in data processing.

The measure of distance is an important routine in data processing and analysis. One of the mostly used dissimilarity measure is the Euclidean distance. It is defined as the L₂ norm (square root of the vector inner product) of the difference of the two vectors or two points. If the similarity is interpreted as a covariance, then is the Euclidean distance could be written as a similarity matrix.

d i , j 2 = ‖ y i − y j ‖ 2 2 = 〈 ( y i − y j ) 2 〉 = ( y i − y j ) ( y i − y j ) T = y i y i T + y j T y j − 2 y i y j T (A-6)

If the covariance is of the form:

k i , j = y i , : T y j , : then d i , j 2 = k i , i + k j , j − 2 k i , j d i , j 2 = { 0 if y i = y j 2if y i ≠ y j (A-7)

So a concept called kernel rises up, which it considered as a transformation between distance and similarity matrix, and it becomes a basic for a number of algorithms in machine learning algorithms. An explanation of the kernel method theorem is introduced in the following section.

Kernel method comes up with a different idea for similarity measurement. And the difference is that, the kernel calculates distance in the space of transformed feature. Given the transformation (Φ), that maps the data from original feature space to some higher dimensional feature space. As shown in the FigureA-1, Φ takes points x_i and x_j mapped them into a Gaussian centered on x_i, and x_j, respectively.

Feature map (Φ) Φ : X → ℝ X , x ↦ k ( . , x ) .

Φ : X ↦ H k ( x i , x j ) = 〈 Φ ( x i ) , Φ ( x j ) 〉 (A-8)

Distance in transformed feature space is computed as the following:

‖ Φ ( x i ) − Φ ( x j ) ‖ 2 2 = 〈 Φ ( x i ) − Φ ( x j ) , Φ ( x i ) − Φ ( x j ) 〉 = 〈 Φ ( x i ) , Φ ( x i ) 〉 − 2 〈 Φ ( x i ) , Φ ( x j ) 〉 + 〈 Φ ( x j ) , Φ ( x j ) 〉 = k ( x i , x i ) − 2 k ( x i , x j ) + k ( x j , x j ) (A-9)

The kernel is the same as a dot product of mapped features. And in a sense of similarity measurement the kernel function is a symmetric function that maps a pair features and give a real number. The kernel gives large number value if the two inputs are similar, whereas in contrast low value if the inputs are dissimilar.

Riesz’ representation theorem tells whenever, there is a linear continuous function (f) it can be represented as a dot product with other some element of Hilbert space (H). The H can be defined as an inner product space that is complete and separable with respect to the norm defined by the inner product.

The Riesz’ representation theorem states that, there is an element r x ∈ H can be written as:

〈 r x , f 〉 = f ( x ) (A-10)

Using Riesz’ representation theorem the kernel can be defined as a reproducing kernel Hilbert space (RKHS) if:

Given kernel k : X × X → ℝ , one can construct the RKHS as the completion of the space of functions spanned by the set with an inner product defined as follows.

Consider:

f ( . ) = ∑ i α i k ( . , x i ) (A-12)

g ( . ) = ∑ j β j k ( . , x j ) (A-13)

〈 f , g 〉 = ∑ i , j β j α i k ( x j , x i ) (A-14)

Note that 〈 f , k ( . , x ) 〉 = ∑ i α i k ( x , x i ) = f ( x ) (k has the reproducing property).

Testing that 〈 f , g 〉 is an inner product is by checking the following conditions:

1) Symmetry

〈 f , g 〉 = ∑ i , j α i β j k ( x i , x j ) = ∑ i , j β j α i k ( x j , x i ) = 〈 g , f 〉

2) Positive definiteness

〈 f , f 〉 = α K α ≥ 0

〈 ∑ i α i k ( . , x i ) , ∑ i α i k ( . , x i ) 〉 = ∑ i j α i α j k ( x i , x j ) ≥ 0 (A-15)

So as long as we define a kernel function and construct the kernel matrix and it is positive definite kernel. It means we could find a mapping such that it is possible to rewrite the kernel function in term of inner product of the mapped features. Conversely, for every RKHS there exists an associated reproducing kernel which is symmetric and positive definite (PD). The mercer kernels is the family of kernel functions for which the kernel trick is guaranteed to work independent of the data you have looking at.

if Φ ( x ) is an extremely high dimensional, constructing the kernel need to represent the extremely high dimensional feature vector and then computing the inner products in the feature space which seem computationally inefficient and very expensive. However, by using kernel trick, it is possible to compute the inner product between these two vectors very inexpensively, and easily working in feature spaces even if they are very high dimensional. So for computing the distances, if given a kernel function, the need is just only evaluating the kernel and gets a value for the transformed inner product without telling how Φ ( x ) basically look like. That means there is no need to think about Φ ( x ) and build inner product but just about the kernel. From the computation point of view, the evaluation of the kernel function is much easier than the computation of the transformation of the feature followed by the inner product computation which is more complex. So just by evaluating the kernel k ( x i , x j ) and knowing that there is a map and inner product). An illustration of the computation expense of computing dot product in the feature space, and basic idea of kernel trick is given in the following example. The example shows that the inner products in the feature space could be evaluated implicitly in the input space. Assuming there is a transformation mapping from original two dimensions features to some higher three dimensional set of features

Φ : ℝ 2 → ℝ 3 ( x 1 , x 2 ) ↦ ( z 1 , z 2 , z 3 ) : = ( x 1 2 + 2 x 1 x 2 + x 2 2 ) (A-16)

where Φ ( x ) is represented in term quadratic form (all possible products of pairs of the components of the variable x.

In this case (n = 2) is a dimension of x, the length of Φ ( x ) is n 2

・ O ( n 2 ) is needed just to compute Φ ( x )

Φ ( x ) = ( x 1 x 1 x 1 x 2 x 2 x 1 x 2 x 2 ) ≈ ( x 1 2 2 x 1 x 2 x 2 2 ) (A-17)

・ Then O ( n ) is needed to compute the kernel which is inner product in the feature space.

k ( x i , x j ) = 〈 Φ ( x i ) , Φ ( x j ) 〉 = ( x 1 2 , 2 x i 1 x i 2 , x i 2 2 ) ( x j 1 2 , 2 x j 1 x j 2 , x j 2 2 ) Τ = 〈 x i , x j 〉 2 = x i 1 2 x j 1 2 + 2 x i 1 x i 2 x j 1 x j 2 , x i 2 2 x j 2 2 = ( x i 1 x j 1 ) 2 + 2 ( x i 1 x j 1 x i 2 x j 2 ) + ( x i 2 x j 2 ) 2 (A-18)

The inner product in the feature space can evaluate in the input space, because of the followings:

・ The x i 1 x j 1 and x i 2 x j 2 are the dot product terms taken in the input space.

・ The ( x i 1 x j 1 + x i 2 x j 2 ) 2 is the dot product term taken in the input space raised to power of 2.

As it is mentioned before that the kernel function has the property that it correspond to dot product in some other representation rather than the input space, and here the other representation is simply the representation computing the all product of order 2. And that it means the dot product in H can be computed in R². The example showed that the kernel trick is very computationally efficient k ( x i , x j ) is defined as 〈 x i , x j 〉 2 , i.e. it is just take the inner product between x_i and x_j which is O(n), then square that and the kernel function is computed, and implicitly working in an extremely high dimensional computing feature space [103]. In the above example examined only the 2D case, but the n dimensional case is just generalization of the 2D case.

x i , x j ∈ ℝ n 〈 x i , x j 〉 r = ( ∑ p = 1 n x i , p x j , p ) r = 〈 Φ ( x i ) , Φ ( x j ) 〉 r (A-19)

So you have n dimensional x_i and x_j space vectors, then calculate the dot product (get a single number) and raise it to power of r. So we get a simple summation operation, and it does not matter if the r is big or large, since we get the same computation complexity [104].

The typical kernel functions that can express the similarity between the x_i and x_j are:

・ Linear: (i.e. there is no transformation, but just it is computing the inner product of two input vectors)

k ( x i , x j ) = 〈 x i , x j 〉 (A-20)

・ Gaussian radial basis function

k ( x i , x j ) = exp ( − ‖ x i − x j ‖ 2 2 σ 2 ) (A-21)

It is considered as one of the preferred kernel function, which computes the Gaussian with square distance between x_i and x_j. It takes the points and mapped them into a Gaussian function centered on the x_i, and x_j points as shown in the FigureA1. In other words, each point is represented by its similarity to all other points [79]. This means that a high dimensional vector can be obtained by evaluating the kernel function on these finite set of points.

This section addresses one of the main applications of the kernel method which explain why kernel method could be useful. The support vector machine (SVM) is one of the powerful classification algorithms, which looking for a decision surface that separates between the two groups of the data points. These points (for example faces and non faces data points) are referred to as the support vectors. FigureA2(b) shows separable training data sets, it seem to it is impossible to use a linear separator. Thus a more complicated (curve instead of line) nonlinear classifier is needed. Applying the kernel trick is a way to create kernel based SVM classifiers, and this allows the algorithm to separate the data points using a hyper plane in a transformed feature space.

Consider a map into a feature space (higher dimensional, 3D) the new coordinates are product of old two coordinate.

Φ : ℝ 2 → ℝ 3 ( x 1 , x 2 ) ↦ ( z 1 , z 2 , z 3 ) : = ( x 1 2 + 2 x 1 x 2 + x 2 2 ) (A-22)

So there is some amount of red data points appear close to the mean in the 2D representation FigureA2(a). But they are actually sitting away from the mean in 3D space.

This means if the high dimensional data are not visualized, then the some of the data is seem to projecting down to the mean. The cross and circle data points are not linearly separable in the original input space, and using kernel they can be linearly separable in 3D high-dimensional feature space. See FigureA2(b)

As mentioned above, that SVD is a way to do the eigenvalue problem by matrix factorization. The kernel method comes up with a different idea, that the kernel represents a dot product in higher dimension space using some nonlinear transformation of the input matrix [81]. A description of the kernel SVD algorithm is presented in this section.

X z = U z Σ z V z T (A-23)

X z = Φ ( X ) (A-24)

So computing the eigenvector of X z X z T and X z T X z is the way to estimate U z , Σ z , V z T matrices, respectively using the definition of SVD in (A-3) and (A-4). The computation of X z X z T or X z T X z (the inner product of transformed matrix) can be implemented implicitly in the input space using kernel trick.

K = X z X z T = U z Σ z 2 V z T (A-25)

K = X z T X z = V z Σ z 2 V z T (A-26)

Kernel SVD does the decomposition in terms of inner product of the transformed feature, which modeled by a kernel function. In other words the inner products X z X z T and X z T X z are replaced by kernels to yield a nonlinear version of SVD. And the decomposition of K will produce the U z , Σ z , V z T matrices which have properties similar to the properties of U , Σ , V T matrices.

K = U h Σ h V h T (A-27)