Density estimation methods provide a faithful estimate of a non-observable probability density function based on a given collection of observed data [1] [2] [3] [4]. The observed data are treated as random samples obtained from a large population which is assumed to be distributed according to the underlying density function. The aim of our current work is to show that the joint density function of the gradient of a sufficiently smooth function S (density function of ∇ S ) can be obtained from the normalized power spectrum of ϕ = exp ( i S τ ) as the free parameter τ tends to zero. The proof of this relationship relies on the higher order stationary phase approximation [5] - [10]. The joint density function of the gradient vector field is usually obtained via a random variable transformation, of a uniformly distributed random variable X over the compact domain Ω ⊂ ℝ d , using ∇ S as the transformation function. In other words, if we define a random variable Y = ∇ S ( X ) where the random variable X has a uniform distribution on the domain Ω ( X ∼ U N I ( Ω ) ), the density function of Y represents the joint density function of the gradient of S.

In computer vision parlance—a popular application area for density estimation—these gradient density functions are popularly known as the histogram of oriented gradients (HOG) and are primarily employed for human and object detection [11] [12]. The approaches developed in [13] [14] demonstrate an application of HOG—in combination with support vector machines [15] —for detecting pedestrians from infrared images. In a recent article [16], an adaption of the HOG descriptor called the Gradient Field HOG (GF-HOG) is used for sketch-based image retrieval. In these systems, the image intensity is treated as a function S ( X ) over a 2D domain, and the distribution of intensity gradients or edge directions is used as the feature descriptor to characterize the object appearance or shape within an image. In Section 5 we provide experimental evidence to showcase the efficacy of our method in computing the density of these oriented gradients (HOG). The present work has also been influenced by recent work on quantum supremacy [17] [18] [19]. Here, the aim is to draw samples from the density function of random variables corresponding to the measurement bases of a high-dimensional quantum mechanical wave function. This work may initially seem far removed from our efforts. However, as we will show, the core of our density estimation approach is based on evaluating interval measures of the squared magnitude of a wave function in the frequency domain. For this reason, our approach is deemed a wave function approach to density estimation and henceforth we refer to it as such.

In our earlier effort [20], we primarily focused on exploiting the stationary phase approximation to obtain gradient densities of Euclidean distance functions (R) in two dimensions. As the gradient norm of R is identically equal to 1 almost everywhere, the density of the gradient is one-dimensional and defined over the space of orientations. The key point to be noted here is that the dimensionality of the gradient density (one) is one less than the dimensionality of the space (two) and the constancy of the gradient magnitude of R causes its Hessian to vanish almost everywhere. In Lemma 2.3 below, we see that the Hessian is deeply connected to the density function of the gradient. The degeneracy of the Hessian precluded us from directly employing the stationary phase method and hence techniques like symmetry-breaking had to be used to circumvent the vanishing Hessian problem. The reader may refer to [20] for a more detailed explanation. In contrast to our previous work, we regard our current effort as a generalization of the gradient density estimation result, now established for arbitrary smooth functions in arbitrary finite dimensions.

We begin with a compact measurable subset Ω of ℝ d on which we consider a smooth function S : Ω → ℝ . We assume that the boundary of Ω is smooth and the function S is well-behaved on the boundary as elucidated in Appendix B. Let H x denote the Hessian of S at a location x ∈ Ω and let det ( H x ) denote its determinant. The signature of the Hessian of S at x , defined as the difference between the number of positive and negative eigenvalues of H x , is represented by σ x . In order to exactly determine the set of locations where the joint density of the gradient of S exists, consider the following three sets:

A u = { x : ∇ S ( x ) = u } , (2.1)

B = { x : det ( H x ) = 0 } , (2.2)

and

C = { ∇ S ( x ) : x ∈ B ∪ ∂ Ω } . (2.3)

Let N ( u ) = | A u | . We employ a number of useful lemma, stated here and proved in Appendix A.

Lemma 2.1. [Finiteness Lemma] A u is finite for every u ∉ C .

As we see from Lemma 2.1 above, for a given u ∉ C , there is only a finite collection of x ∈ Ω that maps to u under the function ∇ S . The inverse map ∇ S ( − 1 ) ( u ) which identifies the set of x ∈ Ω that maps to u under ∇ S is ill-defined as a function as it is a one to many mapping. The objective of the following lemma (Lemma 2.2) is to define appropriate neighborhoods such that the inverse function ∇ S ( − 1 ) , required in the proof of our main Theorem 3.2, when restricted to those neighborhoods is well-defined.

Lemma 2.2. [Neighborhood Lemma] For every u 0 ∉ C , there exists a closed neighborhood N η ( u 0 ) ¯ around u 0 such that N η ( u 0 ) ¯ ∩ C is empty. Furthermore, if | A u 0 | > 0 , N η ( u 0 ) ¯ can be chosen such that we can find a closed neighborhood N η ( x ) ¯ around each x ∈ A u 0 satisfying the following conditions:

1) ∇ S ( N η ( x ) ¯ ) = N η ( u 0 ) ¯ .

2) det ( H y ) ≠ 0, ∀ y ∈ N η ( x ) ¯ .

3) The inverse function ∇ S x ( − 1 ) ( u ) : N η ( u 0 ) ¯ → N η ( x ) ¯ is well-defined.

4) For y , z ∈ N η ( x ) ¯ , σ y = σ z .

Lemma 2.3 [Density Lemma] Given X ∼ U N I ( Ω ) , the probability density of Y = ∇ S ( X ) on ℝ d − C is given by

P ( u ) = 1 μ ( Ω ) ∑ k = 1 N ( u ) 1 | det ( H x k ) | (2.4)

where x k ∈ A u , ∀ k ∈ { 1,2, ⋯ , N ( u ) } and μ is the Lebesgue measure.

From Lemma 2.3, it is clear that the existence of the density function P at a location u ∈ ℝ d necessitates a non-vanishing Hessian matrix ( det ( H ) ≠ 0 ) ∀ x ∈ A u . Since we are interested in the case where the density exists almost everywhere on ℝ d , we impose the constraint that the set B in (2.2), comprising all points where the Hessian vanishes, has zero Lebesgue measure. It follows that μ ( C ) = 0 . Furthermore, the requirement regarding the smoothness of S ( S ∈ C ∞ ( Ω ) ) can be relaxed to functions S in C d 2 + 1 ( Ω ) where d is the dimensionality of Ω as we will see in Section 3.2.2.

Define the function F τ : ℝ d → ℂ as

F τ ( u ) = 1 ( 2 π τ ) d 2 μ ( Ω ) 1 2 ∫ Ω exp ( i τ [ S ( x ) − u ⋅ x ] ) d x (3.1)

or τ > 0 . F τ is very similar to the Fourier transform of the function exp ( i S ( x ) τ ) . The normalizing factor in F τ comes from the following lemma (Lemma 3.1) whose proof is given in Appendix A.

Lemma 3.1. [Integral Lemma ] F τ ∈ L 2 ( ℝ d ) and ‖ F τ ‖ 2 = 1 .

The power spectrum defined as

P τ ( u ) ≡ F τ ( u ) F τ ( u ) ¯ (3.2)

equals the squared magnitude of the Fourier transform. Note that P τ ≥ 0 . From Lemma (3.1), we see that ∫ P τ ( u ) d u = 1 . Our fundamental contribution lies in interpreting P τ ( u ) as a density function and showing its equivalence to the density function P ( u ) defined in (2.4). Formally stated:

Theorem 3.2. For u 0 ∉ C ,

lim α → 0 1 μ ( N α ( u 0 ) ) lim τ → 0 ∫ N α ( u 0 ) P τ ( u ) d u = P (u0)

where N α ( u 0 ) is a ball around u 0 of radius α .

Before embarking on the proof, we would like to emphasize that the ordering of the limits and the integral as given in the theorem statement is crucial and cannot be arbitrarily interchanged. To press this point home, we show below that after solving for P τ , the lim τ → 0 P τ does not exist. Hence, the order of the integral followed by the limit τ → 0 cannot be interchanged. Furthermore, when we swap the limits of and, we get

which also does not exist. Hence, the theorem statement is valid only for the specified sequence of limits and the integral.

The characteristic function for the random variable is defined as the expected value of, namely

Here denotes the density of the uniformly distributed random variable X on.

The inverse Fourier transform of a characteristic function also serves as the density function of the random variable under consideration [30]. In other words, the density function of the random variable Y can be obtained via

Having set the stage, we can now proceed to highlight the close relationship between the characteristic function formulation of the density and our formulation arising from the power spectrum. For simplicity, we choose to consider a region that is the product of closed intervals,. Based on the expression for the scaled Fourier transform in (3.1), the power spectrum is given by

Define the following change of variables,

Then and the integral limits for and are given by

where is the i^th component of. Note that the Jacobian of this transformation is. Now we may write the integral in terms of these new variables as

where

The mean value theorem applied to yields

where

with. When is fixed and, and so for small values of we get

Again we would like to drive the following point home. We do not claim that we have formally proved the above approximation. On the contrary, we believe that it might be an onerous task to do so as the mean value theorem point in (4.5) is unknown and the integration limits for directly depend on. The approximation is stated with the sole purpose of providing an intuitive reason for our theorem (Theorem 3.2) and to provide a clear link between the characteristic function and wave function methods for density estimation.

Furthermore, note that the integral range for depends on and so when, as and hence the above approximation for

in (4.6) might seem to break down. To evade this seemingly ominous problem, we manipulate the domain of integration for as follows. Fix an and let

where

By defining as above, note that in, is deliberately made to be and hence as. Hence the approximation for in (4.6) might hold for this integral range of. For consideration of, recall that Theorem 3.2 requires the power spectrum to be integrated over a small neighborhood around. By using the true expression for from (4.4) and performing the integral for prior to and, we get

Since both in (4.7) and the lower and the upper limits for, namely respectively approach as, the Riemann-Lebesgue lemma [21] guarantees that, the integral

approaches zero as. Hence for small values of, we can expect the integral over to dominate over the other. This leads to the following approximation,

as approaches zero. Combining the above approximation with the approximation for given in (4.6) and noting that the integral domain for and approaches and respectively as, the integral of the power spectrum over the neighborhood at small values of in (4.3) can be approximated by

This form exactly coincides with the expression given in (4.2) obtained through the characteristic function formulation.

The approximations given in (4.6) and (4.8) cannot be proven easily as they involve limits of integration which directly depend on. Furthermore, the mean value theorem point in (4.5) is arbitrary and cannot be determined beforehand for a given value of. The difficulties faced here emphasize the need for the method of stationary phase to formally prove Theorem 3.2.

As we remarked before, the characteristic function and our wave function methods should not be treated as mere reformulations of each other. This distinction is further emphasized when we find our method to be computationally more efficient than the characteristic function approach in the finite sample-set scenario where we estimate the gradient density from N samples of the function S. Given these N sample values and their gradients, the characteristic function defined in (4.1) needs to be computed for N integral values of. Each value of requires summation over the N sampled values of. Hence the total time required to determine the characteristic function is. The joint density function of the gradient is obtained via the inverse Fourier transform of the characteristic function, which is an operation. The overall time complexity is therefore. In our wave function method the Fourier transform of at a given value of can be computed in and the subsequent squaring operation to obtain the power spectrum can be performed in. Hence the density function can be determined in, which is more efficient when compared to the complexity of the characteristic function approach.

We would like to emphasize that our wave function method for computing the gradient density is very fast and straightforward to implement as it requires computation of a single Fourier transform. We ran multiple simulations on many different types of functions to assess the efficacy of our wave function method. Below we show comparisons with the standard histogramming technique where the functions were sampled on a regular grid between

at a grid spacing of. For the sake of convenience, we normalized the functions such that the maximum gradient magnitude value is 1. Using the sampled values, we first computed the fast Fourier transform of at, then computed the power spectrum followed by normalization to obtain the joint gradient density. We also computed the discrete derivative of S at the grid locations to obtain the gradient and then determined the gradient density by histogramming. For better visualization, we marginalized the density along the radial and the orientation directions. The plots shown in Figure 1 provide visual, empirical evidence corroborating our theorem. Notice the near-perfect match between the gradient densities computed via standard histogramming and our wave function method. The accuracy of the density marginalized along the orientations further strengthens our claim made in Section 1 about the wave function method serving as a reliable estimator for the histogram of oriented gradients (HOG). In Figure 2, we verify the convergence of our estimated density to the true density as in accordance with Theorem 3.2.

Observe that the integrals

give the interval measures of the density functions and P respectively. Theorem 3.2 states that at small values of, both the interval measures are approximately equal, with the difference between them being which converges to zero as. Recall that by definition, is the normalized power

spectrum of the wave function. Hence we conclude that

the power spectrum of can potentially serve as a joint density estimator for the gradient of S at small values of, where the frequencies act as gradient histogram bins. We also built an informal bridge between our wave function method and the characteristic function approach for estimating probability densities, by directly trying to recast the former expression into the latter. The difficulties faced in relating the two approaches reinforce the stationary phase method as a powerful tool to formally prove Theorem 3.2. Our earlier result proved in [20], where we employ the stationary phase method to compute the gradient density of Euclidean distance functions in two dimensions, is now generalized in Theorem 3.2 which establishes a similar gradient density estimation result for arbitrary smooth functions in arbitrary finite dimensions.

As mentioned earlier, in [23] we have established error bounds in one dimension for the practical finite sample-set setting, wherein the gradient density is

estimated from a finite, discrete set of samples, instead of assuming that the function is fully described over a compact set. In the future, we plan to extend this work and derive similar finite sample error bounds for gradient density estimation in arbitrary higher dimensions.

This research work benefited from the support of the AIRBUS Group Corporate Foundation Chair in Mathematics of Complex Systems established in ICTS-TIFR.

This research work benefited from the support of NSF IIS 1743050.

The authors declare no conflicts of interest regarding the publication of this paper.

Gurumoorthy, K.S., Rangarajan, A. and Corring, J. (2019) Gradient Density Estimation in Arbitrary Finite Dimensions Using the Method of Stationary Phase. Advances in Pure Mathematics, 9, 1034-1058. https://doi.org/10.4236/apm.2019.912051

1) Proof of Finiteness Lemma

Proof. We prove the result by contradiction. Observe that is a subset of the compact set. If is not finite, then by Theorem (2.37) in [31], has a limit point. If, then giving a contradiction. Otherwise, consider a sequence, with each, converging to. Since for all n, from continuity it follows that and hence. Let and. Then

where the linear operator is the Hessian of at (obtained from the set of derivatives of the vector field at the location). As and is linear, we get

Since is defined above to be a unit vector, it follows that is rank deficient and. Hence and resulting in a contradiction.

2) Proof of Neighborhood Lemma

Proof. Observe that the set defined in (2.2) is closed because if is a limit point of, from the continuity of the determinant function we have and hence. Being a bounded subset of, is also compact. As is also compact and is continuous, is compact and hence is open. Then for, there exists an open neighborhood

for some around such that. By letting, we get the required closed neighborhood containing.

Since, points 1, 2 and 3 of this lemma follow directly from the inverse function theorem. As is finite by Lemma 2.1, the closed neighborhood can be chosen independently of so that points 1 and 3 are satisfied. In order to prove point 4, note that the eigenvalues of are all non-zero and vary continuously for. As the eigenvalues never cross zero, they retain their sign and so the signature of the Hessian stays fixed.

3) Proof of Density Lemma

Proof. Since the random variable X is assumed to have a uniform distribution on, its density at every location equals. Recall that the random variable Y is obtained via a random variable transformation from X, using the function. The Jacobian of at a location equals the Hessian of the function S at. Barring the set corresponding to the union of the image (under) of the set of points (where the Hessian vanishes) and the boundary, the density of Y exists on and is given by (2.4). Please see well known sources such as [30] for a detailed explanation.

For the sake of completeness we explicitly prove the well-known result stated in Integral Lemma 3.1.

4) Proof of Integral Lemma

Proof. Define a function by

Let. Then,

Letting and, we get

As is integrable, by Parseval’s Theorem (see [21] ) we have

By noting that

the result follows.

5) Proof of Cross Factor Nullifier Lemma

Proof. Let denote the phase of the exponential in the cross term (excluding the terms with constant signatures), i.e.,

Its gradient with respect to equals

where is the Jacobian of at whose term equals (with a similar expression for). Since, we get. This means that the phase function of the exponential in the statement of the lemma is non-stationary and hence does not contain any stationary points of the first kind. Let

Since, consider the vector field and as before note that

where is the divergence operator. Inserting (A.3) in the second line of (3.13), integrating over, and applying the divergence theorem we get

Here is the unit outward normal to the positively oriented boundary parameterized by. In the right side of (A.4), notice that all terms inside the integral are bounded. The factor outside the integral ensures that

One of the foremost requirements for Theorem 3.2 to be valid is that the function have a finite number of stationary points of the second kind on the boundary for almost all. The stationary points of the second kind are the critical points on the boundary where a level curve touches for some constant c [9] [10]. Contributions from the second kind are generally, but an infinite number of them could produce a combined effect of, tantamount to a stationary point of the first kind [9]. If so, we need to account for the contribution from the boundary which could in effect invalidate our theorem and therefore our entire approach. However, the condition for the infinite occurrence of stationary points of the second kind is so restrictive that for all practical purposes they can be ignored. If the given function S is well-behaved on the boundary in the sense explained below, these thorny issues can be sidestepped. Furthermore, as we will be integrating over to remove the cross-phase factors, it suffices that the aforementioned finiteness condition be satisfied for almost all instead of for all.

Let the location be parameterized by the variable, i.e.,. Let denote the Jacobian matrix of whose entry is given by

Stationary points of the second kind occur at locations where, which translates to

This leads us to define the notion of a well-behaved function on the boundary.

Definition: A function S is said to be well-behaved on the boundary provided (B.1) is satisfied only at a finite number of boundary locations for almost all.

The definition immediately raises the following questions: 1) Why is the assumption of a well behaved S weak? and 2) Can the well-behaved condition imposed on S be easily satisfied in all practical scenarios? Recall that the finiteness of premise (B.1) entirely depends on the behavior of the function S on the boundary. Scenarios can be manually handcrafted where the finiteness assumption is violated and (B.1) is forced to be satisfied at all locations. Hence it is meaningful to ask: What stringent conditions are required to incur an infinite number of stationary points on the boundary? We would like to convince the reader that in all practical scenarios, S will contain only a finite number of stationary points on the boundary and hence it is befitting to assume that the function S is well-behaved on the boundary. The reader should bear in mind that our explanation here is not a formal proof but an intuitive reasoning of why the well-behaved condition imposed on S is reasonable.

To streamline our discussion, we consider the special case where the boundary is composed of a sequence of hyper-planes as any smooth boundary can be approximated to a given degree of accuracy by a finite number of hyper-planes. On any given hyperplane, remains fixed. Recall that from the outset, we omit the set (i.e.,) which includes the image under of the boundary. Hence for any point for. Since the rank of Q is and is required to be orthogonal to all the rows of Q for condition 33 to hold, is confined to a 1-D subspace. So if we enforce to vary smoothly on the hyperplane and not be constant, we can circumvent the occurrence of an infinite number of stationary points of the second kind for all. Additionally, we can safely disregard the characteristics of the function S at the intersection of these hyperplanes as they form a set of measure zero. To press this point home, we now formulate the worst possible scenario where is a constant vector. Let denote a portion of where. Let and result in infinite number of stationary points of the second kind on. As is limited to a 1-D subspace, we must have for some, i.e.,. So in any given region of, there is at most a 1-D subspace (measure zero) of which results in an infinite number of stationary points of the second kind in that region. Our well-behaved condition is then equivalent to assuming that the number of planar regions on the boundary where is constant is finite.

The boundary condition is best exemplified with a 2D example. Consider a line segment on the boundary. Without loss of generality, assume the parameterization. Then. Equation (B.1) can be interpreted as where,. So if we plot the sum for points along the line, the requirement reduces to the function not oscillating an infinite number of times around an infinite number of ordinate locations. It is easy to see that the imposed condition is indeed weak and is satisfied by almost all smooth functions. Consequently, we can affirmatively conclude that the enforced well-behaved constraint (B) does not impede the usefulness and application of our wave function method for estimating the joint gradient densities of smooth functions.