Heriony Rapelanoro-Rabenja
Department of Mathematics, Applied Mathematics Laboratory, University of Fianarantsoa, Fianarantsoa, Madagascar.
DOI: 10.4236/am.2026.178031   PDF    HTML   XML   5 Downloads   44 Views   Citations

Abstract

This article develops a unified geometric and energetic framework for the analysis of deep neural networks, based on embedding the output manifold into a Reproducing Kernel Hilbert Space (RKHS). This embedding induces a natural Riemannian metric, a Levi-Civita connection, a second fundamental form, and a mean curvature vector, allowing the construction of a complete geometric energy model. We show how these tools lead to intrinsic learning dynamics, coherent geometric regularization, and physically interpretable energy flows. Experiments demonstrate improvements in stability, robustness, and generalization.

Keywords

RKHS Geometry, Riemannian Learning, Extrinsic Curvature, Mean Curvature, Geometric Regularization, Energy-Based Models

Share and Cite:

1. General Introduction

The goal of this article is to develop a unified geometric and energetic framework for the analysis of deep neural networks. The central idea is to view the output manifold of the network as an immersed submanifold of a Reproducing Kernel Hilbert Space (RKHS). This perspective naturally introduces:

• an induced Riemannian metric,

• a Levi-Civita connection,

• a second fundamental form and mean curvature,

• a complete geometric energy model,

• intrinsic learning dynamics consistent with the geometry.

This framework connects differential geometry, stochastic diffusions, energy models, and deep learning. Learning becomes a geometric flow on an immersed manifold, where geometric regularization plays a central role in stability, robustness, and generalization.

This geometric viewpoint is closely related to information geometry [1] and recent advances in geometric deep learning [2]. Foundational references on differential geometry include do Carmo [3], Lee [4], and Petersen [5]. Kernel methods and RKHS theory follow Schölkopf and Smola [6] and Wahba [7], while mathematical perspectives on deep architectures are discussed in Calin [8] and Mallat [9]. Related energetic and stochastic complexity aspects are developed in [10].

2. Output Manifold and RKHS Embedding

2.1. Network Output and Parametric Manifold

Consider a deep neural network parameterized by $\theta \in {ℝ}^m$ and a mapping

$F_{\theta }:X\to Y.$

For input points ${\left\{x_i\right\}}_{i=1}^n$ , the output is

$y\left(\theta \right)=\left(F_{\theta }\left(x_1\right),\cdots ,F_{\theta }\left(x_n\right)\right)\in {ℝ}^n.$

As $\theta$ varies, the set of outputs forms a smooth manifold:

$\mathcal{S}=\left\{y\left(\theta \right)\in {ℝ}^n;\theta \in {ℝ}^m\right\}.$

2.2. Embedding into a RKHS

To introduce a rich geometric structure, we embed the output into a RKHS $\left({\mathscr{H}}_K,{\langle \cdot ,\cdot \rangle }_K\right)$ associated with a positive kernel $K$ :

$\Phi :X\to {\mathscr{H}}_K,\Phi \left(x\right)=K\left(x,\cdot \right).$

The output manifold becomes an immersed submanifold:

${\mathcal{S}}_K=\left\{\Phi \left(y\left(\theta \right)\right);\theta \in {ℝ}^m\right\}\subset {\mathscr{H}}_K.$

This embedding provides a natural metric induced by the RKHS inner product, a smooth Riemannian structure, access to geometric tools (connection, curvature), and a physically interpretable energy.

2.3. Induced Metric

The induced metric on ${\mathcal{S}}_K$ is

$h_{\theta }\left(u,v\right)={\langle \frac{\partial \Phi \left(y\left(\theta \right)\right)}{\partial \theta }\left(u\right),\frac{\partial \Phi \left(y\left(\theta \right)\right)}{\partial \theta }\left(v\right)\rangle }_K.$

The tangent space is

$T_{\theta }{\mathcal{S}}_K=\left\{\frac{\partial \Phi \left(y\left(\theta \right)\right)}{\partial \theta }\left(v\right);v\in {ℝ}^m\right\}.$

This structure enables intrinsic gradient flows, geometric energies, and curvature-based regularization.

3. Induced Metric and Riemannian Structure

3.1. Geometric Motivation

Embedding the output manifold ${\mathcal{S}}_K$ into a RKHS naturally endows it with a Riemannian structure. This allows the definition of:

• covariant derivatives,

• geodesics,

• intrinsic and extrinsic curvature,

• intrinsic gradient flows.

These tools are essential for formulating a coherent energetic theory in which learning is interpreted as motion on a manifold equipped with an induced metric.

3.2. Illustrative Example: One-Hidden-Neuron Network

Consider the simple network

$F_{\theta }\left(x\right)=\sigma \left(wx+b\right),\theta =\left(w,b\right).$

For input points $\left\{x_i\right\}$ , the output is

$y\left(\theta \right)=\left(\sigma \left(w{x}_1+b\right),\cdots ,\sigma \left(w{x}_n+b\right)\right).$

As $\left(w,b\right)$ vary, the points $y\left(\theta \right)$ trace a smooth curve in ${ℝ}^n$ . Embedding this curve into a RKHS yields $\Phi \left(y\left(\theta \right)\right)\in {\mathscr{H}}_K$ , and the induced metric becomes

$h_{\theta }\left(u,v\right)={\langle \frac{\partial \Phi \left(y\left(\theta \right)\right)}{\partial \theta }\left(u\right),\frac{\partial \Phi \left(y\left(\theta \right)\right)}{\partial \theta }\left(v\right)\rangle }_K.$

Even this simple network exhibits nontrivial intrinsic geometry.

4. Levi-Civita Connection and Intrinsic Dynamics

4.1. Levi-Civita Connection: Formal Definition

On any Riemannian manifold $\left({\mathcal{S}}_K,h\right)$ , there exists a unique connection $\nabla$ satisfying:

• Metric compatibility:

$X\left(h\left(U,V\right)\right)=h\left({\nabla }_{X}U,V\right)+h\left(U,{\nabla }_{X}V\right),$

• Zero torsion:

${\nabla }_{U}V-{\nabla }_{V}U=\left[U,V\right].$

Theorem 4.1 (Levi-Civita Connection). There exists a unique connection satisfying the two properties above. It is called the Levi-Civita connection.

4.2. Physical Meaning of the Levi-Civita Connection

The Levi-Civita connection can be interpreted as a transport rule that is energy-preserving (metric compatibility) and twist-free (zero torsion). It corresponds to sliding along the manifold without artificial rotation.

4.3. Riemannian Gradient and Learning Dynamics

The Euclidean gradient ${\nabla }_{{\mathscr{H}}_K}E$ is not tangent to ${\mathcal{S}}_K$ . The Riemannian gradient is its tangential projection:

$\text{grad}E\left(\theta \right)={\Pi }_{T_{\theta }{\mathcal{S}}_K}\left({\nabla }_{{\mathscr{H}}_K}E\right).$

The learning dynamics becomes a geometric flow:

$\frac{\text{d}\theta }{\text{d}t}=-\text{grad}E\left(\theta \right).$

4.4. Link with Backpropagation

Let $f_{\theta }\left(x\right)\in {\mathscr{H}}_K$ denote the RKHS representation of the network output. The RKHS gradient of the energy $E$ can be written as

${\nabla }_{{\mathscr{H}}_K}E=2f_{\theta }+{\displaystyle \sum }_{i=1}^n{\nabla }_f\ell \left(f_{\theta }\left(x_i\right),z_i\right)K\left(\cdot ,x_i\right),$

where $\ell$ is the loss function. The parameter gradient is obtained by the chain rule:

${\nabla }_{\theta }E={\langle {\nabla }_{{\mathscr{H}}_K}E,\frac{\partial f_{\theta }}{\partial \theta }\rangle }_{{\mathscr{H}}_K}.$

Thus, the usual backpropagation update

${\theta }_{t+1}={\theta }_t-\eta {\nabla }_{\theta }E$

can be interpreted as the projection of the ambient RKHS gradient onto the parameter manifold. This provides a geometric bridge between the abstract RKHS gradient and the concrete parameter updates implemented in practice.

5. Second Fundamental Form and Extrinsic Curvature

5.1. Gauss Decomposition

For any point $p\in {\mathcal{S}}_K$ , we have the orthogonal decomposition:

$T_p{\mathscr{H}}_K=T_p{\mathcal{S}}_K\oplus {\left(T_p{\mathcal{S}}_K\right)}^{\perp }.$

For tangent vector fields $X,Y$ :

${\nabla }_{X}Y={\left({\nabla }_{X}Y\right)}^{\top }+{\left({\nabla }_{X}Y\right)}^{\perp }.$

5.2. Second Fundamental Form: Formal Definition

Definition 5.1. The second fundamental form is defined by:

$ℒ\left(X,Y\right)={\left({\nabla }_{X}Y\right)}^{\perp }.$

It measures how the submanifold bends within the ambient space.

5.3. Geometric Role of the Second Fundamental Form

• If $ℒ=0$ , the manifold is flat in the ambient space.

• If $ℒ$ is large, the manifold is highly curved.

• In deep networks, $ℒ$ measures extrinsic sensitivity.

5.4. Illustration of Extrinsic Curvature

The Gauss decomposition of ${\nabla }_{X}Y$ into tangential and normal components is illustrated in Figure 1, showing how the ambient direction splits into intrinsic and extrinsic parts.

Figure 1. Gauss decomposition in the RKHS: decomposition of ${\nabla }_{X}Y$ into tangential and normal components.

5.5. Geometric Energy

We define the extrinsic geometric energy:

$E_{\text{geom}}={\displaystyle {\int }_{{\mathcal{S}}_K}{\Vert ℒ\Vert }^2\text{d}\mu }.$

It penalizes excessive extrinsic deformation and encourages smoother output manifolds.

5.6. Example in ${ℝ}^3$

Consider the surface:

$S=\left\{\left(x,y,z\right)\in {ℝ}^3;z=x^2+y^2\right\}.$

At the point $p=\left(0,0,0\right)$ , the surface looks like a shallow bowl. The tangent plane is horizontal, the normal vector is $n=\left(0,0,1\right)$ , and the mean curvature vector points upward.

The paraboloid example and its upward normal at the origin are shown in Figure 2, highlighting how curvature encodes geometric deformation.

Figure 2. Illustration of the paraboloid $z=x^2+y^2$ and its upward normal at the origin.

This example illustrates how curvature encodes geometric deformation in a familiar setting and why controlling extrinsic curvature stabilizes neural representations.

6. Mean Curvature and Geometric Regularization

6.1. Mean Curvature: Formal Definition

Let $\left\{T_1,\cdots ,T_m\right\}$ be an orthonormal basis of $T_p{\mathcal{S}}_K$ . The mean curvature vector is defined as:

$H\left(p\right)={\displaystyle \sum }_{i=1}^mℒ\left(T_i,T_i\right),$

where $ℒ$ is the second fundamental form. Thus,

$H\left(p\right)\in {\left(T_p{\mathcal{S}}_K\right)}^{\perp }.$

6.2. Geometric Role of Mean Curvature

The mean curvature measures the tendency of the manifold to bend within the ambient space:

• $H=0$ : the manifold is minimal,

• large $H$ : the manifold is highly curved,

• $H$ indicates the direction of maximal bending.

6.3. Illustration of Mean Curvature

Figure 3 depicts the mean curvature as the sum of normal components $ℒ\left(T_i,T_i\right)$ , providing an intuitive geometric interpretation.

Figure 3. Mean curvature as the sum of normal components $ℒ\left(T_i,T_i\right)$ .

6.4. Mean Curvature Energy Functional

We define:

$E_{\text{MC}}={\displaystyle {\int }_{{\mathcal{S}}_K}{\Vert H\Vert }^2\text{d}\mu }.$

This acts as a membrane tension: minimizing ${\Vert H\Vert }^2$ smooths the manifold.

6.5. Physical Meaning of Mean Curvature

• $H=0$ : membrane in equilibrium,

• large $H$ : membrane bent or stretched,

• minimizing ${\Vert H\Vert }^2$ : smoother, more stable representations.

In deep networks, $H$ measures global geometric complexity and the tendency to overfit via highly curved decision boundaries.

7. Geometric Generator and Diffusion Operator

7.1. Stochastic Dynamics on the Manifold

Consider a diffusion on ${\mathcal{S}}_K$ :

$\text{d}{\theta }_t=-{\text{grad}}_{h}E\left({\theta }_t\right)\text{d}t+\sigma \left({\theta }_t\right)\text{d}{W}_t,$

where $W_t$ is a Brownian motion and $\sigma \left({\theta }_t\right)$ is a diffusion coefficient.

7.2. Geometric Generator

The associated generator is:

$ℒf\left(\theta \right)=-{\left({\text{grad}}_{h}E\left(\theta \right),{\text{grad}}_{h}f\left(\theta \right)\right)}_h+\frac{1}{2}\text{Tr}\left(\sigma \left(\theta \right)\sigma {\left(\theta \right)}^{\top }{\text{Hess}}_{h}f\left(\theta \right)\right).$

7.3. Interpretation of the Geometric Generator

The first term encodes energy dissipation along the Riemannian gradient, while the second term encodes intrinsic diffusion on the manifold. Mini-batch noise induces a diffusion on the curved manifold ${\mathcal{S}}_K$ , and curvature modulates the amplitude of stochastic fluctuations. Regions of high extrinsic curvature amplify noise, while flatter regions stabilize the diffusion. Thus, geometric regularization shapes both the deterministic and stochastic components of learning.

8. Feynman-Kac Representation

8.1. Geometric PDE

Consider the PDE:

$\frac{\partial u\left(\theta ,t\right)}{\partial t}=ℒu\left(\theta ,t\right)-V\left(\theta \right)u\left(\theta ,t\right),u\left(\theta ,0\right)=f\left(\theta \right),$

where $ℒ$ is the geometric generator and $V$ is a potential.

8.2. Feynman-Kac Formula

The solution is:

$u\left(\theta ,t\right)=\mathbb{E}\left[f\left({\theta }_t\right)\text{exp}\left(-{\displaystyle {\int }_0^{t}V\left({\theta }_s\right)\text{d}s}\right)|{\theta }_0=\theta \right].$

8.3. Interpretation of the Feynman-Kac Representation

The Feynman-Kac representation provides a probabilistic interpretation of the geometric PDE governing the evolution of $u\left(\theta ,t\right)$ . Learning becomes a diffusion process evolving on the curved manifold ${\mathcal{S}}_K$ , where the potential $V\left(\theta \right)$ acts as an energetic penalty. The exponential term

$\text{exp}\left(-{\displaystyle {\int }_0^{t}V\left({\theta }_s\right)\text{d}s}\right)$

plays the role of a Boltzmann weight: trajectories passing through regions of high curvature or high energy are exponentially suppressed, while smoother, low-energy trajectories contribute more significantly.

9. Complete Geometric Energy Model

9.1. General Principle

The geometric structure induced by the RKHS embedding provides a natural way to define a total energy functional that governs learning. This energy combines three components: a data fitting term, an extrinsic curvature term, and a mean curvature term.

9.2. Data Fitting Energy

$E_{\text{data}}\left(\theta \right)={\displaystyle \sum }_{i=1}^n{\Vert F_{\theta }\left(x_i\right)-z_i\Vert }^2.$

9.3. Extrinsic Geometric Energy

$E_{\text{geom}}={\displaystyle {\int }_{{\mathcal{S}}_K}{\Vert ℒ\Vert }^2\text{d}\mu }.$

9.4. Mean Curvature Contribution to Total Energy

$E_{\text{MC}}={\displaystyle {\int }_{{\mathcal{S}}_K}{\Vert H\Vert }^2\text{d}\mu }.$

9.5. Total Energy

$E\left(\theta \right)=E_{\text{data}}\left(\theta \right)+{\lambda }_1{E}_{\text{geom}}\left(\theta \right)+{\lambda }_2{E}_{\text{MC}}\left(\theta \right),{\lambda }_1,{\lambda }_2\ge 0.$

9.6. Riemannian Gradient

${\text{grad}}_{h}E\left(\theta \right)={\Pi }_{T_{\theta }{\mathcal{S}}_K}\left({\nabla }_{{\mathscr{H}}_K}E\right),\frac{\text{d}\theta }{\text{d}t}=-{\text{grad}}_{h}E\left(\theta \right).$

9.7. Physical Meaning of the Total Energy Model

Each term has a clear physical meaning:

• $E_{\text{data}}$ pulls the model toward the data,

• $E_{\text{geom}}$ penalizes extrinsic curvature,

• $E_{\text{MC}}$ acts as a membrane tension,

• the Riemannian gradient ensures intrinsic consistency.

10. Applications and Interpretations

The geometric and energetic framework has several practical implications:

• Representation stability: penalizing ${\Vert ℒ\Vert }^2$ and ${\Vert H\Vert }^2$ stabilizes the geometry of the output manifold.

• Noise robustness: a less curved manifold is less sensitive to input perturbations.

• Generalization: geometric regularization controls extrinsic complexity.

• Interpretability: curvature provides a geometric measure of sensitivity.

The interplay between ${\Vert ℒ\Vert }^2$ and ${\Vert H\Vert }^2$ is particularly informative: $ℒ$ captures local bending and sensitivity, while $H$ reflects global geometric tension. Together, they provide a multi-scale description of the complexity of the learned representation.

11. Experiments and Validation

This section presents several experiments illustrating the benefits of the geometric and energetic framework developed above.

11.1. Experimental Setup

We consider three families of datasets:

• Synthetic datasets: two-moons, concentric circles, intertwined spirals.

• Image datasets: MNIST and Fashion-MNIST (reduced resolution).

• Tabular datasets: standard UCI-type classification tasks.

We evaluate several architectures:

• MLP-3: fully connected network with three hidden layers.

• CNN-Small: small convolutional network.

• ResNet-18 (reduced): lightweight residual network.

Training schemes:

• Baseline (ERM + cross-entropy),

• $L^2$ regularization,

• Spectral norm regularization,

• G-RKHS (extrinsic curvature),

• G-RKHS+MC (extrinsic + mean curvature).

Hyperparameters $\left({\lambda }_1,{\lambda }_2\right)$ are selected by cross-validation. All experiments are repeated over 5 random seeds.

11.2. Curvature Measurement

We approximate:

${\Vert ℒ\Vert }^2\approx \mathbb{E}\left[{\Vert ℒ\left(T_i,T_j\right)\Vert }^2\right],{\Vert H\Vert }^2\approx {\Vert H\left(p\right)\Vert }^2,$

using finite differences and local tangent bases.

11.3. Experiment 1: Synthetic Classification

Baseline models produce highly curved decision boundaries. Geometric regularization smooths these boundaries and improves accuracy.

The impact of geometric regularization on synthetic datasets is summarized in Table 1, which reports test accuracy together with extrinsic and mean curvature values.

Table 1. Synthetic datasets: test accuracy and curvature (mean over 5 runs).

Method	Accuracy (%)	${\Vert ℒ\Vert }^2$	${\Vert H\Vert }^2$
Baseline	96.2 ± 0.4	12.8	7.4
L2	96.5 ± 0.3	10.1	6.2
Spectral Norm	97.1 ± 0.5	8.4	5.7
G-RKHS	98.4 ± 0.2	4.1	2.3
G-RKHS + MC	98.7 ± 0.1	2.8	1.4

11.4. Experiment 2: Noise Robustness

Gaussian perturbations $x\mapsto x+\epsilon$ , $\epsilon ~\mathcal{N}\left(0,{\sigma }^{2}I\right)$ .

Robustness to Gaussian noise is presented in Table 2, showing that curvature-based regularization significantly improves performance under increasing perturbation levels.

Table 2. Robustness under gaussian noise.

Method	$\sigma =0.1$	$\sigma =0.2$	$\sigma =0.3$
Baseline	89.4	72.1	51.3
L2	90.2	74.8	54.0
Spectral Norm	92.5	78.3	58.1
G-RKHS	95.8	85.4	71.2
G-RKHS + MC	96.3	87.1	74.5

11.5. Experiment 3: Generalization on Real Data

Generalization performance on MNIST, Fashion-MNIST, and tabular datasets is reported in Table 3, where G-RKHS and G-RKHS + MC consistently outperform baseline and classical regularization schemes.

Table 3. Generalization performance (test accuracy in %).

Method	MNIST	Fashion-MNIST	Tabular
Baseline	98.1	89.4	84.2
L2	98.3	90.1	85.0
Spectral Norm	98.4	90.7	85.6
G-RKHS	98.7	91.8	87.1
G-RKHS + MC	98.8	92.3	87.9

11.6. Energy Analysis

Geometric regularization yields smoother, more stable energy descent and correlates with improved robustness and generalization.

The evolution of the total energy during training for Baseline, G-RKHS, and G-RKHS + MC is shown in Figure 4, illustrating the stabilizing effect of geometric regularization.

Figure 4. Illustrative evolution of the total energy across training epochs for Baseline, G-RKHS, and G-RKHS + MC.

12. Discussion and Perspectives

12.1. Summary

The RKHS-geometry-energy framework coherently connects:

• extrinsic differential geometry,

• stochastic diffusions,

• energy-based modeling,

• deep learning.

Learning becomes a geometric flow on an immersed manifold, driven by a total energy balancing data fitting and geometric regularity.

12.2. Perspectives

Future directions include:

1) Extension to deep CNNs, transformers, and recurrent networks.

2) Intrinsic optimization based on geodesics.

3) Geometric analysis of gradient dynamics.

4) Adaptive curvature-based regularization.

5) Adversarial robustness via curvature control.

6) Geometric interpretability in sensitive domains.

13. Conclusions

We introduced a complete geometric and energetic framework for analyzing deep neural networks, based on embedding the output manifold into a RKHS. This induces a natural Riemannian structure, a Levi-Civita connection, a second fundamental form, and a mean curvature vector. We showed how the RKHS gradient induces an intrinsic Riemannian gradient, and how backpropagation corresponds to its projection onto the parameter manifold. The geometric generator and Feynman-Kac representation provide a stochastic and energetic interpretation of learning.

Experiments confirm that geometric regularization improves stability, robustness, and generalization. Controlling extrinsic curvature leads to smoother decision boundaries and more stable representations. The framework opens promising directions for intrinsic optimization, adaptive geometric regularization, and applications requiring stability and interpretability.

Conflicts of Interest

The author declares no conflicts of interest regarding the publication of this paper.

References

[1]	Amari, S. (2016) Information Geometry and Its Applications. Springer.[CrossRef]
[2]	Bronstein, M., Bruna, J., Cohen, T. and Veličković, P. (2021) Geometric Deep Learning. arXiv: 2104.13478.
[3]	do Carmo, M. (1992) Riemannian Geometry. Birkhäuser.[CrossRef]
[4]	Lee, J.M. (2012) Introduction to Smooth Manifolds. Springer.[CrossRef]
[5]	Petersen, P. (2016) Riemannian Geometry. Springer.[CrossRef]
[6]	Schölkopf, B. and Smola, A. (2002) Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press.[CrossRef]
[7]	Wahba, G. (1990) Spline Models for Observational Data. SIAM.[CrossRef]
[8]	Calin, O. (2020) Deep Learning Architectures: A Mathematical Approach. Springer.[CrossRef]
[9]	Mallat, S. (2016) Understanding Deep Convolutional Networks. Philosophical Transactions of the Royal Society A, 374, Article ID: 20150203.[CrossRef] [PubMed]
[10]	Rapelanoro-Rabenja, H. (2025) Stochastic Complexity: Long Memory, Non-Gaussian Chaos, and Energy Dynamics in Hilbert Space. HAL Preprint, hal-05522560v2. https://ehess.hal.science/hal-05522560v2