In recent years, in addition to classical neural networks architectures that solve classification problems, optimization and search, new architectures have appeared such as autoencoders, generative adversarial neural networks but their design and operation principle is basically similar to networks designed for pattern recognition ( Figure 1 ).

A common feature characteristic to all connectionist paradigms is the inability to construct a probabilistic or analytical distribution dependence of neural network weighting coefficients values on the training vectors values components at the input. Parametrically, a neural network is a “black box” and remains so for any even the most modern, accurate and reliable deep learning architectures. There is no way to trace in detail and step by step how the network output values were calculated.

However, the experimentally established networks structural features, their design principles and training are known to ensure the stability and such effects reproducibility as:

Such features have one thing in common to initialize them, it is necessary to operate with the structural network redundancy, i.e. neurons number and location ( Figure 2 ).

The neural network training process is represented as an approximation problem (and functioning as an interpolation problem in the narrow sense) [4] . In this case, the network itself acts as one nonlinear operator ( Figure 2 ). This point of view allows us to consider generalization as the input data nonlinear interpolation result [16] . The network carries out correct interpolation mainly due to continuity (differentiability), neuron activation functions nonlinearity as well as through certain mathematical procedures which for multilayer feed-forward networks ensures overall output function continuity.

However, the overall output function continuity and the trained network functioning reliability are different concepts. Continuity only provides validity when it eliminates network overfitting. Those, when the training samples are similar

and to distinguish them, it is necessary to increase the approximation accuracy or when there are too many examples and it is necessary to increase layers number, neurons in each layer, then according to the training results the network (with certain training algorithms) may be overtrained (the training sample examples are restored accurately but practically does not recognize similar samples). This effect is observed when training in additive or (even worse) multiplicative noise which is present in training examples but is not the simulated mapping function characteristic itself.

The interpolation problem is one of the main numerical methods problems. With its help approximate analytical representation problems, differentiation, the table-specified functions integration or functions with complex analytical representation are solved.

A broader concept the approximation uses methods for calculating approximate function values and its derivatives in the case when the certain fixed points function values are known. These points set is sometimes given to us by external circumstances, in machine learning case—by the training sample parameterization examples. The approximation concepts hierarchy is illustrated in Figure 3 .

Large number of approximation methods presence is caused by the historical theory and practice development of solving applied problems. Many methods arose as previous variants differing from them in the notation form, changing the calculations order which had the reducing goal of rounding errors influence in calculations.

One of the special approximation cases, the interpolation, implies that if the desired function y(x) is specified by the training sample in the table form, i.e. in terms of experiment on a grid $\left\{x_n,n=0,\text{1},\cdots \right\}$ at the nodes of which the values y_n = y(х_n) are known, then the task is to construct a function that restores the values of y(x) at an arbitrary point x. At the same time, we must require a fairly simple behavior of y(x): the function should not have “bursts” between neighboring nodes. Mathematically, it means that y(x) should have a sufficient number of higher derivatives that are not too large in magnitude. Let us choose linearly independent functions system $\left\{f_m\left(х\right),m=0,1,\cdots \right\}$. Such linear functions combination is called a generalized polynomial Ф(x). Approximation of y(x) by a generalized polynomial:

$y\left(x\right)\approx {\Phi }_N\left(x\right)\equiv \underset{m=0}{\overset{N}{{\displaystyle \sum }}}{c}_m{f}_m\left(x\right)$ (1)

where:

$c_m$—coefficients chosen with the condition that the generalized polynomial ${\Phi }_N\left(x\right)$, containing (N + 1) coefficients accurately conveys the tabulated values function at the (N+1)th node:

$\underset{m=0}{\overset{N}{{\displaystyle \sum }}}{c}_m{f}_m\left(x\right)=y_n,0\le n\le N$ (2)

Such approximation method is called interpolation, and the coefficients c_m are found from the linear system solution (2). For it to be solvable it is necessary that:

$\det \left[f_m\left(x_n\right)\right]\ne 0$ (3)

However, condition (3) is not necessary in neural network approximation case in a multilayer architecture.

Multidimensional interpolation means the constructing a function passing through points specified not on a plane but in space (three-, four-dimensional, etc.). Thus, instead of dependence (2) which we approximated with the function $f_m\left(x\right)$, we should find several coordinates function $f_m\left(x_1,x_2,x_3,\cdots ,x_z\right)$. If the nodes are in a regular grid form, for example, in the two-dimensional case, in a rectangular grid form then there are no fundamental problems with multidimensional interpolation construction. In the analysis under consideration, interpolation in the narrow sense of the word is of interest, which implies that the desired the function values $f_m\left(x_1,x_2,x_3,\cdots ,x_z\right)$ do not coincide with the nodes and lie in the intervals:

$\left[\begin{array}{ccccccc}\left[x_1^1,x_1^2\right]& \left[x_1^2,x_1^3\right]& \left[x_1^3,x_1^4\right]& \cdots & \left[x_1^{i-1},x_1^i\right]& \cdots & \left[x_1^{N-1},x_1^N\right]\\ \left[x_2^1,x_2^2\right]& \left[x_2^2,x_2^3\right]& \left[x_2^3,x_2^4\right]& \cdots & \left[x_2^{i-1},x_2^i\right]& \cdots & \left[x_2^{N-1},x_2^N\right]\\ ⋮& ⋮& ⋮& \ddots & ⋮& \ddots & ⋮\\ \left[x_j^1,x_j^2\right]& \left[x_j^2,x_j^3\right]& \left[x_j^3,x_j^4\right]& \cdots & \left[x_j^{i-1},x_j^i\right]& \cdots & \left[x_j^{N-1},x_j^N\right]\\ ⋮& ⋮& ⋮& \ddots & ⋮& \ddots & ⋮\\ \left[x_M^1,x_M^2\right]& \left[x_M^2,x_M^3\right]& \left[x_M^3,x_M^4\right]& \cdots & \left[x_M^{i-1},x_M^i\right]& \cdots & \left[x_M^{N-1},x_M^N\right]\end{array}\right]$ (4)

where:

(N-1)—is the intervals number between data points for each of the M variables.

Solving the task problems of multidimensional interpolation in the narrow sense of the word arise when:

1) The original sample is not tied to an equidistant grid, i.e. represents disparate data series;

2) The desired multidimensional function arguments $f_m\left(x_1,x_2,x_3,\cdots ,x_z\right)$ are not independent.

If in dynamic networks with “all to all” feedback type, the state of the i-th neuron in the system ( $i=1,2,\cdots ,N$) is described by a continuous variable _i(t), the pairwise neurons interaction is described by a matrix of weights with elements W_ij then the Lagrangian NS interaction which is the independent variables functional and W, is described by the expression [9] :

$L=-E=-\frac{1}{2}\lambda {\displaystyle \underset{ij}{\sum }{W}_{ij}{\sigma }_i{\sigma }_j}$ (5)

where:

 > 0 is the interaction parameter.

Dynamic equations of NS for and W_ij:

$\frac{1}{{\gamma }_1}\frac{\text{d}{\sigma }_i}{\text{d}t}-f_i\left(\sigma \right)=\frac{\partial L}{\partial {\sigma }_i}=\lambda {\displaystyle \underset{j}{\sum }{W}_{ij}{\sigma }_j}$ (6)

$\frac{1}{{\gamma }_2}\frac{\text{d}{W}_{ij}}{\text{d}t}-F_{ij}\left(W\right)=\frac{\partial L}{\partial W_{ij}}=\frac{1}{2}\lambda {\sigma }_i{\sigma }_j$ (7)

where f and F are coefficients that prevent an unlimited increase in the modules and W.

The system dynamics (6), (7) significantly depends on its history. The training consists in the fact that in (6) it is applied an external influence acting during time t₁, as a result of which the vector $\sigma$ takes on a stationary value φ¹ corresponding to the “image” with components $\pm {\sigma }_0$. As a training result, the matrix W_ij, in accordance with (7), changes by the amount $\Delta W_{ij}=\frac{1}{2}{\gamma }_2\lambda t_1{\phi }_i^1{\phi }_j^1$. In this case, we assume that the training time t₁ is significantly longer than the relaxation vector time $\sigma$ to its stationary value φ¹. The training procedure can be repeated many times using images φ^s, $s=1,2,\cdots ,s_0$. Assuming for simplicity that before the training start V = 0, after the procedure end we obtain

$W_{ij}={\displaystyle \underset{s=1}{\overset{s_0}{\sum }}{\mu }^s{\phi }_i^s{\phi }_j^s}$ (8)

where the coefficients μ^s depend on the duration of training.

In a discrete model the state q = φ^k realizes a stable local system energy minimum if the number of training sample examples is small (9):

$А\le 0.\text{14}С$, (9)

where

A is the memorized images number, C is the neurons number in the network.

The situation in which A>0.14C is of interest, i.e. when there are an excess neurons number, it is in this case that the input reactions prototype is developed. If an images group ${\Phi }^k=\left({\phi }_1^k,\cdots ,{\phi }_r^k\right)$, $k=\text{1},\cdots ,N$, of the training sample R is obtained with small random distortions ${\Delta }^k=\left({\delta }_1^k,\cdots ,{\delta }_r^k\right)$ of some vector ${\Phi }^{\text{o}}=\left({\phi }_1^{\text{o}},\cdots ,{\phi }_r^{\text{o}}\right)$, then the variation Δ of the vector Ф^o leads to a change in the network energy function corresponding to this vector by the amount

$\Delta E=-\frac{1}{2}\left(\underset{k=1}{\overset{N}{{\displaystyle \sum }}}{\left(\left({\Phi }^{\text{o}}+{\Delta }^k\right)\left({\Phi }^{\text{o}}+\Delta \right)\right)}^2-\underset{k=1}{\overset{N}{{\displaystyle \sum }}}{\left(\left({\Phi }^{\text{o}}+{\Delta }^k\right){\Phi }^{\text{o}}\right)}^2\right)$ (10)

When $\left\{\left(-{\Phi }^{\text{o}}{\Delta }^k\right)\le r,N\gg 1\right\}\Rightarrow \Delta E\ge 0$, therefore, the initial vector Ф^о corresponds to the minimum NS energy. This means that when the vector Р^о = (Z^оX^оK^оY) is applied to the NS Hopfield input is stabilized in the state (Z^оX^оK^оY^о), in which Y^o—is the desired result in the previously unobserved situation of Z^оX^о and K^о.

In the middle of the 20th century. F. Rosenblatt showed [14] that thanks to the redundant structure the multilayer neural network classifier allows one to move from a selective reaction to one image to a generalized reaction to a «similar» image, the characteristics of which may be completely different from the previous one. At the same time, the known basic transformation recognition methods classification included:

a) Analytically descriptive method which consists in reducing the classes alphabet and the features dictionary to a simple canonical description which is invariant under the transformations under consideration;

b) Image conversion method. It is mainly used to transform images based on their topology in a given n-dimensional space and has the serious disadvantage in requiring a huge neurons number in the network;

c) The generalization method by dominance which consists in the fact that a change in the characteristics of X input image in which a certain characteristics part remains the same as the original one will not cause a change in the reaction Y of the neural network to the original image. In other words, many additional features introduction the values of which are common to “similar” input images X will facilitate the rapid formation of a more reliable class alphabet with a training sample fixed size. Actual it is a classic multidimensional nonlinear interpolation.

In the research process (on input images of similar ones, mainly according to one-to-one transformations group), the fourth development direction in generalizing multilayer neural networks characteristics was established, the generalization by similarity, the recognition rule for which states: «Any neural network fragment that reacts to cause X₁ will also react to all mappings (X₁) of a given cause in a transformations group with probability p». Prototype, generalization, similarity are similar concepts that can be combined in multilayer neural networks architecture (including convolutional) to increase the recognition reliability of the desired vector characteristics values that previously extrapolated (interpolated) using the multidimensional linear extrapolation method ( Figure 4 ).

To do this, it is necessary to introduce artificial redundancy into multilayer neural networks architecture, i.e. introduce “false” neurons on which by analogy with dynamic networks the direct signals “potential” will accumulate during the learning process, but the neurons themselves do not participate in learning and minimizing the error functional at the neural network output.