A matrix can be factorized into the product of several matrices with special properties. Particularly, by singular value decomposition (SVD) which is widely used in regression analysis [1] [2] [3], a matrix can be factorized into the product of three matrices with orthogonal or diagonal properties respectively. SVD can be formularized like X = U θ V T [4] [5] [6] [7].

Moore-Penrose generalized inverse is a special case of generalized matrix inverse [8] [9] [10] [11] [12] and can be applied in regression analysis [13] [14] and least square analysis [15]. For example, in the study of Tian and Zhang [13], the linear unbiased estimator of partial coefficients is derived through Moore-Penrose generalized inverse algorithm. Herein, multiplied by Moore-Penrose generalized inverse of the independent variable matrix, the dependent variable vector is transformed into the new coordinate systems, or left space and then right space of the independent variable matrix [16]. In fact, this process can also be regarded as a vector rotating algorithm. In addition to Moore-Penrose’s generalized inverse algorithm, singular value decomposition (SVD) facilitates such transformation by dividing the transformation process into three-step to present a visible geometrical view.

In order to avoid the irrelevant calculation minutia when adopting an independent matrix with a large rank and only to highlight inner transformation, herein,

a simple multiple linear regression model y = X β ^ + ε ^ is adopted with the independent matrix X = [ 1 1 1 2 1 3 ] and the dependent vector y = [ 1 0 2 ] , β ^ symbolizes the regression coefficient estimator vector.

In this multiple linear regression model, the matrix X’s singular value decomposition (SVD) can be demonstrated as below:

X = U θ V T = [ 0.324 0.854 0.420 0.549 0.186 − 0.820 0.774 − 0.490 0.410 ] [ 4.07 0 0 0.6 0 0 ] [ 0.403 0.915 0.915 − 0.403 ] T (E.1)

U and V are respectively the left and right singular vector matrix of X, and θ is eigen value matrix of X [6] [7] [17].

Because X is a matrix with full rank column, β ^ can be demonstrated as β ^ = X + y [13] [16], X + signifies the Moore-Penrose generalized inverse of matrix X. And β ^ can also be demonstrated as below when X is substituted by its SVD form presented in equation (E.1),

β ^ = X + y = ( U θ V T ) + y = ( V T ) − 1 θ + U − 1 y (E.2)

The Equation (E.2) can be regarded as a process that the vector y is transformed 3 times from right to left multiplied by ( V T ) − 1 θ + U − 1 .

Transformation 1

y 1 = U − 1 y = [ 0.324 0.854 0.420 0.549 0.186 − 0.820 0.774 − 0.490 0.410 ] − 1 [ 1 0 2 ] = [ 1.86 − 0.12 1.24 ] (E.3)

U’s columns are demonstrated as u 1 = [ 0.324 0.549 0.774 ] , u 2 = [ 0.854 0.186 − 0.490 ] and u 3 = [ 0.420 − 0.820 0.410 ] .

Let the coordinate axes of a 3-dimension original coordinate system be symbolized by a 1 = [ 1 0 0 ] , a 2 = [ 0 1 0 ] and a 3 = [ 0 0 1 ] . The quintessence of the equation (E.3) is that multiplied by the matrix U − 1 , vector y = [ 1 0 2 ] in the coordinate system constructed by a 1 , a 2 and a 3 , can be transformed into y 1 = [ 1.86 − 0.12 1.24 ]

which is located in a coordinate system constructed by u 1 , u 2 and u 3 , or left space of X. In fact, y and y 1 are at the same spatial location. However, y is in the coordinate system constructed by a 1 , a 2 and a 3 , meanwhile y 1 is presented by the coordinate system constructed by u 1 , u 2 and u 3 , in another word, y 1 is in the left space of X. These locations can be demonstrated as below in Figure 1.

Transformation 2

y 2 = θ + U − 1 y = θ + y 1 = [ 4.07 0 0 0.6 0 0 ] + [ 1.86 − 0.12 1.24 ] = [ 0.46 − 0.20 0 ] (E.4)

During this transformation, multiplied by θ ’s Moore–Penrose inverse θ + , the vector y 1 in u 1 , u 2 and u 3 coordinate system is transformed into the vector y 2 which is in the same coordinate system as y 1 . Such a transformation can be called “Vector Stretching”. During this transformation, the coordinate value of y 1 in u 1 axis diminishes, meanwhile, the coordinate value of y 1 in u 2 axis gets enlarged. The coordinate value of y 1 in u 3 axis vanishes. The “Vector Stretching” of y 1 into y 2 can be demonstrated as below in Figure 2.

The vanishing coordinate value of y 1 in u 3 axis can signify the degree of freedom of X from a geometric view.

Transformation 3

β ^ = ( V T ) − 1 θ + U − 1 y = ( V T ) − 1 y 2 = ( [ 0.403 0.915 0.915 − 0.403 ] T ) − 1 [ 0.46 − 0.20 0 ] = [ 0 0.5 ] (E.5)

X’s right singular vectors can be denoted as v 1 T = [ 0.403 0.915 ] and v 2 T = [ 0.915 − 0.403 ] . v 1 T and v 2 T vectors are located in a coordinate system constructed by u 1 and u 2 vectors [16].

The quintessence of Equation (E.5) is that the vector y 2 in the coordinate system is constructed by u 1 and u 2 can be embodied by the vector β ^ in the coordinate system v 1 T and v 2 T . This transformation is demonstrated as below in Figure 3.

Though the vector β ^ has the same spatial position as y 2 in Figure 3, these two vectors are situated in two different coordinate systems constructed respectively by v 1 T , v 2 T (the right space of X) and u 1 , u 2 (the left space of X). In another word, the distinction between the vector β ^ and y 2 is that β ^ is demonstrated by the coordinate system of v 1 T and v 2 T , meanwhile, y 2 is demonstrated by u 1 and u 2 . As demonstrated by the aforementioned 3 steps of transformation, the dependent variable y vector is inner-transformed into parameter β ^ vector.

The independent variable X’s left singular vector matrix U [6] [7] [17] ) can be blocked into two sub-matrics U 1 and U 2 as below.

U = [ 0.324 0.854 0.420 0.549 0.186 − 0.820 0.774 − 0.490 0.410 ] = [ U 1 U 2 ] = [ 0.324 0.854 0.420 0.549 0.186 − 0.820 0.774 − 0.490 0.410 ]

Let y ^ symbolize the estimator of the dependent variable y. For that U 1 is the left space of X, y ^ can be demonstrated as y ^ = X β ^ = U 1 U 1 T y [16], based on which the following result can be derived.

y ^ = U 1 U 1 T y = [ 0.324 0.854 0.549 0.186 0.774 − 0.490 ] [ 0.324 0.854 0.549 0.186 0.774 − 0.490 ] T [ 1 0 2 ] = [ 0.5 1 1.5 ]

Because U 1 is an orthonormal matrix, U 1 ’s Moore-Penrose inverse U 1 + is equal with U 1 T [12]. So, y ^ can be demonstrated as below:

y ^ = U 1 U 1 + y = [ 0.324 0.854 0.549 0.186 0.774 − 0.490 ] [ 0.324 0.854 0.549 0.186 0.774 − 0.490 ] + [ 1 0 2 ] = [ 0.5 1 1.5 ] (E.6)

Based on E.6, y can be transformed into y ^ by two steps as below.

Transformation 4

y ∗ = U 1 + y = [ 0.324 0.854 0.549 0.186 0.774 − 0.490 ] + [ 1 0 2 ] = [ 1.86 − 0.12 ] (E.7)

In Transformation 4, y transforms into y ∗ by being projected into U 1 , the left space of X. Another word, y ∗ is the projection of y in X.

Transformation 5

y ^ = U 1 U 1 + y = U 1 y ∗ = [ 0.324 0.854 0.549 0.186 0.774 − 0.490 ] [ 1.86 − 0.12 ] = [ 0.5 1 1.5 ] (E.8)

In Transformation 5, multiplied by U 1 , y ∗ transforms into y ^ and is presented in the coordinate system constructed by a 1 , a 2 and a 3 . Their spatial locations are demonstrated in Figure 4.

In Figure 4, it can be found that the error estimator ε ^ is located in the same direction with u 3 which is the left null space of matrix X, and can be expressed as below:

ε ^ = [ 0.5 − 1 0.5 ] ≅ 1.22 [ 0.42 − 0.82 0.41 ] = 1.22 u 3 .

y ^ is the projection of y into the left space of X, but presented in the original coordinate system constructed by a 1 , a 2 and a 3 . ε ^ is located in the left null space of matrix X and perpendicular with y ^ . This perpendicular result can be geometrically demonstrated in Figure 4 as well as in the following multiplication.

y ^ T ε ^ = [ 0.5 1 1.5 ] [ 0.5 − 1 0.5 ] = 0 .

By applying SVD and Moore-Penrose generalized inverse of the independent variable X in a multiple linear regression model, the dependent variable y can be transformed into the regression coefficient estimator vector β ^ and its own estimator y ^ . This process presents a new geometric perspective to study the relationship between X, y, β ^ and y ^ , through the inner-transformation algorithm.

As demonstrated from Figures 1-3, y transforms into β ^ by transforming from the original coordinate system of a 1 , a 2 and a 3 into the right space of X constructed by v 1 T and v 2 T . Through this process, y transforms into y 1 by transferring into the coordinate system of u 1 , u 2 and u 3 , that is presented by Transformation 1 (E.3) and demonstrated in Figure 1.

Multiplied by θ + , the Moore-Penrose inverse of θ , y 1 stretches into y 2 , that is presented by Rotation 2 (E.4) and demonstrated in Figure 2. In this transformation, the coordinate value of y 1 in u 3 axis vanished, which also signifies the degree of freedom of X from a geometric view.

Multiplied by ( V T ) − 1 , the inverse of the right singular value matrix of X, y 2 transforms into β ^ , that is presented by Rotation 3 (E.5) and demonstrated in Figure 3.

As demonstrated in Figure 4, y transforms into y ∗ by projecting into U 1 , the left space of X. And then, multiplied by U 1 , y ∗ transforms into y ^ and returns back into the original coordinate system constructed by a 1 , a 2 and a 3 . y ^ is the projection of y into the left space of X, and perpendicular with error estimator ε ^ which is located in the left null space of X.

With the aid of the algorithms like matrix decomposition and Moore-Penrose generalized matrix inverse, the dependent variable y of a multiple linear regression model can be inner-transformed into the regression coefficient estimator β ^ and its own estimator y ^ . This process is a new approach to illustrate the inner-transformation between variables from a geometric view as well as presenting the spatial locations of the variables. To date, there is no study to explore the relationship between the variables of the multiple linear regression model from the view of geometric transformation. This study fills such a gap and provides a new perspective for studying multiple linear regression.

The limitation of this work is that a simple example of the multiple linear model is adopted to present such intricate inner transformation. But, it can demonstrate more inference art of inner transformation if more complex examples of multiple linear regression model are adopted in future studies.

The authors declare no conflicts of interest regarding the publication of this paper.

Li, Z.Y. and Antoncic, B. (2021) A Geometric View on Inner Transformation between the Variables of a Linear Regression Model. Applied Mathematics, 12, 931-938. https://doi.org/10.4236/am.2021.1210061