It is shown in [1] that each point in a set of observations that are used in a standard least-squares linear fit has an accompanying pivot point or fixed point. If a selected data point is replaced by any point on the vertical line containing the selected point, the new regression line, which is based upon the other original points and the new point on the vertical line, contains the pivot point, i.e., all these regression lines are concurrent, including the one for the original points. Varying the observation in this manner in the direction of prediction, which is parallel to the vertical axis for simple linear regression, is a common technique for diagnosing each observation’s impact on the regression line [1] ( [2] , p. 321) ( [3] , pp. 384-420) [4] .

Figure 1 illustrates this phenomenon for a set of artificial observations. The observation $R$, which is on the dashed vertical line $x=7$, has been selected and consecutively replaced by one other point on the vertical line. The other points plus a replacement point on the vertical line produce the displayed regression lines. All of the regression lines contain $P$, which is the pivot or fixed point corresponding to observation $R$. There is a regression line for each point on $x=7$. This graph is related to the numerical example in Section 6.

Although figures similar to Figure 1 had appeared, the existence of the pivot or fixed point was not pointed out until recently [1] . For example, a figure in ( [4] : p. 390) contains an unnoted point on five of these regression lines for the main data set of that article. In ( [5] : p. 381) and ( [6] : p. 98), the authors use identical graphs to discuss the large effect that one bivariate observation can have, but they do not note the pivot point that is in the graphs.

In Section 2, the existence of pivot points is proven and their coordinates are found. Then, those coordinates are expressed differently in Section 3 in order to show the properties of pivot points. In Section 4, a more general approach is taken to find all points in the plane that can replace the selected point to create a regression line containing the originally selected point’s pivot point. The geometry of these special points is explored in Section 5. A numerical example is supplied in Section 6, followed by concluding comments in Section 7.

Consider the least-squares regression line

$y={\beta }_0+{\beta }_{1}x+\epsilon ,$ (1)

where the intercept ${\beta }_0$ and slope ${\beta }_1$ are to be estimated and the error term $\epsilon$ has a standard normal distribution. The observations are

$S=\left\{\left(x_i,y_i\right)|i=1,2,\cdots ,n\right\},$ (2)

where not all $x$-values are equal. Recall that in order to obtain estimates of the $y$-intercept and slope, which are the outcomes of the minimization of the sum of squared vertical distances of the data from the regression line, apply $\frac{\partial }{\partial {\beta }_0}$ and $\frac{\partial }{\partial {\beta }_1}$ to

${\displaystyle {\sum }_{i=1}^n{\left(y_i–\left({\beta }_0+{\beta }_1{x}_i\right)\right)}^2}.$ (3)

Setting the results equal to zero and introducing the estimates $b_0$ and $b_1$ of ${\beta }_0$ and ${\beta }_1$, respectively, give the Normal Equations

$n{b}_0+\left({\displaystyle {\sum }_{i=1}^n{x}_i}\right)b_1={\displaystyle {\sum }_{i=1}^n{y}_i}$ (4)

and

$\left({\displaystyle {\sum }_{i=1}^n{x}_i}\right)b_0+\left({\displaystyle {\sum }_{i=1}^n{x}_i^2}\right)b_1={\displaystyle {\sum }_{i=1}^n{x}_i{y}_i}$ (5)

( [2] , p. 305) ( [3] , p. 17) Differentiation can be avoided by noticing that the sum (3) can be written, using the estimates, as

$\begin{array}{c}{\displaystyle {\sum }_{i=1}^n{\left(y_i–\left(b_0+b_1{x}_i\right)\right)}^2}={\displaystyle {\sum }_{i=1}^n{y}_i^2}+n{b}_0^2+{\displaystyle {\sum }_{i=1}^n{x}_i^2{b}_1^2}-2{\displaystyle {\sum }_{i=1}^n{y}_i{b}_0}\\ +2{\displaystyle {\sum }_{i=1}^n{x}_i{b}_0{b}_1}-2{\displaystyle {\sum }_{i=1}^n{x}_i{y}_i{b}_1},\end{array}$

which is separately quadratic in $b_0$ and $b_1$ with positive leading coefficients $n$ and ${\displaystyle {\sum }_{i=1}^n{x}_i^2}$, respectively. As a quadratic expression in $b_0$, the minimum is attained at

$b_0=–\frac{2b_1{\displaystyle {\sum }_{i=1}^n{x}_i}-2{\displaystyle {\sum }_{i=1}^n{y}_i}}{2n},$

which is Equation (4). As a quadratic expression in $b_1$, the minimum is attained at

$b_1=–\frac{2b_0{\displaystyle {\sum }_{i=1}^n{x}_i}-2{\displaystyle {\sum }_{i=1}^n{x}_i{y}_i}}{2{\displaystyle {\sum }_{i=1}^n{x}_i^2}}$,

which is Equation (5).

Dividing the Normal Equation (4) by sample size $n$ shows that the point of means

$\left({\displaystyle {\sum }_{i=1}^n{x}_i/n},{\displaystyle {\sum }_{i=1}^n{y}_i/n}\right)$ (6)

is on the regression line in Equation (1).

Solving the Normal Equations (4) and (5) gives

$b_0=\frac{{\displaystyle {\sum }_{i=1}^n{x}_i^2}{\displaystyle {\sum }_{i=1}^n{y}_i}-{\displaystyle {\sum }_{i=1}^n{x}_i}{\displaystyle {\sum }_{i=1}^n{x}_i{y}_i}}{n{\displaystyle {\sum }_{i=1}^n{x}_i^2}-{\left({\displaystyle {\sum }_{i=1}^n{x}_i}\right)}^2}$ (7)

and

$b_1=\frac{n{\displaystyle {\sum }_{i=1}^n{x}_i{y}_i}-{\displaystyle {\sum }_{i=1}^n{x}_i}{\displaystyle {\sum }_{i=1}^n{y}_i}}{n{\displaystyle {\sum }_{i=1}^n{x}_i^2}-{\left({\displaystyle {\sum }_{i=1}^n{x}_i}\right)}^2}.$ (8)

Thus, the regression line in Equation (1) can be written as

$y=\frac{{\displaystyle {\sum }_{i=1}^n{x}_i^2}{\displaystyle {\sum }_{i=1}^n{y}_i}-{\displaystyle {\sum }_{i=1}^n{x}_i}{\displaystyle {\sum }_{i=1}^n{x}_i{y}_i}}{n{\displaystyle {\sum }_{i=1}^n{x}_i^2}-{\left({\displaystyle {\sum }_{i=1}^n{x}_i}\right)}^2}+\frac{n{\displaystyle {\sum }_{i=1}^n{x}_i{y}_i}-{\displaystyle {\sum }_{i=1}^n{x}_i}{\displaystyle {\sum }_{i=1}^n{y}_i}}{n{\displaystyle {\sum }_{i=1}^n{x}_i^2}-{\left({\displaystyle {\sum }_{i=1}^n{x}_i}\right)}^2}x,$ (9)

and as

$y=\frac{{\displaystyle {\sum }_{i=1}^n{y}_i}}{n}+\frac{n{\displaystyle {\sum }_{i=1}^n{x}_i{y}_i}-{\displaystyle {\sum }_{i=1}^n{x}_i}{\displaystyle {\sum }_{i=1}^n{y}_i}}{n{\displaystyle {\sum }_{i=1}^n{x}_i^2}-{\left({\displaystyle {\sum }_{i=1}^n{x}_i}\right)}^2}\left(x-\frac{{\displaystyle {\sum }_{i=1}^n{x}_i}}{n}\right).$ (10)

A fixed point or pivot point in simple linear regression is obtained in the following way. Given a set of bivariate observations, select one of the points, whose $y$-value will be made to vary, while leaving unchanged all the observations’ $x$-values and all their $y$-values except the selected one. Repeatedly, find the regression line for the original observations but with a replacement $y$-value for the selected one. All the regression lines created in this manner contain one point, which is called the pivot point in [1] . The regression lines pivot about the point that is like a fulcrum for a lever, as in Figure 1 . The lever in various positions is analogous to the regression lines. The term fixed point is nearly exclusively used below for this point. Theorem 1 says that each observation in a bivariate data set is accompanied by a fixed point.

Equations (7) and (8) show the well-known linear dependence of $b_0$ and $b_1$, and, hence, the regression line in Equations (9) and (10), on each $y$-value among the observations ( [3] , pp. 18-19) ( [4] , p. 390). Designate by $y_n$ the selected $y$-value that is to be replaced in order to investigate the fixed point. The observations can be ordered so that the data point of interest is the $n$th one having coordinates $\left(x_n,y_n\right)$. Equations (7) and (8) are written to explicitly display the linear dependence of the coefficients on $y_n$ as

$b_0=\frac{{{\displaystyle \sum }}_{i=1}^n{x}_i^2{{\displaystyle \sum }}_{i=1}^{n-1}{y}_i-{{\displaystyle \sum }}_{i=1}^n{x}_i{{\displaystyle \sum }}_{i=1}^{n-1}{x}_i{y}_i}{n{{\displaystyle \sum }}_{i=1}^n{x}_i^2-{\left({{\displaystyle \sum }}_{i=1}^n{x}_i\right)}^2}+\frac{{{\displaystyle \sum }}_{i=1}^n{x}_i^2-\left({{\displaystyle \sum }}_{i=1}^n{x}_i\right)x_n}{n{{\displaystyle \sum }}_{i=1}^n{x}_i^2-{\left({{\displaystyle \sum }}_{i=1}^n{x}_i\right)}^2}{y}_n$ (11)

and

$b_1=\frac{n{{\displaystyle \sum }}_{i=1}^{n-1}{x}_i{y}_i-{{\displaystyle \sum }}_{i=1}^n{x}_i{{\displaystyle \sum }}_{i=1}^{n-1}{y}_i}{n{{\displaystyle \sum }}_{i=1}^n{x}_i^2-{\left({{\displaystyle \sum }}_{i=1}^n{x}_i\right)}^2}+\frac{n{x}_n-{{\displaystyle \sum }}_{i=1}^n{x}_i}{n{{\displaystyle \sum }}_{i=1}^n{x}_i^2-{\left({{\displaystyle \sum }}_{i=1}^n{x}_i\right)}^2}{y}_n.$ (12)

Theorem 1 (Existence and Coordinates of the Fixed Point in Simple Linear Regression). Consider the set of bivariate observations $S$ in Equation (2), where not all $x$-values are equal. Select $R\left(x_n,y_n\right)$, where $x_n$ is not the sample mean, i.e., $x_n\ne {\bar{x}}_{\left(n\right)}$. For these data, each value $t$ that is chosen to replace $y_n$ produces a member of a family of least-squares regression lines. All members of this family contain one point $P$, the fixed point, which is on the regression line for $S$. In terms of the first $n-1$ observations’ coordinates and $x_n$, $P$ is

$P\left(p,q\right)=P\left({\bar{x}}_{\left(n\right)}-\frac{{{\displaystyle \sum }}_{j=1}^n{\left(x_j-{\bar{x}}_{\left(n\right)}\right)}^2}{n\left(x_n-{\bar{x}}_{\left(n\right)}\right)},\frac{n–1}{n}{\bar{y}}_{\left(n-1\right)}-\frac{{{\displaystyle \sum }}_{i=1}^{n-1}\left(x_i-{\bar{x}}_{\left(n\right)}\right)y_i}{n\left(x_n-{\bar{x}}_{\left(n\right)}\right)}\right).$ (13)

If $x_n={\bar{x}}_{\left(n\right)}$, then it is said that the observation has a fixed point and that the point is at infinity. Subscripts in parentheses indicate sample size. In particular, ${\bar{x}}_{\left(n\right)}$ is the sample mean of all $n$ $x$-values in $S$, and ${\bar{y}}_{\left(n-1\right)}$ is the sample mean of the $y$-values for the first $n-1$ observations of $S$.

Proof 1. The slope in Equation (9) and (10) can be written in a standard way as

$\begin{array}{c}{b}_1=\frac{n{{\displaystyle \sum }}_{i=1}^n{x}_i{y}_i-{{\displaystyle \sum }}_{i=1}^n{x}_i{{\displaystyle \sum }}_{i=1}^n{y}_i}{n{{\displaystyle \sum }}_{i=1}^n{x}_i^2-{\left({{\displaystyle \sum }}_{i=1}^n{x}_i\right)}^2}=\frac{n{{\displaystyle \sum }}_{i=1}^n\left(x_i-{\bar{x}}_{\left(n\right)}\right)y_i}{n{{\displaystyle \sum }}_{j=1}^n{\left(x_j-{\bar{x}}_{\left(n\right)}\right)}^2}\\ =\underset{i=1}{\overset{n}{{\displaystyle \sum }}}\left(\frac{x_i-{\bar{x}}_{\left(n\right)}}{{{\displaystyle \sum }}_{j=1}^n{\left(x_j-{\bar{x}}_{\left(n\right)}\right)}^2}\right)y_i\end{array}$ (14)

( [2] : p. 306) ( [3] : pp. 17-18) Recalling that the point of means in Formula (6) is on the line, the regression line for $S$ can be written so that the coefficient of $y_n$ is isolated as

$\begin{array}{c}y={\bar{y}}_{\left(n\right)}+b_1\left(x-{\bar{x}}_{\left(n\right)}\right)\\ =\frac{{{\displaystyle \sum }}_{i=1}^n{y}_i}{n}+{{\displaystyle \sum }}_{i=1}^n\left(\frac{x_i-{\bar{x}}_{\left(n\right)}}{{{\displaystyle \sum }}_{j=1}^n{\left(x_j-{\bar{x}}_{\left(n\right)}\right)}^2}\right)y_i\left(x-{\bar{x}}_{\left(n\right)}\right)\\ =\left(\frac{{{\displaystyle \sum }}_{i=1}^{n–1}{y}_i}{n}+{{\displaystyle \sum }}_{i=1}^{n–1}\left(\frac{x_i-{\bar{x}}_{\left(n\right)}}{{{\displaystyle \sum }}_{j=1}^n{\left(x_j-{\bar{x}}_{\left(n\right)}\right)}^2}\right)y_i\left(x-{\bar{x}}_{\left(n\right)}\right)\right)\\ +\left(\frac{1}{n}+\frac{\left(x_n-{\bar{x}}_{\left(n\right)}\right)\left(x-{\bar{x}}_{\left(n\right)}\right)}{{{\displaystyle \sum }}_{j=1}^n{\left(x_j-{\bar{x}}_{\left(n\right)}\right)}^2}\right)y_n.\end{array}$ (15)

Consider an arbitrary point $T\left(x_n,t\right)$ on the vertical line

$x=x_n$ (16)

as a temporary replacement for the observation $R\left(x_n,y_n\right)$. See Figure 2 . Then, Equation (15) becomes

$\begin{array}{l}y=\left(\frac{{{\displaystyle \sum }}_{i=1}^{n-1}{y}_i}{n}+{{\displaystyle \sum }}_{i=1}^{n-1}\left(\frac{x_i-{\bar{x}}_{\left(n\right)}}{{{\displaystyle \sum }}_{j=1}^n{\left(x_j-{\bar{x}}_{\left(n\right)}\right)}^2}\right)y_i\left(x-{\bar{x}}_{\left(n\right)}\right)\right)\\ +\left(\frac{1}{n}+\frac{\left(x_n-{\bar{x}}_{\left(n\right)}\right)\left(x-{\bar{x}}_{\left(n\right)}\right)}{{{\displaystyle \sum }}_{j=1}^n{\left(x_j-{\bar{x}}_{\left(n\right)}\right)}^2}\right)t.\end{array}$ (17)

If $x$ in Equation (17) is the $x$-coordinate $p$ of a fixed point $P$, then the $y$-coordinate $q$ of $P$ needs to be independent of $t$, and thus the coefficient of $t$ in Equation (17) must be zero, i.e.,

$\frac{1}{n}+\frac{\left(x_n-{\bar{x}}_{\left(n\right)}\right)\left(x-{\bar{x}}_{\left(n\right)}\right)}{{{\displaystyle \sum }}_{j=1}^n{\left(x_j-{\bar{x}}_{\left(n\right)}\right)}^2}=0.$

Thus,

$x={\bar{x}}_{\left(n\right)}-\frac{{{\displaystyle \sum }}_{i=1}^n{\left(x_i-{\bar{x}}_{\left(n\right)}\right)}^2}{n\left(x_n-{\bar{x}}_{\left(n\right)}\right)}$, (18)

which is $p$ in Formula (13). By substituting the $x$-Coordinate (18) into Equation (17),

$\begin{array}{c}y=\frac{{{\displaystyle \sum }}_{i=1}^{n-1}{y}_i}{n}+{{\displaystyle \sum }}_{i=1}^{n-1}\left(\frac{x_i-{\bar{x}}_{\left(n\right)}}{{{\displaystyle \sum }}_{j=1}^n{\left(x_j-{\bar{x}}_{\left(n\right)}\right)}^2}\right)y_i\left(x-{\bar{x}}_{\left(n\right)}\right)\\ =\frac{n-1}{n}{\bar{y}}_{\left(n-1\right)}+{{\displaystyle \sum }}_{i=1}^{n-1}\left(\frac{x_i-{\bar{x}}_{\left(n\right)}}{{{\displaystyle \sum }}_{j=1}^n{\left(x_j-{\bar{x}}_{\left(n\right)}\right)}^2}\right)y_i\left(–\frac{{{\displaystyle \sum }}_{j=1}^n{\left(x_j-{\bar{x}}_{\left(n\right)}\right)}^2}{n\left(x_n-{\bar{x}}_{\left(n\right)}\right)}\right)\\ =\frac{n-1}{n}{\bar{y}}_{\left(n-1\right)}–\frac{1}{n}{{\displaystyle \sum }}_{i=1}^{n-1}\left(\frac{x_i-{\bar{x}}_{\left(n\right)}}{x_n-{\bar{x}}_{\left(n\right)}}\right)y_{i,}\end{array}$

which is $q$ in Formula (13). ∎

Proof 2. All members of the family of lines

$y=\left(a+cr\right)+\left(d+er\right)x$ (19)

with real parameter $r$ and real numbers $a,c,d$ and $e$ for $e\ne 0$ intersect at the point

$\left(-\frac{c}{e},\frac{ae-cd}{e}\right)$. (20)

The validity of Point (20) is checked by substitution for $x$ and $y$ from Point (20) into Equation (19). If $e=0$, then the lines in Equation (19) are parallel with slope $d$ and, thus, do not intersect, unless they coincide, which occurs if, additionally, $c=0$. The existence of the fixed point can be seen from the identification of the coefficients in Formulas (11) and (12) with the corresponding coefficients in Equation (19) for $r=y_n$. The coordinates of $P$ can be obtained by substituting from Formulas (11) and (12) for $a,c,d$ and $e$ into the coordinates of the point in (20). Comparing Formulas (11) and (12) with Equation (19), $e\ne 0$ is equivalent to $x_n\ne {\bar{x}}_{\left(n\right)}$. ∎

Figure 2 displays the chosen observation $R\left(x_n,y_n\right)$, the vertical line $x=x_n$, one example of a replacement point $T\left(x_n,t\right)$, the regression line for the set $S\setminus \left\{\left(x_n,y_n\right)\right\}\cup T$, and the fixed point $P$. The observations are not displayed in the graph.

The first goal of this section is to express the coordinates of the fixed point $P$ so that they include $y_n$ in a way similar to the inclusion of the $x$-coordinate $x_n$. The second goal is to write the coordinates of the fixed point so that the occurrences of $x_n$ are explicit throughout and the coordinates are given in terms of the simplest sums. The advantage of these re-expressions is that they make some properties of fixed points much more apparent. The third goal is to show that the fixed point is the intersection of the original regression line for the set $S$ and the regression line for the set of the first $n-1$ observations of $S$, i.e., for $S\setminus \left\{\left(x_n,y_n\right)\right\}$.

The question addressed in this section is:

Are there any other points in the plane, besides those described in Theorem 1, that together with $S\setminus \left\{\left(x_n,y_n\right)\right\}$, i.e., $\left\{\left(x_i,y_i\right)|i=1,2,\cdots ,n-1\right\}$, produce regression lines containing the fixed point $P\left(p,q\right)$ corresponding to $\left(x_n,y_n\right)$?

This question is answered in the affirmative by Theorems 3 and 4, where all such points are found.

Theorem 3 (Points on the Regression Line for $S\setminus \left\{\left(x_n,y_n\right)\right\}$). Augmenting the set $S\setminus \left\{\left(x_n,y_n\right)\right\}$ with any one point of the line in Equation (32), which is the regression line on that set, yields a line containing the fixed point $P\left(p,q\right)$ that corresponds to the observation $\left(x_n,y_n\right)$.

Proof. This is immediate from the fact that adding a new point that is on a regression line does not alter the identity of the regression line and that $P$ is on the line in Equation (32) by Theorem 2. ∎

Theorem 3 shows that the commonplace diagnostic method of deleting a point to determine its effect is included in the technique of altering the point as in Theorem 1. Replacing the point $\left(x_n,y_n\right)$ with the point at the intersection of the regression line for $S\setminus \left\{\left(x_n,y_n\right)\right\}$ and the line $x=x_n$ produces the regression line for the set $S\setminus \left\{\left(x_n,y_n\right)\right\}$. The $n$th point has been “neutralized,” because it no longer has any role in determining the identity of the regression line, which has become the line on the other $n-1$ observations. This uses the least-squares estimate for the observation at $x=x_n$ that is based upon the other data, which is a familiar method for missing data [7] ( [8] , pp. 32-34) and for outliers [9] .

Theorem 4 (All $n$th Points that Create the Same Fixed Point). Augmenting the set $S\setminus \left\{\left(x_n,y_n\right)\right\}=\left\{\left(x_i,y_i\right)|i=1,2,\cdots ,n-1\right\}$ with any single point on the vertical line in Equation (16) or on the regression line in Equation (32) determines a regression line containing the point $P\left(p,q\right)$, which is the fixed point that is determined by the observation $\left(x_n,y_n\right)$. Those are the only single points in the plane that determine a regression line containing $P$ by augmenting the set $S\setminus \left\{\left(x_n,y_n\right)\right\}$.

Proof. The proof proceeds by replacing the chosen $n$th data point $\left(x_n,y_n\right)$ of $S$ with the point $A\left(s,t\right)$ to form the set $S\setminus \left\{\left(x_n,y_n\right)\right\}\cup \left\{\left(s,t\right)\right\}$. Substituting the coordinates of $P$ from (26) into the regression line for that set yields an equation for $s$ and $t$, which are the desired coordinates. That equation reduces to only two factors. One yields the vertical line in Equation (16). and the other represents the line in Equation (32). Because the proof consists principally of algebraic manipulations, it is placed in Appendix. ∎

It is shown in this section that fixed points are centers about which diagnostic techniques are referenced.

From Equation (15), the regression line for set $S$ is

$y={{\displaystyle \sum }}_{i=1}^n\left(\frac{1}{n}+\frac{\left(x_i-{\bar{x}}_{\left(n\right)}\right)\left(x-{\bar{x}}_{\left(n\right)}\right)}{{{\displaystyle \sum }}_{k=1}^n{\left(k-{\bar{x}}_{\left(n\right)}\right)}^2}\right)y_i$.

The coefficients measure the impacts of the $x$-coordinates on the fitted values. The fitted value at the $j$th observation’s $x$-value $x_j$ is

$y\left(x_j\right)={{\displaystyle \sum }}_{i=1}^n\left(\frac{1}{n}+\frac{\left(x_i-{\bar{x}}_{\left(n\right)}\right)\left(x_j-{\bar{x}}_{\left(n\right)}\right)}{{{\displaystyle \sum }}_{k=1}^n{\left(k-{\bar{x}}_{\left(n\right)}\right)}^2}\right)y_i$. (33)

The parentheses in Equation (33) contain the terms of the hat matrix, which are designated $h_{ij}$ [4] ( [10] , pp. 90-92). They show that the farther an observation’s first coordinate is from the mean, the larger its impact on a fitted value. This effect is called leverage. Most often, the goal is to determine the impact that each observation has on the fitted value for the observation’s $x$-value, so the diagonal elements $h_{ii}$ of the hat matrix are especially important. They are written $h_i$.

Consider the same $n$th observation as in the previous sections. Its influence on $y\left(x_n\right)$ is $h_n{y}_n$. An interpretation is that changing $y_n$ by one unit changes the fitted value $y\left(x_n\right)$ by $h_n$ units. The set $\left\{h_i|i=1,2,\cdots ,n\right\}$ has desirable properties. For example, ${{\displaystyle \sum }}_{i=1}^n{h}_i=2$, which supplies a finite scale for these positive terms. Two approaches to diagnosing whether an observation’s $x$-value has high leverage is to compute the $n$ values in the set $\left\{h_i\right\}$. The large values would indicate that their observations have high leverage. There are rules of thumb, as well [4] ( [10] , p. 91). One is that any observation with $h_i>0.5$ is problematical.

Sometimes, an observation is deleted in order to determine the observation’s impact on some feature of the fit. From Theorem 3, deleting an observation yields the line for the remaining observations and that is the same line as the one obtained by adding a point on the line for the remaining observations. There are other properties and statistics that may be of interest in a study, such as the correlation coefficient. Those might be altered by the addition of a point. If the identity of the line or its slope are the attributes under study, an alternative to deletion is moving the point to the fit of the remaining observations [9] .

Theorem 5 fleshes out the geometric picture.

Theorem 5 (Three Collinear Points). The point of means $\left({\bar{x}}_{\left(n-1\right)},{\bar{y}}_{\left(n-1\right)}\right)$ for $\left\{\left(x_i,y_i\right)|i=1,2,\cdots ,n-1\right\}$, the point of means

$\left(\left({{\displaystyle \sum }}_{i=1}^{n-1}{x}_i+s\right)/n,\left({{\displaystyle \sum }}_{i=1}^{n-1}{y}_i+t\right)/n\right)=\left(\left(\left(n-1\right){\bar{x}}_{\left(n-1\right)}+s\right)/n,\left(\left(n-1\right){\bar{y}}_{\left(n-1\right)}+t\right)/n\right)$

for $\left\{\left(x_i,y_i\right)|i=1,2,\cdots ,n-1\right\}\cup \left\{\left(s,t\right)\right\}$, and the point $\left(s,t\right)$ are collinear or else coincide.

Proof. Assuming the three points do not coincide, the line containing the two points of means is

$y={\bar{y}}_{\left(n-1\right)}+\frac{\left(\left(n-1\right){\bar{y}}_{\left(n-1\right)}+t\right)/n-{\bar{y}}_{\left(n-1\right)}}{\left(\left(n-1\right){\bar{x}}_{\left(n-1\right)}+s\right)/n-{\bar{x}}_{\left(n-1\right)}}\left(x-{\bar{x}}_{\left(n-1\right)}\right),$

i.e.,

$y={\bar{y}}_{\left(n-1\right)}+\frac{t-{\bar{y}}_{\left(n-1\right)}}{s-{\bar{x}}_{\left(n-1\right)}}\left(x-{\bar{x}}_{\left(n-1\right)}\right).$ (34)

The point $\left(s,t\right)$ satisfies Equation (34). ∎

In Theorem 5, if $s\ne {\bar{x}}_{\left(n-1\right)}$ and $t={\bar{y}}_{\left(n-1\right)}$, then the line is horizontal. If $s={\bar{x}}_{\left(n-1\right)}$ and $t\ne {\bar{y}}_{\left(n-1\right)}$, then the line is vertical. If $s={\bar{x}}_{\left(n-1\right)}$ and $t={\bar{y}}_{\left(n-1\right)}$, the three points coincide.

To summarize, by Theorem 1, the fixed point $P$, which corresponds to the chosen datum $R\left(x_n,y_n\right)$ for observations $S$, is a “center of rotation” with “spokes” being all the regression lines on the sets $S\setminus \left\{\left(x_n,y_n\right)\right\}\cup T\left(x_n,t\right)$. See Figure 1 and Figure 2 . Theorem 2 says that the regression line for $S\setminus \left\{\left(x_n,y_n\right)\right\}$ is one of these spokes. The point of intersection $V$ of $x=x_n$ and that regression line can be used to form the set $S\setminus \left\{\left(x_n,y_n\right)\right\}\cup V$, whose regression line is the same line as the regression line for $S\setminus \left\{\left(x_n,y_n\right)\right\}$ by Theorem 3. Point $V$ is an example of a point that is generically labeled $T$. There are two points of means on this regression line. One is $N\left({\bar{x}}_{\left(n-1\right)},{\bar{y}}_{\left(n-1\right)}\right)$, and the other one, $W$, is for the set $S\setminus \left\{\left(x_n,y_n\right)\right\}\cup V$ and is on the line $x={\bar{x}}_{\left(n\right)}$. The point of means for the set $S\setminus \left\{\left(x_n,y_n\right)\right\}\cup T$ is $M\left({\bar{x}}_{\left(n-1\right)},\left(\left(n-1\right){\bar{y}}_{\left(n-1\right)}+t\right)/n\right)$.

Figure 3 contains the regression line for $\left\{\left(x_i,y_i\right)|i=1,2,\cdots ,n-1\right\}$, which is the line with the larger positive slope in the graph, the points $R\left(x_n,y_n\right)$, $N$, $W$, and $V$, and fixed point $P$ corresponding to $R$. The righthand (dashed) vertical line $x=x_n$ contains $R,T\left(x_n,t\right)$, and $V$. The lefthand (dashed) vertical line is $x={\bar{x}}_{\left(n\right)}$, which contains $M$ and $W$. Points $M$, $N$, and $T$ are collinear by Theorem 5. The line containing $P$ and $M$ is the regression line for $\left\{\left(x_i,y_i\right)|i=1,2,\cdots ,n-1\right\}\cup T$.

The “motion” in the system is obtained by varying up and down the position of $T$. This moves the regression line’s intersection with $x=x_n$ proportionally with $h_n$ through the method of least-squares fitting. As $T$ is moved, the point of means $M$of $\left\{\left(x_i,y_i\right)|i=1,2,\cdots ,n-1\right\}\cup T$ moves and is on the regression line that is being created, so that the regression line contains $P$ and $M$. This can be performed with linkages; as $T$ moves, the line containing $M$, $N$, and $T$ rotates about $N$ as the center and, thus, moves $M$ vertically on $x={\bar{x}}_{\left(n\right)}$. The line containing $M$ and $P$ rotates about fixed point $P$ as the center and is the regression line.

Figure 4 illustrates a mechanical device that is a physical embodiment of the effect that one datum has on the location of the regression line. The datum’s fixed point is a center. The blue slider on the right is moved by the operator and the green slider on the left follows, being controlled by the rods. The yellow items are centers of rotation; the red and the tan rods are attached there like hands of a clock. The goal of the device is to move the tan rod in a manner so that its locations reflect least-squares fitting of a regression line as the position of an observation, i.e., the blue slider, is altered vertically by the operator.

When the mechanism is being run, the blue slider ( $T$) is moved up and down continuously in a controlled manner on the righthand vertical rectangular grey track ( $x=x_n$). The red rod (line containing $M$, $N$, and $T$), which passes through swiveling eyebolts, rotates about the upper yellow center ( $N$), guided by and dragging along the green slider ( $M$) on the lefthand vertical rectangular grey track ( $x={\bar{x}}_{\left(n\right)}$). As the green slider ( $M$) is thusly moved up and down indirectly by the movement of the blue slider ( $T$), the tan rod (line containing $P$ and $M$) moves freely through a swiveling eyebolt, rotating about the lower yellow center (fixed point $P$) and is aligned with the regression line (for $\left\{\left(x_i,y_i\right)|i=1,2,\cdots ,n-1\right\}\cup T$).

Consider the set of seven artificial bivariate observations

$\left(1,3\right),\left(2,2\right),\left(3,3\right),\left(3,5\right),\left(5,8\right),\left(5,9\right),\left(7,6\right).$ (35)

The selected point is $R\left(7,6\right)$. These data were used to create Figure 1 . From Formula (26), the fixed point corresponding to $R$ is

$P\left(p,q\right)=\left(\frac{19}{6}-\frac{\left(6\right)\left(73\right)-{\left(19\right)}^2}{6\left(\left(6\right)\left(7\right)-19\right)},\frac{30}{6}-\frac{\left(6\right)\left(116\right)-\left(19\right)\left(30\right)}{\left(6\right)\left(7\right)-19}\right)=\left(\frac{60}{23},\frac{94}{23}\right).$

Replacing $y_n=y_7=6$ with 14, 124/11, 9, 6, 1, −2, and −4 in turn, the regression lines in Figure 1 are obtained. As the replacement value decreases, the slopes of the corresponding lines in that graph decrease. The value 124/11 is the $y$-coordinate of the intersection of the regression line in Equation (32) on the first six observations and the line in Equation (16), $x=x_n=x_7=7$. From Theorems 2 and 3, the replacement value 124/11 yields the regression line on the first six observations, which is one of the regression lines that are found with the construction in Theorem 1 and contains the fixed point. In Figure 1 , it is the line with the second largest slope. Among the regression lines displayed in Figure 1 , the regression line for the original set of all the Observations (35) is the central line with slope that is approximately one.

The elements $h_i$ of the hat matrix in the same order as the Observations (35) are 0.433, 0.259, 0.326, 0.326, 0.416, 0.416, 0.568.

Only $h_7$ is greater than 0.5, indicating that it has high leverage. This high leverage is in the context of the best-fit line. The $z$-score of $x=7$ among the $x$-values is just 1.60, so that the point $\left(7,6\right)$ would not be considered an outlier as a member of the univariate set of $x$-values, but it appears to be impactful for the linear fitting.

The fixed points are far more fundamental and influential than previously realized. Literally, they are at the centers of regression diagnostic tests that involve altering or deleting an observation. The mechanical device in Figure 4 supplies clear insight into the movements of points in the processes of alteration and deletion of an observation and shows the scales of related measurements. This can be useful for visual learners. Instead of just line graphs, the movements can be observed or imagined with the mechanical device.

The two main facts that make the construction in Figure 3 and the mechanical device in Figure 4 possible are Theorem 1, giving the existence of the fixed point $P$, and Theorem 5, giving collinearity of $N$, $M$, and $T$. The figures show that there are two centers, which are the yellow items in Figure 4 . Careful examination of the formulas, shows that they can be written so that every occurrence of $x$- and $y$-values has the appropriate coordinate of $N\left({\bar{x}}_{\left(n-1\right)},{\bar{y}}_{\left(n-1\right)}\right)$ subtracted from it. Thus, $N$ is a center and a natural choice for the origin, if a translation of the coordinates were contemplated. The other center is the fixed point $P$.