Scientific research frequently employs scatter plots in which the relationship between the dependent variable y and the independent variable x can be modeled linearly—that is, by means of a regression line characterized by a slope m and a regression coefficient r of the sample considered.

This article aims to summarize the results of a theory regarding linear regression, whose relationships can always be interpreted as describing the input-output characteristic of a linear system. In the following discussion, I will refer to “linear channels” instead of “linear systems” because of the terminology I use in the article; however, the reader will appreciate that these are simply two different ways of describing the same functional relationship.

I have developed the theory in a series of articles concerning the mathematical properties of literary texts—both ancient and modern—particularly in relation to translations [1]-[17]. Another important application was reported in Reference [18].

The conviction that the theory is applicable to any scientific discipline—provided a linear input-output relationship exists—has led me to summarize the main results in a general form that the readers can easily apply to their specific problem. In any case, in the articles mentioned, there are numerous experimental applications demonstrating how the theory can be applied to real-world data.

After this introduction, Section 2 defines the single channel and its noise-to-signal power ratio; Section 3 presents and discusses the geometrical representation of the noise-to-signal power ratio; Section 4 defines the cross channel; Section 5 compares a stochastic channel versus its deterministic version; Section 6 studies the parallel connection of channels; Section 7 studies the series connection of channels; Section 8 reports a summary and a conclusion. Appendix A lists the mathematical symbols used in the article and their meaning (Table A1); Appendix B proves a fundamental relationship; Appendix C details the calculation of a formula of the main text.

In this section, I recall the general theory on stochastic variables linearly connected [19]. Let y (dependent variable) and x (independent variable) linked by the line:

Equation (1) models a deterministic relationship through the slope m o . The intercept q = 0 ; however, if q ≠ 0 , the theory can be fully applied by defining a new dependent variable:

In the real world, the relationship between y and x is never deterministic, i.e., given by Equation (1), but stochastic (random). Equation (1) models, in fact, two variables perfectly correlated, with correlation coefficient | r | = 1 and slope m o . As these conditions are never found in any application, Equation (1) transforms into:

In Equation (3), m ≠ m o is a multiplicative distortion (bias); n is an additive stochastic variable, with zero mean value, which renders the linear channel partially stochastic, namely “noisy”, using a communication system terminology [20]. In other words, Equation (3) models a noisy channel with multiplicative bias.

Figure 1 shows the flow chart describing Equation (3) with a system/channel representation. The black box indicated with m represents a distorted deterministic channel; the black box indicated with r represents the parallel channel due to the scattering of y around the regression line.

Figure 1.Flow chart of linear systems. Upper panel: Deterministic channel with multiplicative distortion (bias) m . Lower panel: Noisy channel with multiplicative bias and additive noise source.

Now, let us consider:

a) The variance of the difference m x − m o x , referred to as the “regression noise” power N m .

b) The variance of the difference between the values not lying on Equation (3) ( r ≠ 1 ), and those lying on it ( r = 1 ), referred to as the “correlation noise” power N r . This “noise” is due to the spread of y around the line given by Equation (3), due to the stochastic noise n .

c) The variances s x 2 and s y 2 of x and y involved in Equation (3).

In case (a), we get the difference ( m − m o ) x , therefore the variance (or power) of the values of y lying on the regression line is given by:

Now, we define the “regression noise-to-signal power ratio” R m (NSRm) between the output and the input:

In case (b), the fraction of the variance s y 2 due to the values of y not lying on the regression line is given by:

The parameter r 2 is called the coefficient of determination and it gives the fraction of the variance of y explained by the regression line r 2 s y 2 [19]. However, this variance is correlated with the slope m because r 2 s y 2 is related to m according to:

Figure 2 shows the flow chart of variances.

Figure 2.Flow chart of variances: r 2 s y 2 is the output variance of the values lying on the regression line; ( 1 − r 2 ) s y 2 is the output variance due to the values of y not lying on the regression line.

Therefore, inserting Equation (7) in Equation (6), we get the “correlation noise-to-signal power ratio”, R r (NSRr):

Since the two noise variances (powers) are additive, the total noise-to-signal (NSR) R of the channel shown in Figure 2 is given by:

Therefore, R depends only on the two parameters m and r of the regression line, independently of their sign, explicitly given by:

Finally, the signal-to-noise ratio (SNR) γ , is given by:

Of course, we expect that no real channel can yield r = ± 1 and m = m o , in which case γ = ∞ , a channel purely deterministic. In experimental scatterplots, very likely, | r | < 1 , m ≠ m o . It should be noted that the SNR can be considered an indicator of the degree of deterministic nature of a channel: the higher the SNR, the greater the “degree” of the channel’s deterministic nature.

In conclusion, the slope m measures the multiplicative “distortion” of the dependent variable y with respect to the independent variable x in the deterministic channel; the correlation coefficient r measures how “precise” the best linear fit is.

Finally, notice the more direct and insightful analysis that can be achieved by using the NSR instead of the more common SNR because, in Equations (9) and (10), the single channel NSRs simply add together. This makes easier to study, for example, which addend determines R , and thus γ , while this is by far less easy with Equation (11).

Figure 3 shows the signal-to-noise ratio drawn with varying multiplicative bias m and r = 0.99 , by assuming m o = 1 (blue lines), m o = 1.5 (red lines); m o = 2 (green lines). Notice that at m = m o , R m = 0 and 1 R m = ∞ . At these points, R is determined only by R r , R = R r dashed lines. The lower continuous lines refer the full signal-to-noise ratio Γ.

Figure 3.Signal-to-noise ratio drawn as a function of m , by assuming r = 0.99 for all, and m o = 1 (blue lines), m o = 1.5 (red lines); m o = 2 (green lines). Notice that at m = m o , R m = 0 and 1 R m = ∞ , see line labelled SNR_m. At these points, R = R r , see line labelled SNR_r, dashed lines. The lower continuous lines refer to the full signal-to-noise ratio Γ.

Figure 4 shows the full signal-to-noise ratio Γ as a function of m and r . We can notice that as | r | → 1 , Γ → ∞ , at m ≅ m o . In this region, Γ changes vary rapidly as small variations in r give very large variations in Γ, for any m ≅ m o .

The choice of using the NSR instead of the more common SNR also leads to a useful graphical representation of Equation (9) that can guide analysis and design, first used in my deep-space communication studies [20], as shown in the next section.

Figure 4.Γ versus m and r ; as r → 1 , Γ → ∞ , at m ≅ m o = 1 ; 1.5 ; 2.0 . In this region, Γ changes vary rapidly as small variations in r give very large variations in Γ, for any m o .

As discussed in [20] for deep-space radio link design, among other features not of interest here, adopting the noise-to-signal power ratio instead of the more common signal-to-noise ratio allows a geometrical representation, which immediately shows how R r and R m , through their square roots, contribute to the total R.

In fact, we can represent Equation (10) graphically by considering the two orthogonal variables:

By setting the noise-to-signal ratio R = R o , being R o = 1 / γ o a constant, X and Y trace an arch of the circle with radius R o = R m + R r in the first Cartesian quadrant. All points inside the circle correspond to R < R o ( γ > γ o ); the origin of the axes corresponds to R = 0 of the purely deterministic channel, m = m o and r = 1 . The signal-to-noise power ratio γ = 1 / R becomes infinite at the origin and decreases as the radius of the circle increases. Figure 5 shows a universal chart with lines of equal signal-to-noise ratio Γ.

Channels can be combined and compared in various ways; therefore, in the following sections, I will examine the most common combinations and make some comparisons.

Figure 5.Lines of equal signal-to-noise ratio drawn in the universal chart of coordinates X = R m and Y = R r .

Let us study how the output variable y k of channel k relates to the output variable y j of another similar channel j for the same input x k = x j = x . This new channel is termed “cross channel”.

Let us consider the scatterplot k , referred to the couple x k , y k and the scatterplot j , referred to the couple x j , y j , in which the independent variable x and the dependent variable y are linked by the following regression lines in what we can define a “deterministic channel”:

As discussed in Section 2, Equations (15) and (16) do not give the full relationship between the two variables because they link only conditional average values, measured by the slopes m k and m j . According to Equation (3), we can write more general linear relationships by considering the scattering of the data, always present in experiments, modelled by zero-mean noise sources n k and n j .

Now, we can develop a series of interesting investigations on these equations. By eliminating x , we can compare the dependent variable y j of Equation (18) to the dependent variable y k of Equation (17) for x k = x j = x . In doing so, we can find the regression line and the correlation coefficient of the new scatterplot linking y j to y k without the availability of the scatterplot itself, assuming the same value of the independent variable x .

By eliminating x between Equation (17) and Equation (18), we get:

Compared to the new independent variable y k , the slope m k j of the regression line is given by:

Because the two noise sources are additive, the total noise is given by:

Figure 6 shows the flow chart describing the cross-channel.

Figure 6.Chart describing the cross-channel signal flow.

Now, defining the expected (deterministic) ratio between the two slopes m o , k j = m o , j / m o , k , we get R m of the cross channel:

To proceed to find R r , it is necessary to determine the correlation coefficient r k j between − m k j n k and n j because in Equation (21), the two noise sources might be correlated, with unknown correlation coefficient r k j . Appendix B shows that:

Therefore, R r of the cross channel is:

In conclusion, we can determine the slope and the correlation coefficient of the scatterplot between y j and y k for the same value of the independent variable x . Now, the availability of this scatterplot is experimentally very rare because it is unlikely to find values of y k and y j for exactly the same value of x , therefore cross channels can reveal relationships very difficult to discover experimentally.

In the next section, we study how the output of a deterministic channel relates to the output of its stochastic version.

We compare a deterministic channel k (slope m ) with a stochastic channel j derived from channel k by adding noise (slope m , correlation coefficient r ). This is a particular cross channel, therefore, from the theory of Section 4, we get:

In conclusion, in transforming the deterministic channel into its stochastic version, only the correlation noise is present, therefore the SNR is given by:

Equation (31) coincides with the ratio between the variance explained by the regression line (proportional to r 2 ) and the variance due to the scattering (correlation noise), proportional to 1 − r 2 .

The development of this section confirms that the SNR can be considered—in any case—an indicator of the degree of deterministic nature of a channel: the higher the SNR, the greater the “degree” of the channel’s deterministic nature.

So far, we have considered single channels. In the following sections, we examine the series and parallel connection of channels and determine the signal-to-noise ratio of the overall channel.

Let us consider the parallel connection of the channels, Figure 7. The parallel connection gives a single channel with the following input-output relationship:

Now, since x k = x j = x , Equation (32) is equivalent to the single channel

Therefore, Equation (33) coincides with Equation (3) in which:

Figure 7.Parallel connection of channels.

Let us calculate the correlation coefficient of this channel. From Equation (7)

Therefore, after a few mathematical steps (Appendix C), we get:

Notice that if m k = m j then

Figure 8 shows how r varies as a function of r k and r j according to Equation (38). Notice that if r k = r j = 1 , then r = 1 .

In conclusion, the noise-to-signal ratios R m and R r are given, respectively, by Equation (5) and Equation (8), with m given by Equation (34) and r given by Equation (37). Of course, the theory of parallel channels can be extended to the parallel of n > 2 single channels by applying Equations (34) and (37) recursively.

In this section, we consider a channel made by a series of single channels, Figure 9. The “experimental” slope of the overall channel is given by:

Therefore, in a series of n channels, the expected overall slope is given by;

Therefore, the noise-to-signal regression noise R m is given by:

From Figure 9, it is evident that the output noise of a preceding channel produces additive noise at the output of the next channels. Let us calculate the correlation noise-to-signal ratio R r at the output of the overall channel.

Figure 8.Curves showing how r varies as a function of r k = r 2 versus r j = r 1 , by assuming m k = m j , according to Equation (38). Cyan line r 1 = 0.1 ; black line r 1 = 0.3 ; magenta line r 1 = 0.5 ; green r 1 = 0.8 ; red line r 1 = 0.9 ; blue line r 1 = 1 .

Figure 9.Series connection of channels.

Theorem. The correlation noise-to-signal ratio R = R r of n linear channels in series, each characterized by the correlation noise-to-signal ratio R i , is given by:

Proof. Let the three linear input-output relationships of the isolated channels of Figure 9 (i.e., before connecting them in series) given by:

Let s x 2 , s y 2 , s z 2 the variances (power) of the input variables to the single channels, and let N 1 = N 1 r , N 2 = N 2 r , N 3 = N 3 r the variances (power) of the zero-mean noise n 1 , n 2 , n 3 , then the NSRs of the isolated channels are given by:

When the first two channels are connected in series, the input to the second channel must also include the output noise of the first channel, therefore we get the modified output variable z ˘ :

In Equation (49), m 2 m 1 x is the output “signal” and m 2 n 1 + n 2 is the output noise, therefore, the NSR at the output of the second channel is:

Now, for three channels in series, it is sufficient to consider R given by Equation (50) as the input NSR to the third single channel to obtain the final NSR and prove Equation (42):

□

Finally, notice that R of Equation (51) is proportional to the mean 〈 R i 〉 :

In other words, the series connection of channels averages the single correlation noise-to-signal ratios R i .

I have summarized the results of a theory regarding linear regression developed in series of papers [1]-[17]. The linear relationship between the dependent variable y and the independent variable x is interpretable as describing the input-output characteristic of a linear system, or a linear channel.

The conviction that the theory is applicable to any scientific discipline—provided a linear input-output relationship exists—has led me to summarize the main results in a general form that the readers can easily apply to their specific problem.

Defining the regression noise-to-signal power ratio (NSR) R m and the correlation noise-to-signal power ratio, R r , I have shown that the overall noise-to-signal ratio R is the sum of the two. I have highlighted the more direct and insightful analysis achievable by using the NSR instead of the more common signal-to-noise ratio (SNR) because it makes easier to study which addend determines R and allows a useful graphical representation.

I have underlined that the SNR can be considered an indicator of the degree of deterministic nature of a channel, because the higher the SNR, the greater the “degree” of the channel’s deterministic nature.

In the paper, I have considered various connections of single channels (single, cross, deterministic, parallel, series), whose main results are summarized in Table 1.

Table 1. Channels configuration parameters. For the meaning of the mathematical symbols, see the main text and Appendix A.

In conclusion, the theory and its mathematical results concerning the input-output characteristics of linear channels, described by experimental regression lines, should be useful in any scientific discipline in which the experimental problem is modelled linearly by regression lines.

I wish to thank Lucia Matricciani for drawing Figure 1, Figure 2, Figure 6, Figure 7, Figure 9, Figure A1, Figure A2.

Table A1. lists of the mathematical symbols used in the article with their meaning.

Consider the stochastic variables x and y with variances s x 2 and s y 2 respectively. Consider two vectors in the plane whose lengths are proportional to s x and s y and separated by an angle θ whose cosine is given by the correlation coefficient r (Figure A1). The length of the diagonal of the parallelogram determined by them can be computed using the law of cosines ([21], p. 127):

Now Equation (A1) is just the variance of the stochastic variable z = x + y , therefore the length of the diagonal d is the standard deviation of z . Thus, the addition of the stochastic variables x and y corresponds to vector addition of the vectors representing their standard deviations.

Figure A1.Geometrical representation of standard deviations ([21], p. 127).The standard deviations can be represented by two vectors whose lengths are proportional to s x and s y respectively and separated by an angle θ whose cosine is given by the correlation coefficient r , according to the law of cosines.

Let us now apply this representation to the relationships of the problem in this article:

We get

We wish to calculate the correlation coefficient r k j between the two noise sources, here rewritten by changing the negative sign into positive without any impact on the calculations when dealing with noise:

Figure A2 shows the vector representations of the standard deviations of Equations (A4) and (A5).

Figure A2.Vector representations of the standard deviations of Equations (A4) and (A5) to find the correlation coefficient r k j between m k j n k and n j .

Now r k j is the cosine of the angle θ k j between the vectors s n j and s n k , therefore:

Now, since

We get the final relationship, Equation (23):

In Equation (36), the variance, s y 2 , of the two correlated stochastic variables in Equation (32), is given by Equation (A1), here written for our problem:

In Equation (A11), r k j is given by Equation (23). Therefore:

From Equation (A14), we get Equation (37).