Single-equation regression models¶

Introduction¶

Economic theory makes statements or hypotheses that are primarily qualitative in nature. The main concern of mathematical economics, on the other hand, is to express economic theory in mathematical form (equations) without considering whether the theory can be measured or empirically verified. Econometrics, in turn, is mainly concerned with the verification of economic theory. Among the related fields, we may also mention economic statistics, which is primarily concerned with the collection, processing, and presentation of economic data. The information collected constitutes the raw data for econometric work.

We may also distinguish between theoretical econometrics and applied econometrics. The former deals with the development of appropriate methods for measuring the economic relationships specified in econometric models, whereas the latter uses these tools to study one or more fields of economics.

In general terms, the traditional econometric methodology follows these steps:

Statement of the theory or hypothesis.
Specification of the mathematical model of the theory.
Specification of the statistical or econometric model.
Collection of data.
Estimation of the parameters of the econometric model.
Hypothesis testing.
Projection or forecasting.
Use of the model for control or policy purposes.

To illustrate these steps, let us consider the well-known Keynesian theory of consumption.

1. Statement of the theory or hypothesis

The fundamental psychological law [...] is that men [women] are disposed, as a rule and on average, to increase their consumption as their income increases, but not by as much as the increase in income. (Keynes)

Keynes postulated that the marginal propensity to consume (MPC), the rate of change in consumption resulting from a one-unit change in income, is greater than zero but less than 1.

2. Specification of the mathematical model of the theory

Consumption function (relates income and consumption):

y=\beta_1+\beta_2 x \qquad 0<\beta_2 < 1

(1)

$\beta_1$ and $\beta_2$ are parameters (intercept and slope coefficient)
$y$ ( $x$ ) is the dependent (independent or explanatory) variable

3. Specification of the statistical or econometric model

The deterministic model must be modified:

y=\beta_1+\beta_2x+u

(2)

$u$ : called the disturbance (or error term), is a random (or stochastic) variable

4. Collection of data

To estimate the model, data are required.

5. Estimation of the parameters of the econometric model

It is necessary to estimate the values of the parameters from the data. Regression analysis is the main tool for this purpose.

6. Hypothesis testing

It is necessary to perform a hypothesis test (or statistical inference) to verify whether the result is statistically significant in supporting or rejecting the theory or hypothesis proposed in step 1.

7. Projection or forecasting

If the model does not refute the hypothesis under consideration, it may be used to predict future values.

8. Use of the model for control or policy purposes

An estimated model may be used for control purposes or for policy formulation.

Some important issues:

The main idea behind regression analysis is the statistical dependence of one variable (the dependent variable) on one or more other variables (the explanatory variables).
The objective of this analysis is to estimate and/or predict the mean value of the dependent variable based on the known or fixed values of the explanatory variables.
In practice, the success of regression analysis depends on the availability of adequate data.

In addition, regarding notation, we denote $E(y|x)$ as the expected value of $y$ given the value of $x$ , that is, a conditional mean, in contrast to $E(y)$ , which is an unconditional mean. Furthermore, we may write:

E(y|x)=f(x)

(3)

That is, $f(x)$ represents a function of the explanatory variable $x$ . This function may be called: the conditional expectation function (CEF), the population regression function (PRF), or the population regression (PR).

Evidently, if we have $N$ pairs of observations $(y_i,x_i)$ , we may write:

y_i=E(y|x_i)+u_i

(4)

Where the stochastic disturbance gives the difference between the particular value $y_i$ and the expected value $E(y|x_i)$ for the corresponding $x_i$ . By definition, we also have that the expected value, that is, the mean of $u$ , is 0.

It is important to note that up to this point the discussion has focused on data from a population. Frequently, however, we only have a sample from this population. In this case, we have the sample regression function (SRF), and we place a “hat” on the dependent variable and the parameters:

\widehat{y}_{i}=\widehat{\beta}_{1}+\widehat{\beta}_{2}x_{i}

(5)

By this we mean that $\widehat{A}$ is an estimator of $A$ . We may say that the observed $y_i$ can be expressed as:

y_i=\widehat{y}_{i}+\widehat{u}_i

(6)

We say that this equation is the stochastic version of the SRF, and $\widehat{u}_i$ now takes the name of residual term (in the sample).

Some additional relevant issues:

The key concept underlying regression analysis is that of the conditional expectation function (CEF) or population regression function (PRF). Our objective in regression analysis is to examine how the mean value of the dependent variable (or regressand) varies with the value of the explanatory variable (regressor).
This book deals mainly with linear PRFs, that is, regressions that are linear in the parameters. They may or may not be linear in the regressand or in the regressors.
For empirical purposes, what matters is the stochastic PRF. The stochastic error term, $u_i$ , plays a fundamental role in the estimation of the PRF.
The PRF is an idealized concept, since in practice we very rarely have access to the entire population of interest. In general, we only have a sample of observations from the population. Therefore, we use stochastic sample regression functions (SRFs) to estimate the PRF.

Ordinary Least Squares (OLS) Method¶

We have the two-variable PRF:

y_i=\beta_1+\beta_2x_i+u_i

(7)

But we must estimate it through the SRF:

y_i=\widehat{\beta}_1+\widehat{\beta}_2x_i+\widehat{u}_i=\widehat{y}_i+\widehat{u}_i

(8)

We may then write the residuals as:

\widehat{u}_i=y_i-\widehat{y}_i

(9)

We therefore want to minimize the residuals:

\sum\widehat{u}_i^2=\sum(y_i-\widehat{y}_i)^2

(10)

Squaring allows us to ignore the signs, that is, if $y_i-\widehat{y}_i < 0$ , we obtain the same result as when $y_i-\widehat{y}_i>0$ . It also increases the importance of larger residuals in absolute value. Since the data $(y_i,x_i)$ are obtained from the sample, we want to estimate the values $(\widehat{\beta}_1,\widehat{\beta}_2)$ . Therefore, the residual sum of squares function is a function of the estimators:

\sum\widehat{u}_i^2=f(\widehat{\beta}_1,\widehat{\beta}_2)

(11)

Solving by the process of differentiation, we obtain:

\widehat{\beta}_2=\frac{\sum(x_i-\overline{x})(y_i-\overline{y})}{\sum(x_i-\overline{x})^2}\\ \widehat{\beta}_1 =\overline{y}-\widehat{\beta}_2\overline{x}

(12)

Calculus¶

fWe can apply the method of linear regression to a large dataset in order to obtain the line that best describes that dataset. Evidently, for this method to be useful, we need there to exist a line that satisfactorily describes the data. If we have a single independent variable $x$ that we expect to describe a single dependent variable $y$ linearly (a straight line), then, for a set of input points $\left\{ x_{i}\right\}$ , we want to find a set of output points $\left\{ y_{i}\right\}$ that can be written as follows:

y_{i}=b_{0}+b_{1}x_{i}+e_{i}

(13)

where:

$b_{0}$ is the regression constant;
$b_{1}$ is the regression coefficient;
$e_{i}$ is the error obtained when we try to predict $y_{i}$ from $x_{i}$ .

When we apply the linear regression model, we want to minimize the error. To do so, we need a measure of the error. We will use the sum of squared errors:

E=\sum^{N}_{i=0}e^{2}_{i}

(14)

Considering that we have $N$ points, we want to find $b_{0}$ and $b_{1}$ that minimize this quantity. We may rewrite this as:

E=\sum^{N}_{i=0}\left(y_{i}-b_{0}-b_{1}x_{i}\right)^{2}

(15)

We may use the gradient to find the minimum of a function. The minimum must satisfy $\nabla f=0$ . Since we want to find the minimum with respect to both terms $b_{j}$ , we have:

\nabla E=\left(\frac{\partial E}{\partial b_{0}},\frac{\partial E}{\partial b_{1}}\right)

(16)

Calculating the derivative with respect to $b_{0}$ :

\begin{align} \frac{\partial E}{\partial b_{0}} & =\frac{\partial}{\partial b_{0}}\sum^{N}_{i=0}\left(y_{i}-b_{0}-b_{1}x_{i}\right)^{2}\\ & =\sum^{N}_{i=0}\frac{\partial}{\partial b_{0}}\left(y_{i}-b_{0}-b_{1}x_{i}\right)^{2}\\ & =\sum^{N}_{i=0}2\left(y_{i}-b_{0}-b_{1}x_{i}\right)\frac{\partial}{\partial b_{0}}\left(y_{i}-b_{0}-b_{1}x_{i}\right)\\ & =\sum^{N}_{i=0}2\left(y_{i}-b_{0}-b_{1}x_{i}\right)\left(-1\right)\\ & =-2\sum^{N}_{i=0}\left(y_{i}-b_{0}-b_{1}x_{i}\right) \end{align}

(17)

Proceeding similarly for $b_{1}$ :

\frac{\partial E}{\partial b_{1}}=-2\sum^{N}_{i=0}x_{i}\left(y_{i}-b_{0}-b_{1}x_{i}\right)

(18)

To minimize, we require the following condition to be satisfied:

\nabla E=\left(\frac{\partial E}{\partial b_{0}},\frac{\partial E}{\partial b_{1}}\right)=\left(0,0\right)

(19)

Then:

\begin{align} 0 & =-2\sum^{N}_{i=0}\left(y_{i}-b_{0}-b_{1}x_{i}\right)\\ 0 & =-2\sum^{N}_{i=0}x_{i}\left(y_{i}-b_{0}-b_{1}x_{i}\right) \end{align}

(20)

To ensure that this is a minimum rather than a maximum, we must verify that the second derivative is positive:

\begin{align} \frac{\partial^{2}E}{\partial b^{2}_{0}} & =2\sum^{N}_{i=0}1=2N\\ \frac{\partial^{2}E}{\partial b^{2}_{1}} & =2\sum^{N}_{i=0}x_{i}x_{i}=2\sum^{N}_{i=0}x^{2}_{i} \end{align}

(21)

Since $N>0$ and $x^{2}\geq0$ , we can be certain that this is a minimum. Manipulating the first equation in the system, we obtain:

\begin{align} 0 & =-2\sum^{N}_{i=0}\left(y_{i}-b_{0}-b_{1}x_{i}\right)\\ 0 & =\sum^{N}_{i=0}y_{i}-\sum^{N}_{i=0}b_{0}-\sum^{N}_{i=0}b_{1}x_{i}\\ b_{0}\sum^{N}_{i=0}1 & =\sum^{N}_{i=0}y_{i}-\sum^{N}_{i=0}b_{1}x_{i}\\ Nb_{0} & =\sum^{N}_{i=0}y_{i}-\sum^{N}_{i=0}b_{1}x_{i}\\ b_{0} & =\sum^{N}_{i=0}\frac{y_{i}}{N}-b_{1}\sum^{N}_{i=0}\frac{x_{i}}{N}\\ b_{0} & =\overline{y}-b_{1}\overline{x} \end{align}

(22)

And for the second equation in the system, we have:

\begin{align} 0 & =-2\sum^{N}_{i=0}x_{i}\left(y_{i}-b_{0}-b_{1}x_{i}\right)\\ 0 & =\sum^{N}_{i=0}x_{i}y_{i}-b_{0}\sum^{N}_{i=0}x_{i}-\sum^{N}_{i=0}b_{1}x^{2}_{i}\\ b_{1}\sum^{N}_{i=0}x^{2}_{i} & =\sum^{N}_{i=0}x_{i}y_{i}-b_{0}\sum^{N}_{i=0}x_{i} \end{align}

(23)

Combining the results, we obtain:

\begin{align} b_{1}\sum^{N}_{i=0}x^{2}_{i} & =\sum^{N}_{i=0}x_{i}y_{i}-b_{0}\sum^{N}_{i=0}x_{i}\\ b_{1}\sum^{N}_{i=0}x^{2}_{i} & =\sum^{N}_{i=0}x_{i}y_{i}-\left(\sum^{N}_{i=0}\frac{y_{i}}{N}-b_{1}\sum^{N}_{i=0}\frac{x_{i}}{N}\right)\sum^{N}_{i=0}x_{i}\\ b_{1}\sum^{N}_{i=0}x^{2}_{i} & =\sum^{N}_{i=0}x_{i}y_{i}-\frac{1}{N}\left(\sum^{N}_{i=0}y_{i}\right)\left(\sum^{N}_{i=0}x_{i}\right)+\frac{b_{1}}{N}\left(\sum^{N}_{i=0}x_{i}\right)^{2}\\ b_{1}\left(\sum^{N}_{i=0}x^{2}_{i}-\frac{1}{N}\left(\sum^{N}_{i=0}x_{i}\right)^{2}\right) & =\sum^{N}_{i=0}x_{i}y_{i}-\frac{1}{N}\left(\sum^{N}_{i=0}y_{i}\right)\left(\sum^{N}_{i=0}x_{i}\right) \end{align}

(24)

Therefore, $b_{1}$ is:

b_{1}=\frac{\sum^{N}_{i=0}x_{i}y_{i}-\frac{1}{N}\left(\sum^{N}_{i=0}y_{i}\right)\left(\sum^{N}_{i=0}x_{i}\right)}{\left(\sum^{N}_{i=0}x^{2}_{i}-\frac{1}{N}\left(\sum^{N}_{i=0}x_{i}\right)^{2}\right)}

(25)

That is:

b_{1}=\frac{\sum^{N}_{i=0}x_{i}y_{i}-N\overline{y}\,\overline{x}}{\sum^{N}_{i=0}x^{2}_{i}-N\overline{x}^{2}}, \qquad b_{0}=\overline{y}-b_{1}\overline{x}

(26)

From these constants, we construct the following line equation:

y=b_{0}+b_{1}x

(27)

We may also demonstrate that $b_{1}=\widehat{\beta}_{2}$ :

\widehat{\beta}_{2}= \frac{\sum(x_{i}-\overline{x})(y_{i}-\overline{y})}{\sum(x_{i}-\overline{x})^{2}} =\frac{\sum(x_{i}y_{i}-y_{i}\overline{x}-\overline{y}x_{i}-\overline{y}\overline{x})}{\sum(x^{2}_{i}-2x_{i}\overline{x}+\overline{x}^{2})}

(28)

=\frac{\sum\left(x_{i}y_{i}-\overline{y}\,\overline{x}\right)-N\left(\sum\frac{y_{i}}{N}\right)\overline{x}-N\left(\sum\frac{x_{i}}{N}\right)\overline{y}}{\sum(x^{2}_{i}-2N\left(\sum\frac{x_{i}}{N}\right)\overline{x}+\overline{x}^{2})}

(29)

=\frac{\sum x_{i}y_{i}-\overline{y}\,\overline{x}\sum1-N\overline{y}\,\overline{x}-N\overline{x}\,\overline{y}}{\sum(x^{2}_{i}-2\overline{x}^{2}+\overline{x}^{2})}

(30)

=\frac{\sum x_{i}y_{i}-N\overline{y}\,\overline{x}}{\sum x^{2}_{i}-\overline{x}^{2}\sum1} =\frac{\sum x_{i}y_{i}-N\overline{y}\,\overline{x}}{\sum x^{2}_{i}-N\overline{x}^{2}} =b_{1}

(31)

pip install -q linearmodels

# @title
import numpy as np
import statsmodels.api as sm


#Manual
I = np.random.randint(0,100)
x = np.random.randint(0, 100, I)
y = np.random.randint(0, 100, I)

#x = np.array([1,2,3,4,5,6,7,8,9,10])
#y = np.array([5,8,9,13,15,18,20,25,27,30])

N=len(x)
xm=np.mean(x)
ym=np.mean(y)
s1=0;s2=0
for i in (range(N)):
    s1+=x[i]*y[i]
    s2+=x[i]*x[i]

b1=(s1-N*ym*xm)/(s2-N*xm*xm)
b0=ym-b1*xm
print("Manual")
print("const: "+str(b0)+"\nx1:    "+str(b1))

### Auto


X = sm.add_constant(x)

# regressão OLS
modelo = sm.OLS(y, X).fit()

# resultados
# print(modelo.summary())
print("\nAuto")
print("const: "+str(modelo.params[0])+"\nx1:    "+str(modelo.params[1]))

print("\nOs coeficientes angulares são {:.2f}% iguais.".format(100*b1/modelo.params[1]))

Manual
const: 48.71590353827139
x1:    -0.06073409789888189

Auto
const: 48.7159035382714
x1:    -0.06073409789888215

Os coeficientes angulares são 100.00% iguais.

OLS with Mean Transformation (FE)¶

If we have time-series data for more than one country, we may be interested only in the variation of each series over time, disregarding average level differences across countries. A method typically employed in this case is to subtract from each observation the time mean of its respective series, so that all series become centered around zero. OLS is then applied to the transformed data. In this case, the estimated coefficient depends only on the internal variations of each series over time. Thus, we are interested only in the slope coefficient $b_{1}$ and not in the intercept $b_{0}$ obtained through OLS.

# @title
from linearmodels.panel import PanelOLS
import pandas as pd


#Manual
x1 = np.random.randint(0, 100, I)
y1 = np.random.randint(0, 100, I)

nx=x-np.mean(x)
nx1=x1-np.mean(x1)
ny=y-np.mean(y)
ny1=y1-np.mean(y1)

x2=np.concatenate((nx,nx1))
y2=np.concatenate((ny,ny1))

N=len(x2)
xm=np.mean(x2)
ym=np.mean(y2)
s1=0;s2=0
for i in (range(N)):
    s1+=x2[i]*y2[i]
    s2+=x2[i]*x2[i]

b1=(s1-N*ym*xm)/(s2-N*xm*xm)
b0=ym-b1*xm
print("Manual")
print("x1:    "+str(b1))

#Auto
# dataframe painel
df = pd.DataFrame({
    'id': [1]*I + [2]*I,
    'tempo': list(range(I)) + list(range(I)),
    'x': np.concatenate([x, x1]),
    'y': np.concatenate([y, y1])
})

#print(df)

# índice painel
df = df.set_index(['id', 'tempo'])

# modelo FE
modelo = PanelOLS.from_formula(
    'y ~ x + EntityEffects',
    data=df
)

resultado = modelo.fit()

print("\nAuto")
print("x1:    "+str(resultado.params['x']))

print("\nOs coeficientes angulares são {:.2f}% iguais.".format(100*b1/resultado.params['x']))

Manual
x1:    0.1577693520786498

Auto
x1:    0.15776935207864973

Os coeficientes angulares são 100.00% iguais.

OLS with Mean Transformation and Weighting (FE + Weighted)¶

Now that we are dealing with different countries, it may also be the case that, when applying the final linear estimation, we wish to assign different weights to different observations. To do so, we may return to the error function minimized by OLS, $E=\sum^{N}_{i=0}e^{2}_{i},$ and assign different weights $w_{i}$ to each error term $E=\sum^{N}_{i=0}w_{i}e^{2}_{i}.$ We recover the usual OLS case when $w_{i}=1$ for all observations. The remainder of the derivation of $b_{1}$ is analogous, now considering the term $w_{i}$ .

The derivatives with respect to $b_{0}$ and $b_{1}$ become, respectively:

\frac{\partial E}{\partial b_{0}} =-2\sum^{N}_{i=0}w_{i}\left(y_{i}-b_{0}-b_{1}x_{i}\right)

(32)

\frac{\partial E}{\partial b_{1}} =-2\sum^{N}_{i=0}w_{i}x_{i}\left(y_{i}-b_{0}-b_{1}x_{i}\right)

(33)

Summing the weights over all $N$ data points, $\sum^{N}_{i}w_{i},$ is equivalent to summing the weights over all $M$ observations of each one of the $K$ series, $\sum^{K}_{n}\sum^{M}_{t}w_{n,t}$ . We are considering that all series have the same length, such that $K\cdot M=N$ . Since we may interchange the order of summation, then:

\sum^{N}_{i}w_{i} = \sum^{K}_{n}\sum^{M}_{t}w_{n,t} = \sum^{M}_{t}\left(\sum^{K}_{n}w_{n,t}\right) = \sum^{M}_{t}1 = M

(34)

This is because our weights are distributed such that, if we sum all the weights of all series corresponding to any given time $t$ , we must necessarily obtain 1, that is $\sum_{n}w_{n,t}=1$ . Making the additional assumption that the weights of each series remain constant over time, $w_{n}=w_{n,t},$ we may factor $w_{n}$ out of the summation over time. Recalling that, in order to minimize the error, we must satisfy $\left(\frac{\partial E}{\partial b_{0}},\frac{\partial E}{\partial b_{1}}\right)=\left(0,0\right)$ , we may arrive at the first equation:

\begin{align} b_{0}\sum^{N}_{i=0}w_{i} & =\sum^{N}_{i=0}w_{i}y_{i}-\sum^{N}_{i=0}w_{i}b_{1}x_{i}\\ b_{0}\sum^{K}_{n}\sum^{M}_{t}w_{n,t} & =\sum^{K}_{n}\sum^{M}_{t}w_{n,t}y_{n,t}-\sum^{K}_{n}\sum^{M}_{t}w_{n,t}b_{1}x_{n,t}\\ b_{0}M & =\sum^{K}_{n}w_{n}\sum^{M}_{t}y_{n,t}-\sum^{K}_{n}w_{n}b_{1}\sum^{M}_{t}x_{n,t}\\ b_{0} & =\sum^{K}_{n}w_{n}\frac{\sum^{M}_{t}y_{n,t}}{M}-\sum^{K}_{n}w_{n}b_{1}\frac{\sum^{M}_{t}x_{n,t}}{M}\\ b_{0} & =\sum^{K}_{n}w_{n}\overline{y}_{n}-b_{1}\sum^{K}_{n}w_{n}\overline{x}_{n}\\ b_{0} & =\widehat{y}-b_{1}\widehat{x} \end{align}

(35)

That is, we obtain something quite similar to the usual OLS case. The difference is that, instead of having only a single mean over all observations, we now have a weighted sum of the mean of each series, where the weights correspond to those assigned to each series. Essentially, we replace $\overline{z}\rightarrow\sum^{K}_{n}w_{n}\overline{z}_{n}=\widehat{z}$ .Now, regarding the second equation, we have:

\begin{align} b_{1}\sum^{K}_{n}w_{n}\sum^{M}_{t}x^{2}_{n,t} & =\sum^{K}_{n}w_{n}\sum^{M}_{t}x_{n,t}y_{n,t}-b_{0}\sum^{K}_{n}w_{n}\sum^{M}_{t}x_{n,t}\\ b_{1}\sum^{K}_{n}w_{n}\sum^{M}_{t}x^{2}_{n,t} & =\sum^{K}_{n}w_{n}\sum^{M}_{t}x_{n,t}y_{n,t}-b_{0}M\sum^{K}_{n}w_{n}\overline{x}_{n}\\ b_{1}\sum^{K}_{n}w_{n}\sum^{M}_{t}x^{2}_{n,t} & =\sum^{K}_{n}w_{n}\sum^{M}_{t}x_{n,t}y_{n,t}-b_{0}M\widehat{x} \end{align}

(36)

Combining the results, we obtain:

\begin{align} b_{1}\sum^{K}_{n}w_{n}\sum^{M}_{t}x^{2}_{n,t} & =\sum^{K}_{n}w_{n}\sum^{M}_{t}x_{n,t}y_{n,t}-\left(\widehat{y}-b_{1}\widehat{x}\right)M\widehat{x}\\ b_{1}\left(\sum^{K}_{n}w_{n}\sum^{M}_{t}x^{2}_{n,t}-M\widehat{x}^{2}\right) & =\sum^{K}_{n}w_{n}\sum^{M}_{t}x_{n,t}y_{n,t}-M\widehat{y}\widehat{x} \end{align}

(37)

Therefore, $b_{1}$ is:

b_{1}= \frac{\sum^{K}_{n}w_{n}\sum^{M}_{t}x_{n,t}y_{n,t}-M\widehat{y}\widehat{x}} {\left(\sum^{K}_{n}w_{n}\sum^{M}_{t}x^{2}_{n,t}-M\widehat{x}^{2}\right)}

(38)

Or, if we wish to return to the notation over the entire dataset, associating a weight $w_{i}$ with each observation $i$ , in a more general form:

b_{1}= \frac{\sum^{N}_{i}w_{i}x_{i}y_{i}-\frac{1}{M}\left(\sum^{N}_{i}w_{i}y_{i}\right)\left(\sum^{N}_{i}w_{i}x_{i}\right)} {\left(\sum^{N}_{i}w_{i}x^{2}_{i}-\frac{1}{M}\left(\sum^{N}_{i}w_{i}x_{i}\right)^{2}\right)}

(39)

However, we should remember that we obtain this result only under the assumption that all series have the same length and that the sum of the weights across all series for any given $t$ must be equal to 1. Comparing this expression with the previous $b_{1}$ , we replaced the total population size with the size of each series and inserted a weight into each summation.

# @title
M=len(nx) #Tamanho original de cada série
N=len(y2)

s1=0;s2=0;s3=0;s4=0;s5=0
for i in (range(N)):
    w = 0.25 if (i<M) else 0.75
    s1+=w*x2[i]*y2[i]
    s2+=w*y2[i]
    s3+=w*x2[i]
    s4+=w*x2[i]*x2[i]
    s5+=w*x2[i]

b1=(s1-s2*s3/M)/(s4-s5*s5/M)
print("Manual")
print("x1:    "+str(b1))

#Auto
df = pd.DataFrame({
    'id': [1]*I + [2]*I,
    'tempo': list(range(I)) + list(range(I)),
    'x': np.concatenate([x, x1]),
    'y': np.concatenate([y, y1])
})

df['peso'] = np.where(df['id'] == 1, 0.25, 0.75)

df = df.set_index(['id', 'tempo'])

modelo = PanelOLS.from_formula(
    'y ~ x + EntityEffects',
    data=df,
    weights=df['peso']
)

resultado = modelo.fit()

print("\nAuto")
print("x1:    "+str(resultado.params['x']))

print("\nOs coeficientes angulares são {:.2f}% iguais.".format(100*b1/resultado.params['x']))

Manual
x1:    0.2969533853308433

Auto
x1:    0.29695338533084326

Os coeficientes angulares são 100.00% iguais.

Correlation¶

Pearson¶

We should begin with Pearson correlation, and I believe the best way is through a geometric interpretation. If we have a set of points $X=\left(x_{1},x_{2},\dots,x_{n}\right)**negrito**$ and another set $Y=\left(y_{1},y_{2},\dots,y_{n}\right),$ then, if we subtract the mean values, we obtain:

\tilde{X}=\left(x_{1}-\overline{x},x_{2}-\overline{x},\dots,x_{n}-\overline{x}\right), \quad \tilde{Y}=\left(y_{1}-\overline{y},y_{2}-\overline{y},\dots,y_{n}-\overline{y}\right)

(40)

We may think of this as shifting the point distributions so that they are centered around the origin. We can then analyze how each distribution departs from the origin. If we have only three points, then $\tilde{X}$ and $\tilde{Y}$ denote vectors in three-dimensional space. The dot product of both is given by:

\tilde{Y}\cdot\tilde{X}=\left\Vert \tilde{X}\right\Vert \left\Vert \tilde{Y}\right\Vert \cos\left(\theta\right)

(41)

Then:

\cos\left(\theta\right) = \frac{\tilde{Y}\cdot\tilde{X}}{\left\Vert \tilde{X}\right\Vert \left\Vert \tilde{Y}\right\Vert } = \frac{\sum^{n}_{i=1}\left(x_{i}-\overline{x}\right)\left(y_{i}-\overline{y}\right)} {\sqrt{\sum^{n}_{i=1}\left(x_{i}-\overline{x}\right)^{2}} \sqrt{\sum^{n}_{i=1}\left(y_{i}-\overline{y}\right)^{2}}} =r

(42)

Therefore, Pearson correlation gives us the cosine of the angle between the vectors formed by the variables after removing their mean values. In other words, it measures the alignment between the variations of $X$ and $Y$ around their respective means, interpreted as vectors in an $n$ -dimensional space.

# @title
from scipy.stats import pearsonr


I = np.random.randint(0,100)
x = np.random.randint(0, 100, I)
y = np.random.randint(0, 100, I)

xn=x-np.mean(x)
yn=y-np.mean(y)
p1 = np.dot(xn, yn)
p2 = np.dot(xn, xn)
p3 = np.dot(yn, yn)
c=p1/(np.sqrt(p2)*np.sqrt(p3))
print("Manual")
print("Correlaçao:    "+str(c))

corr, _= pearsonr(x, y)

print("\nAuto")
print("Correlação:    "+str(corr))

print("\nOs coeficientes de correlação são {:.2f}% iguais.".format(100*c/corr))

Manual
Correlaçao:    0.012785408476462182

Auto
Correlação:    0.012785408476462127

Os coeficientes de correlação são 100.00% iguais.

Spearmn¶

It does not differ much from the previous case, except that instead of working with the original data, we transform them into ranks and only then apply the Pearson correlation.

In both this section and the previous one, we should keep in mind that we do not analyze standard errors or p-values. In general, this additional analysis is important for evaluating the statistical significance of the results, especially in correlation tests. However, in many cases linear regression may be used merely as a descriptive tool, that is, with the objective of finding the line that best represents the distribution of the points, independently of statistical inference regarding the coefficients.

# @title
from scipy.stats import rankdata
rx = rankdata(x)
ry = rankdata(y)

xn=rx-np.mean(rx)
yn=ry-np.mean(ry)
p1 = np.dot(xn, yn)
p2 = np.dot(xn, xn)
p3 = np.dot(yn, yn)
c=p1/(np.sqrt(p2)*np.sqrt(p3))
print("Manual")
print("Correlaçao:    "+str(c))

corr, _= spearmanr(x, y)

print("\nAuto")
print("Correlação:    "+str(corr))

print("\nOs coeficientes de correlação são {:.2f}% iguais.".format(100*c/corr))

Manual
Correlaçao:    0.07954084943381937

Auto
Correlação:    0.07954084943381935

Os coeficientes de correlação são 100.00% iguais.