Trend - Properties - Cheatsheet

Trend: Introduction

Trend: Enhanced

Trend: Language

Local Group

Preface

Goal: Getting to know statistical properties, equation by cheatsheet step by step.

Basic math is required, when the case you face become complex. When you have to test your calculation manually, before using buil-in formula or python library. We need to know how the manual calculation works.

Even with calculator online, How do I suppose to understand how the manual calculation works, How can we be sure that the result is correct? if it doesn’t comes with the math? Of course there are books for this. But even when I know the concept and the math, how do I automate my job with scripting?

Here I provide the equation cheatsheet, as a fundamental knowledge before processiong further. If you think my explanation, or calculation is wrong, I welcome any better opinion.

Cheatsheet

For beginner, I provide equation cheatsheet for linear regression for samples.

Trend: TeX: Cheatsheet: Equation Flow

You can obtain both the SVG source and Tex draft here:

Note that population and samples has different notation.

Equation for Linear Regression

Least Square Method

We have introduced basic properties in the previous least square article. Beyond that there are other statistic properties required, for regression analysis and correlation.

Trend: TeX: Cheatsheet: Population and Sample

You can get the source of the cheatsheet figure above here:

github.com/…/math/trend/50-cheatsheet.tex

Data Series

With sample of data points

$\begin{array}{|c|c|} \hline x_i & \textbf{observed x} \\ \hline y_i & \textbf{observed y} \\ \hline \hat{y}_i & \textbf{predicted y} \\ \hline \end{array}$

Total

The count and sum can be represented as:

$\begin{align*} \text{count} = \sum\limits_{i=1}^{n} 1 && \text{sum}_{x} = \sum\limits_{i=1}^{n} x_i && \text{sum}_{y} = \sum\limits_{i=1}^{n} y_i && \end{align*}$

Mean

Just the average

$\begin{align} \begin{aligned} &\text{Mean (Average):} \\ &\begin{array}{|c|c|} \hline \mathrm{\textbf{Population}} & \mathrm{\textbf{Sample}} \\ \hline \mu_X = \frac{1}{N}\sum\limits_{i=1}^{N} x_i & \bar{x} = \frac{1}{n}\sum\limits_{i=1}^{n} x_i \\ \hline \end{array} \end{aligned} \end{align}$

Variance

The spreads of each data series.

$\begin{align} \begin{aligned} &\text{Variance:} \\ &\begin{array}{|c|c|} \hline \mathrm{\textbf{Population}} & \mathrm{\textbf{Sample}} \\ \hline \sigma_X^2 = \frac{1}{N}\sum\limits_{i=1}^{N} (x_i - \mu)^2 & s_x^2 = \frac{1}{n-1}\sum\limits_{i=1}^{n} (x_i - \bar{x})^2 \\ \hline \end{array} \end{aligned} \end{align}$

Covariance

Just like variance, but both axis.

$\begin{align} \begin{aligned} &\text{Covariance:} \\ &\begin{array}{|c|} \hline \mathrm{\textbf{Population}} \\ \hline \text{Cov}(X,Y) = \frac{1}{N}\sum\limits_{i=1}^{N} (x_i - \mu_X)(y_i - \mu_Y) \\ \hline \mathrm{\textbf{Sample}} \\ \hline \text{Cov}(x,y) = \frac{1}{n-1}\sum\limits_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y}) \\ \hline \end{array} \end{aligned} \end{align}$

Standard Deviation

Square root of variance.

$\begin{align} \begin{aligned} &\text{Standard Deviation:} \\ &\begin{array}{|c|c|} \hline \mathrm{\textbf{Population}} & \mathrm{\textbf{Sample}} \\ \hline \sigma_X = \sqrt{\sigma_X^2} & s_x = \sqrt{s_x^2} \\ \hline \end{array} \end{aligned} \end{align}$

Slope and Intercept

We can calculate the slope (m) using this equation:

$m = \frac{\text{Cov}(x,y)}{s_x^2}$

You can also represent the equation for the slope m as

$m = \frac{\sum\limits_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sum\limits_{i=1}^{n} (x_i - \bar{x})^2}$

This way you can get the intercept (b) as follows:

$b = \bar{y} - m \bar{x}$

Correlation Coefficient

Pearson

Similar to slope, but using both sₓ and Sy.

$\begin{align} \begin{aligned} &\text{Correlation Coefficient:} \\ &\begin{array}{|c|c|} \hline \mathrm{\textbf{Population}} & \mathrm{\textbf{Sample}} \\ \hline \rho_{X,Y} = \frac{\text{Cov}(X,Y)}{\sigma_X \sigma_Y} & r_{x,y} = \frac{\text{Cov}(x,y)}{s_x s_y} \\ \hline \end{array} \end{aligned} \end{align}$

The Pearson correlation coefficient can also be expressed as follows:

$r = \frac{\sum (x_i - \bar{x}) \cdot (y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \cdot \sum (y_i - \bar{y})^2}}$

Coefficient of Determination

The R Square (R²)

In general, the R Square (R²) can be defined as follows.

$R^2 = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2}$

It is interesting that this is just comparing of (dispersed yᵢ observed againts ŷᵢ predicted) and, (dispersed yᵢ observed againts mean ȳ).

$R^2 = 1 - \frac{SSE}{TSS}$

The abbreviation are as below:

SSE: sum of squared errors, represents the sum of squared differences between the observed values of the dependent variable (yᵢ) and the predicted values (ŷᵢ)
TSS: total sum of squares, which measures the total variability of the dependent variable around its mean (ȳ).

R² for simple linear regression

The R Square (R²) for simple linear regression is the same as r². This can be expressed as follows.

$\begin{align*} r_{x,y} &= \frac{\text{Cov}(x,y)}{s_x . s_y} \\ R^2 &= r_{x,y}^2 \end{align*}$

The equation above is not true in general. In cases where the relationship between the variables is non-linear, the coefficient of determination (R²) may still be calculated using the last one.

$R^2 = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2}$

Residual

SSE can be defined as

$\begin{align*} SSE &= \sum\limits_{i=1}^{n} (e_i)^2 \\ &= \sum (y_i - \hat{y}_i)^2 \\ \end{align*}$

While the TSS measure against mean. The SSE measure against error. This similarity help me remember the concept.

MSE

MSE can be defined as

$MSE = \frac{SSE}{df}$

While variance divided by total degrees of freedom (n-1). The MSE divided by residual degrees of freedom (n-k-1). This similarity help me remember the concept.

Standard Error of Slope

This is also similar to Standard Deviation, except that we need to divide (error in y axis) by (variation in x) to get the slope.

$SE(\beta_1) = \sqrt{\frac{\sum (y_i - \hat{y}_i)^2}{(n - 2) \cdot \sum (x_i - \bar{x})^2}}$

This can be written as:

$SE(\beta_1) = \sqrt{\frac{MSE}{TSS_x}}$

This similarity help me remember the concept.

t-value

The equation for calculating the t-value, in the context of linear regression analysis is as follows. Where β̅₁ is the m slope.

$t = \frac{\hat{\beta}_1}{SE(\hat{\beta}_1)}$

However, for non-linear regression models, the concept of a t-value is not directly applicable in the same way as it is in linear regression. The interpretation and calculation of significance for parameters in non-linear models can differ significantly based on the estimation method used and the specific context of the model.

Example

For example with this (x,y) sample series. we can have all the calculated statistics as follows.

Regression provides an equation to predict one variable from another.

Regression analysis involves predicting the value of one variable based on the value of another variable. In this case, we are predicting the values of y based on the values of x.

n = 13

∑x (total) = 78
∑y (total) = 2327

x̄ (mean) = 6
ȳ (mean) =  179

∑(xᵢ-x̄) = 0
∑(yᵢ-ȳ) = 0

∑(xᵢ-x̄)² = 182
∑(xᵢ-x̄)(yᵢ-ȳ) = 7280

m (slope) = 40
b (intercept) = -61

With this we get the formula equation as:

y = -61 + 40*x

Correlation measures the strength and direction of the linear relationship between two variables.

Correlation measures the strength and direction of the linear relationship between two variables. It does not imply causation, only association.

The correlation coefficient (r) ranges from -1 to 1. A value closer to 1 indicates a strong positive correlation, while a value closer to -1 indicates a strong negative correlation. A value of 0 indicates no linear correlation.

∑(yᵢ-ȳ)²      = 309218
∑(xᵢ-x̄)(yᵢ-ȳ) = 7280

sₓ² (variance) = 15,17
sy² (variance) = 25.768,17
covariance     = 606,67

sₓ (std dev)   = 3,89
sy (std dev)   = 160,52
r (pearson)    = 0,9704

R² = 0,9417

In this case, the correlation coefficient (r) is 0.97, indicating a very strong positive correlation between x and y.

The standard error of the slope (SE(β₁)), T-value, and p-value are related to the significance of the slope coefficient in a regression model. A low p-value indicates that the slope coefficient is significant.

SSR = ∑ϵ² = 18.018
MSE = ∑ϵ²/(n-2) = 1.638
SE(β₁)  = √(MSE/sₓ) = 3,00
t-value = β̅₁/SE(β₁) = 13,33
p-value = 0,0000000391

We can calculate above example using worksheet, or scripting language such as python.

What Lies Ahead 🤔?

It is fun, right?

We can describe mathematical equation, in practical way.

It is nice to know the manual calculation, step by step using this fundamental equation. But this might not be suitable for daily basis, we need to explore different calculation method. That above is a complex way to get the result. Of course there is a simple way get the result, such as built-in worksheet formula, or even the oneliner method from python library.

Consider diving into the next step by exploring [ Trend - Properties - Formula ].