A Modern Introduction to Probability and Statistics

Chapter 17: Basic Statistical Models

Linear Regression

For a bivariate data set $(x_{1},y_{1}),(x_{2},y_{2}),\dots ,(x_{n},y_{n})$:

Assume that Failed to parse (Conversion error. Server ("https://en.wikipedia.org/api/rest_") reported: "Cannot get mml. Server problem."): {\displaystyle x_{1},x_{2},\dots x_{n}} are not random

$y_{1},y_{2},\dots ,y_{n}$ are realizations of random variables $Y_{1},Y_{2},\dots ,Y_{n}$ that satisfy

$Y_{i}=\alpha +\beta x_{i}+U_{i}$

for Failed to parse (Conversion error. Server ("https://en.wikipedia.org/api/rest_") reported: "Cannot get mml. Server problem."): {\displaystyle i=1\dots n}

where Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle U_i} are independent random variables with Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle E(U_i) = 0} (because random fluctuations, expected to be zero about the regression line) and Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle Var(U_i) = \sigma^2} (each point has same variance, because assuming each random fluctuation has same amount of variability)

Expectation of each Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle Y_i} is different:

Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle E[Y_i] = E[\alpha + \beta x_i + U_i] = \alpha + \beta x_i + E[U_i] = \alpha + \beta x_i }

Multiple Linear Regression

If we considered that the data were better matched by a function like

Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://en.wikipedia.org/api/rest_v1/":): {\displaystyle y = \alpha + \beta x + \gamma x^2 }

then it's no longer linear regression, it's multiple linear regression

Chapter 20

Mean Squared Error (MSE)

Discussing unbiased estimators

Comparison of two unbiased estimators:

1. Variance (spread): less spread means better estimator

2. The lower the spread, the lower the MSE, the better the estimator