Fitting a Line by Least Squares

Intuition

Among all straight lines, pick the one that makes the squared vertical gaps to the points as small as possible. Squaring treats over- and under-predictions alike and gives a closed-form answer.

Precise explanation

Minimise L(a,b)=∑(yi−(a+bxi))2L(a, b) = \sum (y_i - (a + b x_i))^2. Setting both partial derivatives to zero gives

b=∑(xi−xˉ)(yi−yˉ)∑(xi−xˉ)2=cov⁡(x,y)var⁡(x),a=yˉ−bxˉ.b = \frac{\sum (x_i-\bar x)(y_i-\bar y)}{\sum (x_i-\bar x)^2} = \frac{\operatorname{cov}(x,y)}{\operatorname{var}(x)},\qquad a = \bar y - b\bar x.

The fitted line always passes through (xˉ,yˉ)(\bar x, \bar y). If all xx are equal the slope is undefined. In matrix form (many features) this is the normal equation w=(X⊤X)−1X⊤yw = (X^\top X)^{-1} X^\top y (NP-12 used lstsq; ML-20 adds regularisation).

Least-squares residuals

Figure (original): the blue line minimises the sum of squared red residuals.

Worked example

x = [1, 2, 3, 4], y = [2, 4, 5, 8] → b = 9.5/5 = 1.9, a = 4.75 − 1.9·2.5 = 0. Prediction at x = 5: 9.5.

Common mistakes

  • Minimising perpendicular distances (that is a different method).
  • Forgetting the intercept.
  • Extrapolating far beyond the training x-range.

Why this matters

Linear regression is the baseline model for every regression task and the building block of GLMs, neural network output layers and many explanations (XA-01).

Library implementation

np.polyfit(x, y, 1) and scikit-learn's LinearRegression().fit(x.reshape(-1, 1), y) fit the same line; LinearRegression.score is the same R2R^2. The NumPy comparison always runs; the scikit-learn one runs when it is installed.

References