Polynomial Regression

Difference with Linear Regression

Although the equation is polynomial in x, the regression remains linear with respect to the coefficients a₀, a₁, …, a_n. By incorporating higher-degree terms, such as quadratic or cubic components, the model can show non-linear patterns in the data.

Polynomial regression is useful when the data follows a curved pattern, where simple linear regression fails to capture the trend accurately. It predicts a curve that matches the underlying data.

Let us fit a 2^nd-degree polynomial regression model y = a₀ + a₁x + a₂x² to the following dataset:

x	y
1	2
2	4
3	9
4	16

Step 1. Setting Up the Design Matrix (X) 

X = ${\begin{bmatrix} 1 & 1 & 1^{2} \\ 1 & 2 & 2^{2} \\ 1 & 3 & 3^{2} \\ 1 & 4 & 4^{2} \end{bmatrix}}$ = ${\begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 4 \\ 1 & 3 & 9 \\ 1 & 4 & 16 \end{bmatrix}}$

Step 2. Setting Up the Dependent Variable Vector (y) 

y = ${\begin{bmatrix} 2 \\ 4 \\ 9 \\ 16 \end{bmatrix}}$

Step 3. Solving for Coefficients (a) 

a = (X^TX)^-1X^Ty 

Here, 

X^T = ${\begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & 2 & 3 & 4 \\ 1 & 4 & 9 & 16 \end{bmatrix}}$

On multiplying X^T and X, we get

X^TX = ${\begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & 2 & 3 & 4 \\ 1 & 4 & 9 & 16 \end{bmatrix}\begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 4 \\ 1 & 3 & 9 \\ 1 & 4 & 16 \end{bmatrix}}$

= ${\begin{bmatrix} 4 & 10 & 30 \\ 10 & 30 & 100 \\ 30 & 100 & 354 \end{bmatrix}}$ …..(i)

Now, calculating X^Ty, we get

X^Ty = ${\begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & 2 & 3 & 4 \\ 1 & 4 & 9 & 16 \end{bmatrix}\begin{bmatrix} 2 \\ 4 \\ 9 \\ 16 \end{bmatrix}}$ = ${\begin{bmatrix} 31 \\ 101 \\ 355 \end{bmatrix}}$ …..(ii)

As we know, the inverse of a matrix A, denoted by A^-1, is defined as

${A^{-1}=\dfrac{1}{\det \left( A\right) }\cdot Adj\left( A\right)}$ …..(iii)

Here, 

• det(A) = determinant of A and
• Adj(A) = adjugate or adjoint matrix (the transpose of the cofactor matrix)

For A = ${\begin{bmatrix} a & b & c \\ d & e & f \\ g & h & i \end{bmatrix}}$ …..(iv)

det(A) = a(ei – fh) – b(di – fg) + c(dh – eg) …..(v)

Now, comparing matrices (i) and (iv), we have

a = 4, b = 10, c = 30, d = 10, e = 30, f = 100, g = 30, h = 100, and i = 354

Substituting these values in (v),

det(A) = 4[(30)(354) – (100)(100)] – 10[(10)(354) – (100)(30)] + 30[(10)(100) – (30)(30)]

⇒ det(A) = 4[10620 – 10000] – 10[3540 – 3000] + 30[1000 – 900]

 ⇒ det(A) = 4(620) – 10(540) + 30(100)

⇒ det(A) = 2480 – 5400 + 3000

⇒ det(A) = 80 …..(vi)

To find the cofactor matrix Adj(A), we calculate the determinant of each 2 × 2 matrix by removing the row and column of each element.

Calculating the first row of cofactors, we get

1. Cofactor of a₁₁ (4):

Minor = ${\begin{bmatrix} 30 & 100 \\ 100 & 354 \end{bmatrix}}$

Determinant = (30)(354) – (100)(100) = 10620 – 10000 = 620

Thus, Cofactor = +620

2. Cofactor of a₁₂ (10):

Minor = ${\begin{bmatrix} 10 & 100 \\ 30 & 354 \end{bmatrix}}$ 

Determinant = (10)(354) – (100)(30) = 3540 – 3000 = 540

Thus, Cofactor = -540

Note: The negative sign appears due to the alternating sign rule (-1)^{i + j}, which ensures the correct orientation of the cofactor matrix.

3. Cofactor of a₁₃ (30):

Minor = ${\begin{bmatrix} 10 & 30 \\ 30 & 100 \end{bmatrix}}$

Determinant = (10)(100) – (30)(30) = 1000 – 900 = 100

Thus, Cofactor = +100

Calculating the second row cofactors, we get

1. Cofactor of a₂₁ (10):

Minor = ${\begin{bmatrix} 10 & 30 \\ 100 & 354 \end{bmatrix}}$

Determinant = (10)(354) – (30)(100) = 3540 – 3000 = 540

Thus, the cofactor = -540

2. Cofactor of a₂₂ (30):

Minor = ${\begin{bmatrix} 4 & 30 \\ 30 & 354 \end{bmatrix}}$

Determinant = (4)(354) – (30)(30) = 1416 – 900 = 516

Thus, the cofactor = +516

3. Cofactor of a₂₃ (100):

Minor = ${\begin{bmatrix} 4 & 10 \\ 30 & 100 \end{bmatrix}}$

Determinant = (4)(100) – (10)(30) = 400 – 300 = 100

Thus, the cofactor = -100

Calculating the third row cofactors, we get

1. Cofactor of a₃₁ (30):

Minor = ${\begin{bmatrix} 10 & 30 \\ 30 & 100 \end{bmatrix}}$

Determinant = (10)(100) – (30)(30) = 1000 – 900 = 100

Thus, the cofactor = +100

2. Cofactor of a₃₂ (100):

Minor = ${\begin{bmatrix} 4 & 30 \\ 10 & 100 \end{bmatrix}}$

Determinant = (4)(100) – (30)(10) = 400 – 300 = 100

Thus, the cofactor = -100

3. Cofactor of a₃₃ (354):

Minor = ${\begin{bmatrix} 4 & 10 \\ 10 & 30 \end{bmatrix}}$

Determinant = (4)(30) – (10)(10) = 120 – 100 = 20

Thus, the cofactor = +20

Hence, the cofactor matrix = ${\begin{bmatrix} 620 & -540 & 100 \\ -540 & 516 & -100 \\ 100 & -100 & 20 \end{bmatrix}}$

Adj(A) = transpose of the cofactor matrix = ${\begin{bmatrix} 620 & -540 & 100 \\ -540 & 516 & -100 \\ 100 & -100 & 20 \end{bmatrix}^{T}}$

⇒ Adj(A) = ${\begin{bmatrix} 620 & -540 & 100 \\ -540 & 516 & -100 \\ 100 & -100 & 20 \end{bmatrix}}$ …..(vii)

Now, by substituting (vi) and (vii) in (iii), we get

A^-1 = (X^TX)^-1 = ${\dfrac{1}{80}\begin{bmatrix} 620 & -540 & 100 \\ -540 & 516 & -100 \\ 100 & -100 & 20 \end{bmatrix}}$

⇒ (X^TX)^-1 = ${\begin{bmatrix} 7\cdot 75 & -6\cdot 75 & 1\cdot 25 \\ -6\cdot 75 & 6\cdot 45 & -1\cdot 25 \\ 1\cdot 25 & -1\cdot 25 & 0\cdot 25 \end{bmatrix}}$ …..(viii)

Now, using (ii) and (viii), we get

a = ${\begin{bmatrix} 7\cdot 75 & -6\cdot 75 & 1\cdot 25 \\ -6\cdot 75 & 6\cdot 45 & -1\cdot 25 \\ 1\cdot 25 & -1\cdot 25 & 0\cdot 25 \end{bmatrix}\begin{bmatrix} 31 \\ 101 \\ 355 \end{bmatrix}}$

⇒ a = ${\begin{bmatrix} 2\cdot 25 \\ -1\cdot 55 \\ 1\cdot 25 \end{bmatrix}}$

The coefficients are a₀ = 2.25, a₁ = -1.55, and a₂ = 1.25

Thus, the polynomial regression equation is:

y = 2.25 – 1.55x + 1.25x² 

Alternatively, polynomial regression can be solved computationally using Python or other tools instead of lengthy manual derivations.