Polynomial regression

In statistics, polynomial regression is a form of regression analysis in which the relationship between the independent variable x and the dependent variable y is modeled as an nth degree polynomial in x. Polynomial regression fits a nonlinear relationship between the value of x and the corresponding conditional mean of y, denoted E(y |x).

Although polynomial regression fits a nonlinear model to the data, as a statistical estimation problem it is linear, in the sense that the regression function E(y | x) is linear in the unknown parameters that are estimated from the data.

[2] Polynomial regression models are usually fit using the method of least squares.

The least-squares method minimizes the variance of the unbiased estimators of the coefficients, under the conditions of the Gauss–Markov theorem.

The least-squares method was published in 1805 by Legendre and in 1809 by Gauss.

The first design of an experiment for polynomial regression appeared in an 1815 paper of Gergonne.

[3][4] In the twentieth century, polynomial regression played an important role in the development of regression analysis, with a greater emphasis on issues of design and inference.

[citation needed] The goal of regression analysis is to model the expected value of a dependent variable y in terms of the value of an independent variable (or vector of independent variables) x.

In simple linear regression, the model is used, where ε is an unobserved random error with mean zero conditioned on a scalar variable x.

For example, if we are modeling the yield of a chemical synthesis in terms of the temperature at which the synthesis takes place, we may find that the yield improves by increasing amounts for each unit increase in temperature.

In this case, we might propose a quadratic model of the form In this model, when the temperature is increased from x to x + 1 units, the expected yield changes by

For infinitesimal changes in x, the effect on y is given by the total derivative with respect to x:

The fact that the change in yield depends on x is what makes the relationship between x and y nonlinear even though the model is linear in the parameters to be estimated.

In general, we can model the expected value of y as an nth degree polynomial, yielding the general polynomial regression model Conveniently, these models are all linear from the point of view of estimation, since the regression function is linear in terms of the unknown parameters β0, β1, ....

Therefore, for least squares analysis, the computational and inferential problems of polynomial regression can be completely addressed using the techniques of multiple regression.

This is done by treating x, x2, ... as being distinct independent variables in a multiple regression model.

Then the model can be written as a system of linear equations: which when using pure matrix notation is written as The vector of estimated polynomial regression coefficients (using ordinary least squares estimation) is assuming m < n which is required for the matrix to be invertible; then since

is a Vandermonde matrix, the invertibility condition is guaranteed to hold if all the

The above matrix equations explain the behavior of polynomial regression well.

However, to physically implement polynomial regression for a set of xy point pairs, more detail is useful.

The below matrix equations for polynomial coefficients are expanded from regression theory without derivation and easily implemented.

estimated y variable based on the polynomial regression calculations.

It is often difficult to interpret the individual coefficients in a polynomial regression fit, since the underlying monomials can be highly correlated.

Although the correlation can be reduced by using orthogonal polynomials, it is generally more informative to consider the fitted regression function as a whole.

Point-wise or simultaneous confidence bands can then be used to provide a sense of the uncertainty in the estimate of the regression function.

[9] In modern statistics, polynomial basis-functions are used along with new basis functions, such as splines, radial basis functions, and wavelets.

These families of basis functions offer a more parsimonious fit for many types of data.

Some of these methods make use of a localized form of classical polynomial regression.

[10] An advantage of traditional polynomial regression is that the inferential framework of multiple regression can be used (this also holds when using other families of basis functions such as splines).

If residuals have unequal variance, a weighted least squares estimator may be used to account for that.