Regression analysis

Regression analysis is primarily used for two conceptually distinct purposes. First, regression analysis is widely used for prediction and forecasting, where its use has substantial overlap with the field of machine learning. Second, in some situations regression analysis can be used to infer causal relationships between the independent and dependent variables. Importantly, regressions by themselves only reveal relationships between a dependent variable and a collection of independent variables in a fixed dataset. To use regressions for prediction or to infer causal relationships, respectively, a researcher must carefully justify why existing relationships have predictive power for a new context or why a relationship between two variables has a causal interpretation. The latter is especially important when researchers hope to estimate causal relationships using observational data.^[2][3]

In practice, researchers first select a model they would like to estimate and then use their chosen method (e.g., ordinary least squares) to estimate the parameters of that model. Regression models involve the following components:


In various fields of application, different terminologies are used in place of dependent and independent variables.


Most regression models propose that ${\displaystyle Y_{i}}$ is a function (regression function) of ${\displaystyle X_{i}}$ and ${\displaystyle \beta }$, with ${\displaystyle e_{i}}$ representing an additive error term that may stand in for un-modeled determinants of ${\displaystyle Y_{i}}$ or random statistical noise:


The researchers' goal is to estimate the function ${\displaystyle f(X_{i},\beta )}$ that most closely fits the data. To carry out regression analysis, the form of the function ${\displaystyle f}$ must be specified. Sometimes the form of this function is based on knowledge about the relationship between ${\displaystyle Y_{i}}$ and ${\displaystyle X_{i}}$ that does not rely on the data. If no such knowledge is available, a flexible or convenient form for ${\displaystyle f}$ is chosen. For example, a simple univariate regression may propose ${\displaystyle f(X_{i},\beta )=\beta _{0}+\beta _{1}X_{i}}$, suggesting that the researcher believes ${\displaystyle Y_{i}=\beta _{0}+\beta _{1}X_{i}+e_{i}}$ to be a reasonable approximation for the statistical process generating the data.


Once researchers determine their preferred statistical model, different forms of regression analysis provide tools to estimate the parameters ${\displaystyle \beta }$. For example, least squares (including its most common variant, ordinary least squares) finds the value of ${\displaystyle \beta }$ that minimizes the sum of squared errors ${\displaystyle \sum _{i}(Y_{i}-f(X_{i},\beta ))^{2}}$. A given regression method will ultimately provide an estimate of ${\displaystyle \beta }$, usually denoted ${\displaystyle {\hat {\beta }}}$ to distinguish the estimate from the true (unknown) parameter value that generated the data. Using this estimate, the researcher can then use the fitted value ${\displaystyle {\hat {Y_{i}}}=f(X_{i},{\hat {\beta }})}$ for prediction or to assess the accuracy of the model in explaining the data. Whether the researcher is intrinsically interested in the estimate ${\displaystyle {\hat {\beta }}}$ or the predicted value ${\displaystyle {\hat {Y_{i}}}}$ will depend on context and their goals. As described in ordinary least squares, least squares is widely used because the estimated function ${\displaystyle f(X_{i},{\hat {\beta }})}$ approximates the conditional expectation ${\displaystyle E(Y_{i}|X_{i})}$.^[5] However, alternative variants (e.g., least absolute deviations or quantile regression) are useful when researchers want to model other functions ${\displaystyle f(X_{i},\beta )}$.


It is important to note that there must be sufficient data to estimate a regression model. For example, suppose that a researcher has access to ${\displaystyle N}$ rows of data with one dependent and two independent variables: ${\displaystyle (Y_{i},X_{1i},X_{2i})}$. Suppose further that the researcher wants to estimate a bivariate linear model via least squares: ${\displaystyle Y_{i}=\beta _{0}+\beta _{1}X_{1i}+\beta _{2}X_{2i}+e_{i}}$. If the researcher only has access to ${\displaystyle N=2}$ data points, then they could find infinitely many combinations ${\displaystyle ({\hat {\beta }}_{0},{\hat {\beta }}_{1},{\hat {\beta }}_{2})}$ that explain the data equally well: any combination can be chosen that satisfies ${\displaystyle {\hat {Y}}_{i}={\hat {\beta }}_{0}+{\hat {\beta }}_{1}X_{1i}+{\hat {\beta }}_{2}X_{2i}}$, all of which lead to ${\displaystyle \sum _{i}{\hat {e}}_{i}^{2}=\sum _{i}({\hat {Y}}_{i}-({\hat {\beta }}_{0}+{\hat {\beta }}_{1}X_{1i}+{\hat {\beta }}_{2}X_{2i}))^{2}=0}$ and are therefore valid solutions that minimize the sum of squared residuals. To understand why there are infinitely many options, note that the system of ${\displaystyle N=2}$ equations is to be solved for 3 unknowns, which makes the system underdetermined. Alternatively, one can visualize infinitely many 3-dimensional planes that go through ${\displaystyle N=2}$ fixed points.


More generally, to estimate a least squares model with ${\displaystyle k}$ distinct parameters, one must have ${\displaystyle N\geq k}$ distinct data points. If ${\displaystyle N>k}$, then there does not generally exist a set of parameters that will perfectly fit the data. The quantity ${\displaystyle N-k}$ appears often in regression analysis, and is referred to as the degrees of freedom in the model. Moreover, to estimate a least squares model, the independent variables ${\displaystyle (X_{1i},X_{2i},...,X_{ki})}$ must be linearly independent: one must not be able to reconstruct any of the independent variables by adding and multiplying the remaining independent variables. As discussed in ordinary least squares, this condition ensures that ${\displaystyle X^{T}X}$ is an invertible matrix and therefore that a unique solution ${\displaystyle {\hat {\beta }}}$ exists.