Regularization (mathematics)

In mathematics, statistics, finance,^[1] and computer science, particularly in machine learning and inverse problems, regularization is a process that changes the result answer to be "simpler". It is often used to obtain results for ill-posed problems or to prevent overfitting.^[2]

Although regularization procedures can be divided in many ways, the following delineation is particularly helpful:

In explicit regularization, independent of the problem or model, there is always a data term, that corresponds to a likelihood of the measurement and a regularization term that corresponds to a prior. By combining both using Bayesian statistics, one can compute a posterior, that includes both information sources and therefore stabilizes the estimation process. By trading off both objectives, one chooses to be more addictive to the data or to enforce generalization (to prevent overfitting). There is a whole research branch dealing with all possible regularizations. In practice, one usually tries a specific regularization and then figures out the probability density that corresponds to that regularization to justify the choice. It can also be physically motivated by common sense or intuition.

In machine learning, the data term corresponds to the training data and the regularization is either the choice of the model or modifications to the algorithm. It is always intended to reduce the generalization error, i.e. the error score with the trained model on the evaluation set and not the training data.^[3]

One of the earliest uses of regularization is Tikhonov regularization (ridge regression), related to the method of least squares.

L1 regularization (also called ) leads to sparse models by adding a penalty based on the absolute value of coefficients.

LASSO

L2 regularization (also called ) encourages smaller, more evenly distributed weights by adding a penalty based on the square of the coefficients.^[4]

ridge regression

Bayesian interpretation of regularization

Bias–variance tradeoff

Matrix regularization

Regularization by spectral filtering

Regularized least squares

Lagrange multiplier

Neumaier, A. (1998). (PDF). SIAM Review. 40 (3): 636–666. Bibcode:1998SIAMR..40..636N. doi:10.1137/S0036144597321909.

"Solving ill-conditioned and singular linear systems: A tutorial on regularization"

$_$_$DEEZ_NUTS#0__titleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#0__subtitleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#0__call_to_action.textDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#8__titleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#8__descriptionDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#7__titleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#7__descriptionDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#1__titleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#1__descriptionDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#4__descriptionDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#5__titleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#5__descriptionDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#3__titleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#3__descriptionDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#3__heading--0DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#3__description--0DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#3__heading--1DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#3__description--1DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#3__heading--2DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#3__description--2DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#6__titleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#6__descriptionDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#6__heading--0DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#6__description--0DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#6__heading--1DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#6__description--1DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__titleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__subtextDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__quote--0DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__name--0DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__company_or_position--0DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__quote--1DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__name--1DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__company_or_position--1DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__quote--2DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__name--2DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__company_or_position--2DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__quote--3DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__name--3DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__company_or_position--3DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__quote--4DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__name--4DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__company_or_position--4DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__quote--5DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__name--5DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__company_or_position--5DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__quote--6DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__name--6DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#9__company_or_position--6DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__titleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__subtextDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__quote--0DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__name--0DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__company_or_position--0DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__quote--1DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__name--1DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__company_or_position--1DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__quote--2DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__name--2DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__company_or_position--2DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__quote--3DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__name--3DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__company_or_position--3DEEZ_NUTS$_$_$