Kernel method

In machine learning, kernel machines are a class of algorithms for pattern analysis, whose best known member is the support-vector machine (SVM). These methods involve using linear classifiers to solve nonlinear problems.^[1] The general task of pattern analysis is to find and study general types of relations (for example clusters, rankings, principal components, correlations, classifications) in datasets. For many algorithms that solve these tasks, the data in raw representation have to be explicitly transformed into feature vector representations via a user-specified feature map: in contrast, kernel methods require only a user-specified kernel, i.e., a similarity function over all pairs of data points computed using inner products. The feature map in kernel machines is infinite dimensional but only requires a finite dimensional matrix from user-input according to the Representer theorem. Kernel machines are slow to compute for datasets larger than a couple of thousand examples without parallel processing.

Kernel methods owe their name to the use of kernel functions, which enable them to operate in a high-dimensional, implicit feature space without ever computing the coordinates of the data in that space, but rather by simply computing the inner products between the images of all pairs of data in the feature space. This operation is often computationally cheaper than the explicit computation of the coordinates. This approach is called the "kernel trick".^[2] Kernel functions have been introduced for sequence data, graphs, text, images, as well as vectors.

Algorithms capable of operating with kernels include the kernel perceptron, support-vector machines (SVM), Gaussian processes, principal components analysis (PCA), canonical correlation analysis, ridge regression, spectral clustering, linear adaptive filters and many others.

Most kernel algorithms are based on convex optimization or eigenproblems and are statistically well-founded. Typically, their statistical properties are analyzed using statistical learning theory (for example, using Rademacher complexity).

$_$_$DEEZ_NUTS#0__titleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#0__subtitleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#0__call_to_action.textDEEZ_NUTS$_$_$

${\hat {y}}\in \{-1,+1\}$ is the kernelized binary classifier's predicted label for the unlabeled input $\mathbf {x'}$ whose hidden true label $y$ is of interest;

$k\colon {\mathcal {X}}\times {\mathcal {X}}\to \mathbb {R}$ is the kernel function that measures similarity between any pair of inputs $\mathbf {x} ,\mathbf {x'} \in {\mathcal {X}}$ ;

the sum ranges over the $n$ labeled examples $\{(\mathbf {x} _{i},y_{i})\}_{i=1}^{n}$ in the classifier's training set, with $y_{i}\in \{-1,+1\}$ ;

the $w_{i}\in \mathbb {R}$ are the weights for the training examples, as determined by the learning algorithm;

the $\operatorname {sgn}$ determines whether the predicted classification ${\hat {y}}$ comes out positive or negative.

sign function

Kernel methods can be thought of as instance-based learners: rather than learning some fixed set of parameters corresponding to the features of their inputs, they instead "remember" the $i$ -th training example $(\mathbf {x} _{i},y_{i})$ and learn for it a corresponding weight $w_{i}$ . Prediction for unlabeled inputs, i.e., those not in the training set, is treated by the application of a similarity function $k$ , called a kernel, between the unlabeled input $\mathbf {x'}$ and each of the training inputs $\mathbf {x} _{i}$ . For instance, a kernelized binary classifier typically computes a weighted sum of similarities

Kernel classifiers were described as early as the 1960s, with the invention of the kernel perceptron.^[3] They rose to great prominence with the popularity of the support-vector machine (SVM) in the 1990s, when the SVM was found to be competitive with neural networks on tasks such as handwriting recognition.

$_$_$DEEZ_NUTS#3__titleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#3__descriptionDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#1__titleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#1__descriptionDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__titleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#2__descriptionDEEZ_NUTS$_$_$

Applications[edit]

Application areas of kernel methods are diverse and include geostatistics,^[8] kriging, inverse distance weighting, 3D reconstruction, bioinformatics, cheminformatics, information extraction and handwriting recognition.

Fisher kernel

Graph kernels

Kernel smoother

Polynomial kernel

(RBF)

Radial basis function kernel

String kernels

Neural tangent kernel

(NNGP) kernel

Neural network Gaussian process

Kernel methods for vector output

Kernel density estimation

Representer theorem

Similarity learning

Cover's theorem

; Cristianini, N. (2004). Kernel Methods for Pattern Analysis. Cambridge University Press.

Shawe-Taylor, J.

Liu, W.; Principe, J.; Haykin, S. (2010). . Wiley. ISBN 9781118211212.

Kernel Adaptive Filtering: A Comprehensive Introduction

; Smola, A. J.; Bach, F. (2018). Learning with Kernels : Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press. ISBN 978-0-262-53657-8.

Schölkopf, B.

—community website

Kernel-Machines Org

onlineprediction.net Kernel Methods Article

$_$_$DEEZ_NUTS#4__titleDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#4__subtextDEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#4__quote--0DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#4__name--0DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#4__company_or_position--0DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#4__quote--1DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#4__name--1DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#4__company_or_position--1DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#4__quote--2DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#4__name--2DEEZ_NUTS$_$_$

$_$_$DEEZ_NUTS#4__company_or_position--2DEEZ_NUTS$_$_$