Statistical model

A statistical model is a mathematical model that embodies a set of statistical assumptions concerning the generation of sample data (and similar data from a larger population). A statistical model represents, often in considerably idealized form, the data-generating process.^[1] When referring specifically to probabilities, the corresponding term is probabilistic model. All statistical hypothesis tests and all statistical estimators are derived via statistical models. More generally, statistical models are part of the foundation of statistical inference. A statistical model is usually specified as a mathematical relationship between one or more random variables and other non-random variables. As such, a statistical model is "a formal representation of a theory" (Herman Adèr quoting Kenneth Bollen).^[2]

Introduction[edit]

Informally, a statistical model can be thought of as a statistical assumption (or set of statistical assumptions) with a certain property: that the assumption allows us to calculate the probability of any event. As an example, consider a pair of ordinary six-sided dice. We will study two different statistical assumptions about the dice.

The first statistical assumption is this: for each of the dice, the probability of each face (1, 2, 3, 4, 5, and 6) coming up is 1/6. From that assumption, we can calculate the probability of both dice coming up 5: 1/6 × 1/6 = 1/36. More generally, we can calculate the probability of any event: e.g. (1 and 2) or (3 and 3) or (5 and 6). The alternative statistical assumption is this: for each of the dice, the probability of the face 5 coming up is 1/8 (because the dice are weighted). From that assumption, we can calculate the probability of both dice coming up 5: 1/8 × 1/8 = 1/64. We cannot, however, calculate the probability of any other nontrivial event, as the probabilities of the other faces are unknown.

The first statistical assumption constitutes a statistical model: because with the assumption alone, we can calculate the probability of any event. The alternative statistical assumption does not constitute a statistical model: because with the assumption alone, we cannot calculate the probability of every event. In the example above, with the first assumption, calculating the probability of an event is easy. With some other examples, though, the calculation can be difficult, or even impractical (e.g. it might require millions of years of computation). For an assumption to constitute a statistical model, such difficulty is acceptable: doing the calculation does not need to be practicable, just theoretically possible.

Formal definition[edit]

In mathematical terms, a statistical model is usually thought of as a pair ( $S,{\mathcal {P}}$ ), where $S$ is the set of possible observations, i.e. the sample space, and ${\mathcal {P}}$ is a set of probability distributions on $S$ .^[3] The intuition behind this definition is as follows. It is assumed that there is a "true" probability distribution induced by the process that generates the observed data. We choose ${\mathcal {P}}$ to represent a set (of distributions) which contains a distribution that adequately approximates the true distribution. Note that we do not require that ${\mathcal {P}}$ contains the true distribution, and in practice that is rarely the case. Indeed, as Burnham & Anderson state, "A model is a simplification or approximation of reality and hence will not reflect all of reality"^[4]—hence the saying "all models are wrong". The set ${\mathcal {P}}$ is almost always parameterized: ${\mathcal {P}}=\{F_{\theta }:\theta \in \Theta \}$ . The set of distributions $\Theta$ defines the parameters of the model. A parameterization is generally required to have distinct parameter values give rise to distinct distributions, i.e. $F_{\theta _{1}}=F_{\theta _{2}}\Rightarrow \theta _{1}=\theta _{2}$ must hold (in other words, it must be injective). A parameterization that meets the requirement is said to be identifiable.^[3]

An example[edit]

Suppose that we have a population of children, with the ages of the children distributed uniformly, in the population. The height of a child will be stochastically related to the age: e.g. when we know that a child is of age 7, this influences the chance of the child being 1.5 meters tall. We could formalize that relationship in a linear regression model, like this: height_i = b₀ + b₁age_i + ε_i, where b₀ is the intercept, b₁ is a parameter that age is multiplied by to obtain a prediction of height, ε_i is the error term, and i identifies the child. This implies that height is predicted by age, with some error.

An admissible model must be consistent with all the data points. Thus, a straight line (height_i = b₀ + b₁age_i) cannot be the equation for a model of the data—unless it exactly fits all the data points, i.e. all the data points lie perfectly on the line. The error term, ε_i, must be included in the equation, so that the model is consistent with all the data points. To do statistical inference, we would first need to assume some probability distributions for the ε_i. For instance, we might assume that the ε_i distributions are i.i.d. Gaussian, with zero mean. In this instance, the model would have 3 parameters: b₀, b₁, and the variance of the Gaussian distribution. We can formally specify the model in the form ( $S,{\mathcal {P}}$ ) as follows. The sample space, $S$ , of our model comprises the set of all possible pairs (age, height). Each possible value of $\theta$ = (b₀, b₁, σ²) determines a distribution on $S$ ; denote that distribution by $F_{\theta }$ . If $\Theta$ is the set of all possible values of $\theta$ , then ${\mathcal {P}}=\{F_{\theta }:\theta \in \Theta \}$ . (The parameterization is identifiable, and this is easy to check.)

In this example, the model is determined by (1) specifying $S$ and (2) making some assumptions relevant to ${\mathcal {P}}$ . There are two assumptions: that height can be approximated by a linear function of age; that errors in the approximation are distributed as i.i.d. Gaussian. The assumptions are sufficient to specify ${\mathcal {P}}$ —as they are required to do.

Predictions

Extraction of information

Description of stochastic structures

A statistical model is a special class of mathematical model. What distinguishes a statistical model from other mathematical models is that a statistical model is non-deterministic. Thus, in a statistical model specified via mathematical equations, some of the variables do not have specific values, but instead have probability distributions; i.e. some of the variables are stochastic. In the above example with children's heights, ε is a stochastic variable; without that stochastic variable, the model would be deterministic. Statistical models are often used even when the data-generating process being modeled is deterministic. For instance, coin tossing is, in principle, a deterministic process; yet it is commonly modeled as stochastic (via a Bernoulli process). Choosing an appropriate statistical model to represent a given data-generating process is sometimes extremely difficult, and may require knowledge of both the process and relevant statistical analyses. Relatedly, the statistician Sir David Cox has said, "How [the] translation from subject-matter problem to statistical model is done is often the most critical part of an analysis".^[5]

There are three purposes for a statistical model, according to Konishi & Kitagawa.^[6]

Those three purposes are essentially the same as the three purposes indicated by Friendly & Meyer: prediction, estimation, description.^[7] The three purposes correspond with the three kinds of logical reasoning: deductive reasoning, inductive reasoning, abductive reasoning.

Davison, A. C. (2008), Statistical Models,

Cambridge University Press

Drton, M.; Sullivant, S. (2007), (PDF), Statistica Sinica, 17: 1273–1297

"Algebraic statistical models"

Freedman, D. A. (2009), Statistical Models,

Cambridge University Press

Helland, I. S. (2010), Steps Towards a Unified Basis for Scientific Models and Methods,

World Scientific

; Chan, J. C. C. (2014), Statistical Modeling and Computation, Springer

Kroese, D. P.

Shmueli, G. (2010), "To explain or to predict?", , 25 (3): 289–310, arXiv:1101.0891, doi:10.1214/10-STS330, S2CID 15900983

Statistical model

Introduction[edit]

Formal definition[edit]

An example[edit]

Cambridge University Press

"Algebraic statistical models"

Cambridge University Press

World Scientific

Kroese, D. P.

Statistical Science