A sufficient statistic contains all of the information that the dataset provides about the model parameters.
[1] The Kolmogorov structure function deals with individual finite data; the related notion there is the algorithmic sufficient statistic.
[2] Stephen Stigler noted in 1973 that the concept of sufficiency had fallen out of favor in descriptive statistics because of the strong dependence on an assumption of the distributional form (see Pitman–Koopman–Darmois theorem below), but remained very important in theoretical work.
of independent identically distributed data conditioned on an unknown parameter
For example, for a Gaussian distribution with unknown mean and variance, the jointly sufficient statistic, from which maximum likelihood estimates of both parameters can be estimated, consists of two functions, the sum of all data points and the sum of all squared data points (or equivalently, the sample mean and sample variance).
[5] In other words, the data processing inequality becomes an equality: As an example, the sample mean is sufficient for the mean (μ) of a normal distribution with known variance.
What we want to prove is that Y1 = u1(X1, X2, ..., Xn) is a sufficient statistic for θ if and only if, for some function H, First, suppose that We shall make the transformation yi = ui(x1, x2, ..., xn), for i = 1, ..., n, having inverse functions xi = wi(y1, y2, ..., yn), for i = 1, ..., n, and Jacobian
, we have With the first equality by the definition of pdf for multiple variables, the second by the remark above, the third by hypothesis, and the fourth because the summation is not over
A useful characterization of minimal sufficiency is that when the density fθ exists, S(X) is minimal sufficient if and only if[citation needed] This follows as a consequence from Fisher's factorization theorem stated above.
[12] However, under mild conditions, a minimal sufficient statistic does always exist.
If X1, ...., Xn are independent Bernoulli-distributed random variables with expected value p, then the sum T(X) = X1 + ... + Xn is a sufficient statistic for p (here 'success' corresponds to Xi = 1 and 'failure' to Xi = 0; so T is the total number of successes) This is seen by considering the joint probability distribution: Because the observations are independent, this can be written as and, collecting powers of p and 1 − p, gives which satisfies the factorization criterion, with h(x) = 1 being just a constant.
Note the crucial feature: the unknown parameter p interacts with the data x only via the statistic T(x) = Σ xi.
If X1, ...., Xn are independent and uniformly distributed on the interval [0,θ], then T(X) = max(X1, ..., Xn) is sufficient for θ — the sample maximum is a sufficient statistic for the population maximum.
Because the observations are independent, the pdf can be written as a product of individual densities where 1{...} is the indicator function.
Because the observations are independent, the pdf can be written as a product of individual densities, i.e.
The joint density of the sample takes the form required by the Fisher–Neyman factorization theorem, by letting Since
Because the observations are independent, the pdf can be written as a product of individual densities, i.e.
The joint density of the sample takes the form required by the Fisher–Neyman factorization theorem, by letting Since
are independent and exponentially distributed with expected value θ (an unknown real-valued positive parameter), then
Because the observations are independent, the pdf can be written as a product of individual densities, i.e.
The joint density of the sample takes the form required by the Fisher–Neyman factorization theorem, by letting Since
Because the observations are independent, the pdf can be written as a product of individual densities, i.e.
The joint density of the sample takes the form required by the Fisher–Neyman factorization theorem, by letting Since
Sufficiency finds a useful application in the Rao–Blackwell theorem, which states that if g(X) is any kind of estimator of θ, then typically the conditional expectation of g(X) given sufficient statistic T(X) is a better (in the sense of having lower variance) estimator of θ, and is never worse.
According to the Pitman–Koopman–Darmois theorem, among families of probability distributions whose domain does not vary with the parameter being estimated, only in exponential families is there a sufficient statistic whose dimension remains bounded as sample size increases.
Intuitively, this states that nonexponential families of distributions on the real line require nonparametric statistics to fully capture the information in the data.
[14] This theorem shows that the existence of a finite-dimensional, real-vector-valued sufficient statistics sharply restricts the possible forms of a family of distributions on the real line.
When the parameters or the random variables are no longer real-valued, the situation is more complex.
Thus the requirement is that, for almost every x, More generally, without assuming a parametric model, we can say that the statistics T is predictive sufficient if It turns out that this "Bayesian sufficiency" is a consequence of the formulation above,[16] however they are not directly equivalent in the infinite-dimensional case.
[18] A concept called "linear sufficiency" can be formulated in a Bayesian context,[19] and more generally.