In statistics, the Rao–Blackwell theorem, sometimes referred to as the Rao–Blackwell–Kolmogorov theorem, is a result that characterizes the transformation of an arbitrarily crude estimator into an estimator that is optimal by the mean-squared-error criterion or any of a variety of similar criteria.
The Rao–Blackwell theorem states that if g(X) is any kind of estimator of a parameter θ, then the conditional expectation of g(X) given T(X), where T is a sufficient statistic, is typically a better estimator of θ, and is never worse.
Sometimes one can very easily construct a very crude estimator g(X), and then evaluate that conditional expected value to get an estimator that is in various senses optimal.
The process of transforming an estimator using the Rao–Blackwell theorem can be referred to as Rao–Blackwellization.
[1][2][3] One case of Rao–Blackwell theorem states: In other words, The essential tools of the proof besides the definition above are the law of total expectation and the fact that for any random variable Y, E(Y2) cannot be less than [E(Y)]2.
That inequality is a case of Jensen's inequality, although it may also be shown to follow instantly from the frequently mentioned fact that More precisely, the mean square error of the Rao-Blackwell estimator has the following decomposition[4] Since
If the loss function is twice-differentiable, as in the case for mean-squared-error, then we have the sharper inequality[4] The improved estimator is unbiased if and only if the original estimator is unbiased, as may be seen at once by using the law of total expectation.
The theorem holds regardless of whether biased or unbiased estimators are used.
[5] Phone calls arrive at a switchboard according to a Poisson process at an average rate of λ per minute.
This rate is not observable, but the numbers X1, ..., Xn of phone calls that arrived during n successive one-minute periods are observed.
It is desired to estimate the probability e−λ that the next one-minute period passes with no phone calls.
An extremely crude estimator of the desired probability is i.e., it estimates this probability to be 1 if no phone calls arrived in the first minute and zero otherwise.
The sum can be readily shown to be a sufficient statistic for λ, i.e., the conditional distribution of the data X1, ..., Xn, depends on λ only through this sum.
Therefore, we find the Rao–Blackwell estimator After doing some algebra we have Since the average number of calls arriving during the first n minutes is nλ, one might not be surprised if this estimator has a fairly high probability (if n is big) of being close to So δ1 is clearly a very much improved estimator of that last quantity.
In fact, since Sn is complete and δ0 is unbiased, δ1 is the unique minimum variance unbiased estimator by the Lehmann–Scheffé theorem.
as an initial (crude) unbiased estimator for
), it may be improved using the Rao–Blackwell theorem as follows: However, the following unbiased estimator can be shown to have lower variance: And in fact, it could be even further improved when using the following estimator: The model is a scale model.
Optimal equivariant estimators can then be derived for loss functions that are invariant.