SUFFICIENCY_(STATISTICS)

(Redirected from Sufficient statistic)
In statistics, a statistic is 'sufficient' for the parameter ''θ'', which indexes the distribution family of the data, precisely when the data's conditional probability distribution, given the statistic's value, no longer depends on ''θ''.
Intuitively, a sufficient statistic for ''θ'' captures all the possible information about ''θ'' that is in a particular sample. Both the statistic and ''θ'' can be vectors.
The concept is due to Sir Ronald Fisher in 1920, and later naming the concept ''sufficiency'' in 1922. Studies in the History of Probability and Statistics. XXXII: Laplace, Fisher and the Discovery of the Concept of Sufficiency, , Stephen, Stigler, Biometrika, 1973 Stigler notes that the concept has fallen out of favor in descriptive statistics because of the strong dependence on a assumption of the distributional form, but remains very important in theoretical work.

Contents

Mathematical definition

Fisher-Neyman's factorization theorem

Interpretation

Proof for the continuous case

Minimal sufficiency

Examples

Bernoulli distribution

Uniform distribution

Poisson distribution

Rao-Blackwell theorem

See also

References

External Links

Mathematical definition

A statistic ''T''(''X'') is 'sufficient for ''θ''' precisely if the conditional probability distribution of the data ''X'', given the statistic ''T''(''X''), is independent of the parameter ''θ'', i.e.
:$Pr(X=x|T(X)=t,\; heta)\; =\; Pr(X=x|T(X)=t),\; ,$
or in shorthand
:$Pr(x|t,\; heta)\; =\; Pr(x|t).,$
As an example, the mean of a sample is sufficient to find the parameter $mu$ in a Normal distribution, so that the probability that $mu$ takes on some value conditional on the sample mean and other information from the sample is the same as the probability that $mu$ takes on the same value conditional only on the sample mean.

Fisher-Neyman's factorization theorem

''Fisher's factorization theorem'' or ''factorization criterion'' provides a convenient 'characterization' of a sufficient statistic. If the likelihood function of ''X'' is ''L''_''θ''(''x''), then ''T'' is sufficient for ''θ'' if and only if functions ''g'' and ''h'' can be found such that
:$L\_\; heta(x)=h(x)\; ,\; g\_\; heta(T(x)),\; ,!$
i.e. the likelihood ''L'' can be factored into a product such that one factor, ''h'', does not depend on ''θ'' and the other factor, which does depend on ''θ'', depends on ''x'' only through ''T''(''x'').

Interpretation

A way to think about this is to consider varying ''x'' in such a way as to maintain a constant value of ''T''(''X'') and ask whether such a variation has any effect on inferences one might make about ''θ''. If the factorization criterion above holds, the answer is "none" because the dependence of the likelihood function ''f'' on ''θ'' is unchanged.

Proof for the continuous case

Due to Hogg and Craig (ISBN 978-0023557224). Let ''X''₁, ''X''₂, ..., ''X''_n, denote a random sample from a distribution having the pdf ''f''(''x'',θ) for γ < θ < δ. Let ''Y'' = ''u''(''x''₁, ''x''₂, ..., ''x''_n) be a statistic whose pdf is ''g''(''y'';θ). Then ''Y'' = ''u''(''X''₁, ''X''₂, ..., ''X''_n) is a sufficient statistics if and only if, for some function ''H'',
:$prod\_\{i=1\}^\{n\}\; f(x\_i;\; heta)\; =\; g\; left[u(X\_1,\; X\_2,\; dots,\; X\_n);\; heta$
ight] H(x_1, x_2, dots, x_n). ,!
First, suppose that:
:$prod\_\{i=1\}^\{n\}\; f(x\_i;\; heta)\; =\; g\; left[u(X\_1,\; X\_2,\; dots,\; X\_n);\; heta$
ight] H(x_1, x_2, dots, x_n). ,!
We shall make the transformation ''y''_i =''u''_i(''X''₁, ''X''₂, ..., ''X''_n), for ''i'' = 1, ..., ''n'', having inverse functions ''x''_i = ''w''_i(''y''₁, ''y''₂, ..., ''y''_n), for ''i'' = 1, ..., ''n'', and Jacobian ''J''. Thus,
:$$
prod_{i=1}^{n} f left[ w_i(y_1, y_2, dots, y_n); heta
ight] =
|J| g(y; heta) H left[ w_1(y_1, y_2, dots, y_n), dots, w_n(y_1, y_2, dots, y_n)
ight]

The left-hand member is the joint pdf ''g''(''y''₁, ''y''₂, ..., ''y''_n; θ) of ''Y''₁=''u''₁(''X''₁, ..., ''X''_n), ..., ''Y''_n=''u''_n(''X''₁, ..., ''X''_n). In the right-hand member, $g(y\_1,dots,y\_n;\; heta)$ is the pdf of $Y\_1$, so that $H[\; w\_1,\; dots\; ,\; w\_n]\; |J|$ is the quotient of $g(y\_1,dots,y\_n;\; heta)$ and $g\_1(y\_1;\; heta)$; that is, it is the conditional pdf $h(y\_2,\; dots,\; y\_n\; |\; y\_1;\; heta)$ of $Y\_2,dots,Y\_n$ given $Y\_1=y\_1$.
But $H(x\_1,x\_2,dots,x\_n)$, and thus $Hleft[w\_1(y\_1,dots,y\_n),\; dots,\; w\_n(y\_1,\; dots,\; y\_n))$
ight], was given not to depend upon $heta$. Since $heta$ was not introduced in the transformation and accordingly not in the Jacobian $J$, it follows that $h(y\_2,\; dots,\; y\_n\; |\; y\_1;\; heta)$ does not depend upon $heta$ and that $Y\_1$ is a sufficient statistics for $heta$.
The converse is proven by taking:
:$g(y\_1,dots,y\_n;\; heta)=g\_1(y\_1;\; heta)\; h(y\_2,\; dots,\; y\_n\; |\; y\_1),,$
where $h(y\_2,\; dots,\; y\_n\; |\; y\_1)$ does not depend upon $heta$ because $Y\_2\; ...\; Y\_n$ depend only upon $X\_1\; ...\; X\_n$ which are independent on $Theta$ when conditioned by $Y\_1$, a sufficient statistics by hypothesis. Now divide both members by the absolute value of the non-vanishing Jacobian $J$, and replace $y\_1,\; dots,\; y\_n$ by the functions $u\_1(x\_1,\; dots,\; x\_n),\; dots,\; u\_n(x\_1,dots,\; x\_n)$ in $x\_1,dots,\; x\_n$. This yields
:
ight]}=g_1left[u_1(x_1,dots,x_n); heta
ight] rac{h(u_2, dots, u_n | u_1)}
where $J$
★ is the Jacobian with $y\_1,dots,y\_n$ replaced by their value in terms $x\_1,\; dots,\; x\_n$. The left-hand member is necessarily the joint pdf $f(x\_1;\; heta)cdots\; f(x\_n;\; heta)$ of $X\_1,dots,X\_n$. Since $h(y\_2,dots,y\_n|y\_1)$, and thus $h(u\_2,dots,u\_n|u\_1)$, does not depend upon $heta$, then:
:
is a function that does not depend upon $heta$.
===Proof for the discrete case [1]===
We use the shorthand notation to denote the joint probability of $(X,\; T(X))$ by $f\_\; heta(x,t)$. Since $T$ is a function of $X$, we have $f\_\; heta(x,t)\; =\; f\_\; heta(x)$ and thus:
:$f\_\; heta(x)\; =\; f\_\; heta(x,t)\; =\; f\_\{\; heta\; |\; t\}(x)\; f\_\; heta(t)$
with the last equality being true by the definition of conditional probability distributions. Thus $f\_\; heta(x)=a(x)\; b\_\; heta(t)$ with $a(x)\; =\; f\_\{\; heta\; |\; t\}(x)$ and $b(x)\; =\; f\_\; heta(t)$.
Reciprocally, if $f\_\; heta(x)=a(x)\; b\_\; heta(t)$, we have
:$$
egin{align}
f_ heta(t) & = sum _{x : T(x) = t} f_ heta(x, t) \
& = sum _{x : T(x) = t} f_ heta(x) \
& = sum _{x : T(x) = t} a(x) b_ heta(t) \
& = left( sum _{x : T(x) = t} a(x)
ight) b_ heta(t)
end{align}
With the first equality by the definition of pdf for multiple variables, the second by the remark above, the third by hypothesis, and the fourth because the summation is not over $t$.
Thus, the conditional probability distribution is:
:$$
egin{align}
f_{ heta|t}(x)
& = rac{f_ heta(x, t)}{f_ heta(t)} \
& = rac{f_ heta(x)}{f_ heta(t)} \
& = rac{a(x) b_ heta(t)}{left( sum _{x : T(x) = t} a(x)
ight) b_ heta(t)} \
& = rac{a(x)}{sum _{x : T(x) = t} a(x)}
end{align}
With the first equality by definition of conditional probability density, the second by the remark above, the third by the equality proven above, and the fourth by simplification. This expression does not depend on $heta$ and thus $T$ is a sufficient statistic.

Minimal sufficiency

A sufficient statistic is 'minimal sufficient' if it can be represented as a function of any other sufficient statistic.
In other words, ''S''(''X'') is 'minimal sufficient' iff
#''S''(''X'') is sufficient, and
#if ''T''(''X'') is sufficient, then there exists a function ''f'' such that ''S''(''X'') = ''f''(''T''(''X'')).
Intuitively, a minimal sufficient statistic most efficiently captures as much information as is possible about the parameter ''θ''.
A complete statistic is necessarily minimal sufficient.
An often used characterization of minimal sufficiency is that when the density $f\_\; heta,$ exists, ''S''(''X'') is 'minimal sufficient' if and only if
: is independent of ''θ'' :$Longleftrightarrow$ ''S''(''x'') = ''S''(''y'')
This follows as a direct consequence from the Fisher's factorization theorem stated above.

Examples

Bernoulli distribution

If ''X''₁, ...., ''X''_''n'' are independent Bernoulli-distributed random variables with expected value ''p'', then the sum ''T''(''X'') = ''X''₁ + ... + ''X''_''n'' is a sufficient statistic for ''p'' (here 'success' corresponds to $X\_i=1$ and 'failure' to $X\_i=0$; so ''T'' is the total number of successes)
This is seen by considering the joint probability distribution:
:$Pr(X=x)=P(X\_1=x\_1,X\_2=x\_2,ldots,X\_n=x\_n).$
Because the observations are independent, this can be written as
:$$
p^{x_1}(1-p)^{1-x_1} p^{x_2}(1-p)^{1-x_2}cdots p^{x_n}(1-p)^{1-x_n} ,!
and, collecting powers of ''p'' and 1 − ''p'', gives
:$$
p^{sum x_i}(1-p)^{n-sum x_i}=p^{T(x)}(1-p)^{n-T(x)} ,!

which satisfies the factorization criterion, with ''h''(''x'')=1 being just a constant.
Note the crucial feature: the unknown parameter ''p'' interacts with the observation ''x'' only via the statistic ''T''(''x'') = Σ ''x''_''i''.

Uniform distribution

If ''X''₁, ...., ''X''_''n'' are independent and uniformly distributed on the interval [0,θ], then ''T''(''X'') = max(''X''₁, ...., ''X''_''n'' ) is sufficient for θ.
To see this, consider the joint probability distribution:
:$$
Pr(X=x)=P(X_1=x_1,X_2=x_2,ldots,X_n=x_n).

Because the observations are independent, this can be written as
:$$
rac{operatorname{H}( heta-x_1)}{ heta}cdot
rac{operatorname{H}( heta-x_2)}{ heta}cdot,cdots,cdot
rac{operatorname{H}( heta-x_n)}{ heta} ,!

where H(''x'') is the Heaviside step function. This may be written as
:$$
rac{operatorname{H}left( heta-max_i {,x_i,}
ight)}{ heta^n},!

which can be viewed as a function of only ''θ'' and max_i(''X''_i) = ''T''(''X''). This shows that the factorization criterion is satisfied, again where ''h''(''x'')=1 is constant.

Poisson distribution

If ''X''₁, ...., ''X''_''n'' are independent and have a Poisson distribution with parameter ''λ'', then the sum ''T''(''X'') = ''X''₁ + ... + ''X''_''n'' is a sufficient statistic for ''λ''.
To see this, consider the joint probability distribution:
:$$
Pr(X=x)=P(X_1=x_1,X_2=x_2,ldots,X_n=x_n).

Because the observations are independent, this can be written as
:$$
{e^{-lambda} lambda^{x_1} over x_1 !} cdot
{e^{-lambda} lambda^{x_2} over x_2 !} cdot,cdots,cdot
{e^{-lambda} lambda^{x_n} over x_n !} ,!

which may be written as
:$$
e^{-nlambda} lambda^{(x_1+x_2+cdots+x_n)} cdot
{1 over x_1 ! x_2 !cdots x_n ! } ,!

which shows that the factorization criterion is satisfied, where ''h''(''x'') is the reciprocal of the product of the factorials.

Rao-Blackwell theorem

'Sufficiency' finds a useful application in the Rao-Blackwell theorem. It states that if ''g''(''X'') is any kind of estimator of ''θ'', then typically the conditional expectation of ''g''(''X'') given ''T''(''X'') is a better estimator of ''θ'', and is never worse. Sometimes one can very easily construct a very crude estimator ''g''(''X''), and then evaluate that conditional expected value to get an estimator that is in various senses optimal.

See also

★ Completeness of a statistic

★ Basu's theorem on independence of complete sufficient & ancillary statistics

★ The Rao-Blackwell theorem on improving an estimator through conditioning with a sufficient statistic

★ The Lehman-Scheffe theorem stating complete sufficient estimator is the best estimator of its expectation

References

External Links

★ Sufficient, complete, and ancillary statistics - University of Alabama in Huntsville