FISHER INFORMATION

In statistics and information theory, the 'Fisher information' (denoted $mathcal\{I\}(\; heta)$) is the variance of the score. It is named in honor of its inventor, the statistician R.A. Fisher.

Contents

Definition

Single-parameter Bernoulli experiment

Matrix form

Multivariate normal distribution

See also

References

Definition

The Fisher information is the amount of information that an observable random variable ''X'' carries about an unknown parameter θ upon which the likelihood function of $heta$, $L(\; heta)=\; f(X;\; heta)$, depends. The likelihood function is the joint probability of the data, the ''X''s, conditional on the value of θ, ''as a function of θ''. Since the expectation of the score is zero, the variance is simply the second moment of the score, the derivative of the log of the likelihood function with respect to θ. Hence the Fisher information can be written
:$$
mathcal{I}( heta)
=
mathrm{E}
left{left.
left[
rac{partial}{partial heta} ln f(X; heta)

ight]^2
ight| heta
ight},

which implies . The Fisher information is thus the expectation of the squared score. A random variable carrying high Fisher information implies that the absolute value of the score is often high.
The Fisher information is not a function of a particular observation, as the random variable ''X'' has been averaged out. The concept of information is useful when comparing two methods of observing a given random process.
If the following regularity condition is met:
:
then the Fisher information may also be written as:
:$$
mathcal{I}( heta) = - mathrm{E} left[ rac{partial^2}{partial heta^2} ln f(X; heta)
ight].

Thus Fisher information is the negative of the expectation of the second derivative of the log of ''f'' with respect to θ.
Information may thus be seen to be a measure of the "sharpness" of the support curve near the maximum likelihood estimate of θ. A "blunt" support curve (one with a shallow maximum) would have a low expected second derivative, and thus low information; while a sharp one would have a high expected second derivative and thus high information.
Information is additive, in that the information yielded by two independent experiments is the sum of the information from each experiment separately:
:$mathcal\{I\}\_\{X,Y\}(\; heta)\; =\; mathcal\{I\}\_X(\; heta)\; +\; mathcal\{I\}\_Y(\; heta).$
This result follows from the elementary fact that if random variables are independent, the variance of their sum is the sum of their variances.
Hence the information in a random sample of size ''n'' is ''n'' times that in a sample of size 1 (if observations are independent).
The information provided by a sufficient statistic is the same as that of the sample ''X''. This may be seen by using Neyman's factorization criterion for a sufficient statistic. If $T(X)$ is sufficient for θ, then
:$f(X;\; heta)\; =\; g(T(X),\; heta)\; h(X)$
for some functions ''g'' and ''h''. See sufficient statistic for a more detailed explanation. The equality of information then follows from the following fact:
:
ight]
= rac{partial}{partial heta} ln left[g(T(X); heta)
ight]
which follows from the definition of Fisher information, and the independence of $h(X)$ from θ. More generally, if $T=t(X)$ is a statistic, then
:$$
mathcal{I}_T( heta)
leq
mathcal{I}_X( heta)

with equality if and only if ''T'' is a sufficient statistic.
The Cramér-Rao inequality states that the inverse of the Fisher information is an asymptotic lower bound on the variance of any unbiased estimator of θ.
Given a likelihood with $p$ parameters, we say that two parameters $heta\_\{i\}$ and $heta\_\{j\}$ are orthogonal if the element of the i-th row and j-th column of the Fisher Information is zero. Orthogonal parameters are easy to deal with in the sense that their maximum likelihood estimates are independent and can be calculated separately. When dealing with research problems, it is very common for the researcher to invest some time searching for an orthogonal parametrization of the densities involved in the problem.

Single-parameter Bernoulli experiment

A Bernoulli trial is a random variable with two possible outcomes, "success" and "failure", with "success" having a probability of $heta$. The outcome can be thought of as determined by a coin toss, with the probability of obtaining a "head" being $heta$ and the probability of obtaining a "tail" being $1\; -\; heta$.
The Fisher information contained in ''n'' independent Bernoulli trials may be calculated as follows. In the following, ''A'' represents the number of successes, ''B'' the number of failures, and $n\; =\; A\; +\; B$ is the total number of trials.
:$$
mathcal{I}( heta)
=
-mathrm{E}
left[
rac{partial^2}{partial heta^2} ln(f(A; heta))
ight] qquad (1)

::$$
=
-mathrm{E}
left[
rac{partial^2}{partial heta^2} ln
left[
heta^A(1- heta)^Brac{(A+B)!}{A!B!}

ight]
ight] qquad (2)

::$$
=
-mathrm{E}
left[
rac{partial^2}{partial heta^2}
left[
A ln ( heta) + B ln(1- heta)

ight]
ight] qquad (3)

::$$
=
-mathrm{E}
left[
rac{partial}{partial heta}
left[
rac{A}{ heta} - rac{B}{1- heta}

ight]
ight]
(on differentiating ln ''x'', see logarithm) $qquad\; (4)$
::$$
=
+mathrm{E}
left[
rac{A}{ heta^2} + rac{B}{(1- heta)^2}
ight] qquad (5)

::$$
=
rac{n heta}{ heta^2} + rac{n(1- heta)}{(1- heta)^2}
(as the expected value of $A\; =\; n\; heta$, etc.) $qquad\; (6)$
::
(1) defines Fisher information.
(2) invokes the fact that the information in a sufficient statistic is the same as that of the sample itself.
(3) expands the log term and drops a constant.
(4) and (5) differentiate with respect to $heta$.
(6) replaces ''A'' and ''B'' with their expectations. (7) is algebra.
The end result, namely,
:
is the reciprocal of the variance of the mean number of successes in ''n'' Bernoulli trials, as expected (see last sentence of the preceding section).

Matrix form

When there are ''N'' parameters, so that θ is a ''N''x1 vector
heta_{1}, heta_{2}, cdots , heta_{N} end{bmatrix},, then the Fisher information takes the form of an ''N''x''N'' matrix, the Fisher information matrix (FIM), with typical element:
:$$
{left(mathcal{I} left( heta
ight)
ight)}_{i, j}
=
mathrm{E}
left[
rac{partial}{partial heta_i} ln f(X; heta)
rac{partial}{partial heta_j} ln f(X; heta)
ight].

The FIM is a ''N''x''N'' positive definite symmetric matrix, defining a metric on the ''N''-dimensional parameter space. Exploring this topic requires differential geometry.

Multivariate normal distribution

The FIM for a ''N''-variate multivariate normal distribution has a special form. Let
mu_{1}( heta), mu_{2}( heta), dots , mu_{N}( heta) end{bmatrix}, and let $Sigma(\; heta)$ be the covariance matrix of $mu(\; heta)$. Then the typical element $mathcal\{I\}\_\{m,n\}$, 0 ≤ ''m'', ''n'' < ''N'', of the FIM for $X\; sim\; N(mu(\; heta),\; Sigma(\; heta))$ is:
:$$
mathcal{I}_{m,n}
=
rac{partial mu}{partial heta_m}
Sigma^{-1}
rac{partial mu^ op}{partial heta_n}
+
rac{1}{2}
mathrm{tr}
left(
Sigma^{-1}
rac{partial Sigma}{partial heta_m}
Sigma^{-1}
rac{partial Sigma}{partial heta_n}
ight),

where $(..)^\; op$ denotes the transpose of a vector, $mathrm\{tr\}(..)$ denotes the trace of a square matrix, and:

★ $$
rac{partial mu}{partial heta_m}
=
egin{bmatrix}
rac{partial mu_1}{partial heta_m} &
rac{partial mu_2}{partial heta_m} &
cdots &
rac{partial mu_N}{partial heta_m} &
end{bmatrix};


★ $$
rac{partial Sigma}{partial heta_m}
=
egin{bmatrix}
rac{partial Sigma_{1,1}}{partial heta_m} &
rac{partial Sigma_{1,2}}{partial heta_m} &
cdots &
rac{partial Sigma_{1,N}}{partial heta_m} \ \
rac{partial Sigma_{2,1}}{partial heta_m} &
rac{partial Sigma_{2,2}}{partial heta_m} &
cdots &
rac{partial Sigma_{2,N}}{partial heta_m} \ \
dots & dots & ddots & dots \ \
rac{partial Sigma_{N,1}}{partial heta_m} &
rac{partial Sigma_{N,2}}{partial heta_m} &
cdots &
rac{partial Sigma_{N,N}}{partial heta_m}
end{bmatrix}.

See also

★ Cramér-Rao inequality

★ Formation matrix
Other measures employed in information theory:

★ Self-information

★ Kullback-Leibler divergence

★ Shannon entropy

References

★ Theory of Statistics, , Mark J., Schervish, Springer, ,