Fisher information

From Wikipedia, the free encyclopedia

In mathematical statistics and information theory, the Fisher information (sometimes simply called information^[1]) is the variance of the score. Its role in the asymptotic theory of maximum-likelihood estimation was emphasized by the statistician R.A. Fisher (following some initial results by F. Y. Edgeworth).^[2]

[edit] Definition

The Fisher information is a way of measuring the amount of information that an observable random variable X carries about an unknown parameter θ upon which the likelihood function of $θ$ , $L (θ) = f (X;θ)$ , depends. The likelihood function is the joint probability of the data, the Xs, conditional on the value of θ, as a function of θ. Since the expectation of the score is zero, the variance is simply the second moment of the score, the derivative of the log of the likelihood function with respect to θ. Hence the Fisher information can be written

$\mathcal{I}(\theta) = \mathrm{E} \left\{\left. \left[ \frac{\partial}{\partial\theta} \ln f(X;\theta) \right]^2 \right|\theta\right\},$

which implies $0 \leq \mathcal{I}(\theta) < \infty$ . The Fisher information is thus the expectation of the squared score. A random variable carrying high Fisher information implies that the absolute value of the score is often high.

The Fisher information is not a function of a particular observation, as the random variable X has been averaged out. The concept of information is useful when comparing two methods of observing a given random process.

If $ln f (x;θ)$ is twice differentiable with respect to $θ$ , and if the regularity condition

$\int \frac{\partial^2}{\partial \theta^2} f(x ; \theta ) \, dx = 0$

holds, then the Fisher information may also be written as^[3]

$\mathcal{I}(\theta) = - \mathrm{E} \left[\left. \frac{\partial^2}{\partial\theta^2} \ln f(X;\theta)\right|\theta \right].$

Thus Fisher information is the negative of the expectation of the second derivative of the log of f with respect to θ. Information may thus be seen to be a measure of the "sharpness" of the support curve near the maximum likelihood estimate of θ. A "blunt" support curve (one with a shallow maximum) would have a low expected second derivative, and thus low information; while a sharp one would have a high expected second derivative and thus high information.

Information is additive, in that the information yielded by two independent experiments is the sum of the information from each experiment separately:

$\mathcal{I}_{X,Y}(\theta) = \mathcal{I}_X(\theta) + \mathcal{I}_Y(\theta).$

This result follows from the elementary fact that if random variables are independent, the variance of their sum is the sum of their variances. Hence the information in a random sample of size n is n times that in a sample of size 1 (if observations are independent).

The information provided by a sufficient statistic is the same as that of the sample X. This may be seen by using Neyman's factorization criterion for a sufficient statistic. If $T (X)$ is sufficient for θ, then

$f(X;\theta) = g(T(X), \theta) h(X) \!$

for some functions g and h. See sufficient statistic for a more detailed explanation. The equality of information then follows from the following fact:

$\frac{\partial}{\partial\theta} \ln \left[f(X ;\theta)\right] = \frac{\partial}{\partial\theta} \ln \left[g(T(X);\theta)\right]$

which follows from the definition of Fisher information, and the independence of $h (X)$ from θ. More generally, if $T = t (X)$ is a statistic, then

$\mathcal{I}_T(\theta) \leq \mathcal{I}_X(\theta)$

with equality if and only if T is a sufficient statistic.

The Cramér-Rao inequality states that the inverse of the Fisher information is a lower bound on the variance of any unbiased estimator of θ.

[edit] Informal derivation

Van Trees (1968) and Frieden (2004) provide the following method of deriving the Fisher information informally:

Consider an unbiased estimator $\hat\theta(X)$ . Mathematically, we write

$\mathrm{E}\left[ \hat\theta(X) - \theta \right] = \int \left[ \hat\theta(X) - \theta \right] \cdot f(X ;\theta) \, dx = 0.$

The likelihood function $f (X;θ)$ describes the probability that we observe a given sample $x$ given a known value of $θ$ . If $f$ is sharply peaked, it is easy to intuit the "correct" value of $θ$ given the data, and hence the data contains a lot of information about the parameter. If the likelihood $f$ is flat and spread-out, then it would take many, many samples of $X$ to estimate the actual "true" value of $θ$ . Therefore, we would intuit that the data contain much less information about the parameter.

Now, we differentiate the unbiased-ness condition above to get

$\frac{\partial}{\partial\theta} \int \left[ \hat\theta(X) - \theta \right] \cdot f(X ;\theta) \, dx = \int \left(\hat\theta-\theta\right) \frac{\partial f}{\partial\theta} \, dx - \int f \, dx = 0.$

We now make use of two facts. The first is that the likelihood $f$ is just the probability of the data given the parameter. Since it is a probability, it must be normalized, implying that

$\int f \, dx = 1.$

Second, we know from basic calculus that

$\frac{\partial f}{\partial\theta} = f \, \frac{\partial \ln f}{\partial\theta}$ .

Using these two facts in the above let us write

$\int \left(\hat\theta-\theta\right) f \, \frac{\partial \ln f}{\partial\theta} \, dx = 1.$

Factoring the integrand gives

$\int \left(\left(\hat\theta-\theta\right) \sqrt{f} \right) \left( \sqrt{f} \, \frac{\partial \ln f}{\partial\theta} \right) \, dx = 1.$

If we square the equation, the Cauchy-Schwarz inequality lets us write

$\left[ \int \left(\hat\theta - \theta\right)^2 f \, dx \right] \cdot \left[ \int \left( \frac{\partial \ln f}{\partial\theta} \right)^2 f \, dx \right] \geq 1.$

The right-most factor is defined to be the Fisher Information

$\mathcal{I}\left(\theta\right) = \int \left( \frac{\partial \ln f}{\partial\theta} \right)^2 f \, dx.$

The left-most factor is the expected mean-squared error of the estimator $θ$ , since

$\mathrm{E}\left[ \left( \hat\theta\left(X\right) - \theta \right)^2 \right] = \int \left(\hat\theta - \theta\right)^2 f \, dx.$

Notice that the inequality tells us that, fundamentally,

$\mbox{Var}\left[\hat\theta\right] \, \geq \, {1} / {\mathcal{I}\left(\theta\right)}.$

In other words, the precision to which we can estimate $θ$ is fundamentally limited by the Fisher Information of likelihood function.

[edit] Single-parameter Bernoulli experiment

A Bernoulli trial is a random variable with two possible outcomes, "success" and "failure", with "success" having a probability of $θ$ . The outcome can be thought of as determined by a coin toss, with the probability of obtaining a "head" being $θ$ and the probability of obtaining a "tail" being $1 - θ$ .

The Fisher information contained in n independent Bernoulli trials may be calculated as follows. In the following, A represents the number of successes, B the number of failures, and $n = A + B$ is the total number of trials.

$\mathcal{I}(\theta) = -\mathrm{E} \left[ \frac{\partial^2}{\partial\theta^2} \ln(f(A;\theta)) \right] \qquad (1)$

$= -\mathrm{E} \left[ \frac{\partial^2}{\partial\theta^2} \ln \left[ \theta^A(1-\theta)^B\frac{(A+B)!}{A!B!} \right] \right] \qquad (2)$

$= -\mathrm{E} \left[ \frac{\partial^2}{\partial\theta^2} \left[ A \ln (\theta) + B \ln(1-\theta) \right] \right] \qquad (3)$

$= -\mathrm{E} \left[ \frac{\partial}{\partial\theta} \left[ \frac{A}{\theta} - \frac{B}{1-\theta} \right] \right]$ (on differentiating ln x, see logarithm) $\qquad (4)$

$= +\mathrm{E} \left[ \frac{A}{\theta^2} + \frac{B}{(1-\theta)^2} \right] \qquad (5)$

$= \frac{n\theta}{\theta^2} + \frac{n(1-\theta)}{(1-\theta)^2}$ (as the expected value of

A = n θ

, etc.) $\qquad (6)$

$= \frac{n}{\theta(1-\theta)} \qquad (7)$

(1) defines Fisher information. (2) invokes the fact that the information in a sufficient statistic is the same as that of the sample itself. (3) expands the log term and drops a constant. (4) and (5) differentiate with respect to $θ$ . (6) replaces A and B with their expectations. (7) is algebra.

The end result, namely,

$\mathcal{I}(\theta) = \frac{n}{\theta(1-\theta)},$

is the reciprocal of the variance of the mean number of successes in n Bernoulli trials, as expected (see last sentence of the preceding section).

[edit] Matrix form

When there are N parameters, so that θ is a Nx1 vector $\theta = \begin{bmatrix} \theta_{1}, \theta_{2}, \dots , \theta_{N} \end{bmatrix},$ then the Fisher information takes the form of an NxN matrix, the Fisher Information Matrix (FIM), with typical element:

${\left(\mathcal{I} \left(\theta \right) \right)}_{i, j} = \mathrm{E} \left[\left. \frac{\partial}{\partial\theta_i} \ln f(X;\theta) \frac{\partial}{\partial\theta_j} \ln f(X;\theta) \right|\theta\right].$

The FIM is a NxN positive semidefinite symmetric matrix, defining a Riemannian metric on the N-dimensional parameter space, thus connecting Fisher information to differential geometry. In that context, this metric is known as the Fisher information metric, and the topic is called information geometry.

If the following regularity condition is met:

$\int \frac{\partial^2}{\partial \theta_i \partial \theta_j} f(X ; \theta ) \, dx = 0,$

then the Fisher Information Matrix may also be written as:

${\left(\mathcal{I} \left(\theta \right) \right)}_{i, j} = - \mathrm{E} \left[\left. \frac{\partial^2}{\partial\theta_i \partial\theta_j} \ln f(X;\theta) \right|\theta\right].$

[edit] Orthogonal parameters

We say that two parameters $θ i$ and $θ j$ are orthogonal if the element of the i-th row and j-th column of the Fisher Information Matrix is zero. Orthogonal parameters are easy to deal with in the sense that their maximum likelihood estimates are independent and can be calculated separately. When dealing with research problems, it is very common for the researcher to invest some time searching for an orthogonal parametrization of the densities involved in the problem.

[edit] Multivariate normal distribution

The FIM for a N-variate multivariate normal distribution has a special form. Let $\mu(\theta) = \begin{bmatrix} \mu_{1}(\theta), \mu_{2}(\theta), \dots , \mu_{N}(\theta) \end{bmatrix}^\top,$ and let $Σ(θ)$ be the covariance matrix. Then the typical element $\mathcal{I}_{m,n}$ , 0 ≤ m, n < N, of the FIM for $X \sim N(\mu(\theta), \Sigma(\theta))$ is:

$\mathcal{I}_{m,n} = \frac{\partial \mu^\top}{\partial \theta_m} \Sigma^{-1} \frac{\partial \mu}{\partial \theta_n} + \frac{1}{2} \mathrm{tr} \left( \Sigma^{-1} \frac{\partial \Sigma}{\partial \theta_m} \Sigma^{-1} \frac{\partial \Sigma}{\partial \theta_n} \right),$

where $(..)^\top$ denotes the transpose of a vector, $tr(..)$ denotes the trace of a square matrix, and:

$\frac{\partial \mu}{\partial \theta_m} = \begin{bmatrix} \frac{\partial \mu_1}{\partial \theta_m} & \frac{\partial \mu_2}{\partial \theta_m} & \cdots & \frac{\partial \mu_N}{\partial \theta_m} & \end{bmatrix}^\top;$

$\frac{\partial \Sigma}{\partial \theta_m} = \begin{bmatrix} \frac{\partial \Sigma_{1,1}}{\partial \theta_m} & \frac{\partial \Sigma_{1,2}}{\partial \theta_m} & \cdots & \frac{\partial \Sigma_{1,N}}{\partial \theta_m} \\ \\ \frac{\partial \Sigma_{2,1}}{\partial \theta_m} & \frac{\partial \Sigma_{2,2}}{\partial \theta_m} & \cdots & \frac{\partial \Sigma_{2,N}}{\partial \theta_m} \\ \\ \vdots & \vdots & \ddots & \vdots \\ \\ \frac{\partial \Sigma_{N,1}}{\partial \theta_m} & \frac{\partial \Sigma_{N,2}}{\partial \theta_m} & \cdots & \frac{\partial \Sigma_{N,N}}{\partial \theta_m} \end{bmatrix}.$

Note that a special, but very common case is the one where $Σ(θ) = Σ$ , a constant. Then

$\mathcal{I}_{m,n} = \frac{\partial \mu^\top}{\partial \theta_m} \Sigma^{-1} \frac{\partial \mu}{\partial \theta_n}$ .

In this case the Fisher information matrix may be identified with the coefficient matrix of the normal equations of least squares estimation theory.

[edit] Properties

The Fisher information depends on the parametrization of the problem. If θ and η are two different parameterizations of a problem, such that $θ = h (η)$ and h is a differentiable function, then

${\mathcal I}_\eta(\eta) = {\mathcal I}_\theta(h(\eta)) \left( h'(\eta) \right)^2$

where ${\mathcal I}_\eta$ and ${\mathcal I}_\theta$ are the Fisher information measures of η and θ, respectively.^[4]

In information geometry, this is seen as a change of coordinates on a Riemannian manifold, and the intrinsic properties of curvature are unchanged under different parametrization.

[edit] Applications

[edit] Optimal design of experiments

Fisher information is widely used in optimal experimental design. Because of the reciprocity of estimator-variance and Fisher information, minimizing the variance corresponds to maximizing the information.

When the linear (or linearized) statistical model has several parameters, the mean of the parameter-estimator is a vector and its variance is a matrix. The inverse matrix of the variance-matrix is called the "information matrix". Because the variance of the estimator of a parameter vector is a matrix, the problem of "minimizing the variance" is complicated. Using statistical theory, statisticians compress the information-matrix using real-valued summary statistics; being real-valued functions, these "information criteria" can be maximized.^[5] The traditional optimality-criteria are the information-matrix's invariants; algebraically, the traditional optimality-criteria are functionals of the eigenvalues of the (Fisher) information matrix.^[6]

[edit] See also

Other measures employed in information theory:

[edit] Notes

^ Lehmann and Casella, p. 115
^ The Fisher information was discussed by several early statisticians, notably F. Y. Edgeworth:
- Leonard J. Savage (May 1976)). "On Rereading R. A. Fisher". The Annals of Statistics 4 (3): pp. 441-500. Stable URL: http://www.jstor.org/stable/2958221
  - "In it [Fisher information], he [Fisher] was to some extent anticipated (Egeworth 1908--9 esp. 502, 507--8, 662, 677--8,82-5 and references he [Edgeworth] cites including Pearson and Filon 1898 [. . .])." [page 156 in Savage]
- F. Y. Edgeworth (Sep. 1908). "On the Probable Errors of Frequency-Constants". Journal of the Royal Statistical Society 71 (3): pp. 499--512. Stable URL: http://www.jstor.org/stable/2339293
- F. Y. Edgeworth (Dec. 1908). "On the Probable Errors of Frequency-Constants". Journal of the Royal Statistical Society 71 (4): pp. 651--678. Stable URL: http://www.jstor.org/stable/2339378
- John W. Pratt (May 1976). "F. Y. Edgeworth and R. A. Fisher on the Efficiency of Maximum Likelihood Estimation". The Annals of Statistics 4 (3): pp. 501-514. Stable URL: http://www.jstor.org/stable/2958222
- Stephen M. Stigler (1978). "Francis Ysidro Edgeworth, Statistician". Journal of the Royal Statistical Society. Series A (General) 141 (3). Stable URL: http://www.jstor.org/stable/2344804
- Stephen Stigler. The History of Statistics: The Measurement of Uncertainty before 1900. Harvard University Press. ISBN 0-674-40340-1.
- Stephen Stigler. Statistics on the Table: The History of Statistical Concepts and Methods. Harvard University Press. ISBN 0-674-83601-4.
- Anders Hald (May 1999). "On the History of Maximum Likelihood in Relation to Inverse Probability and Least Squares". Statistical Science 14 (2). Stable URL: http://www.jstor.org/stable/2676741
- Hald, A. (1998). A History of Mathematical Statistics from 1750 to 1930. New York: Wiley.
^ Lehmann and Casella, eq. (2.5.16).
^ Lehmann and Casella, eq. (2.5.11).
^ Traditionally, statistians have evaluated estimators and designs by considering some summary statistic of the covariance matrix (of a mean-unbiased estimator), usually with positive real values (like the determinant or matrix trace). Working with positive real-numbers brings several advantages: If the estimator of a single parameter has a positive variance, then the variance and the Fisher information are both positive real numbers; hence they are members of the convex cone of nonnegative real numbers (whose nonzero members have reciprocals in this same cone).
For several parameters, the covariance-matrices and information-matrices are elements of the convex cone of nonnegative-definite symmetric matrices in a partially ordered vector space, under the Loewner (Löwner) order. This cone is closed under matrix-matrix addition, under matrix-inversion, and under the multiplication of positive real-numbers and matrices.
An exposition of matrix theory and the Loewner-order appears in Pukelsheim.
^ The traditional criteria are the information-matrix's invariants; algebraically, the traditional optimality-criteria are functionals of the eigenvalues of the inverse-matrix of the covariance matrix.
- A-optimality ("average" or trace)
  - One criterion is A-optimality, which seeks to minimize the trace of the inverse of the information matrix. This criterion results in minimizing the average variance of the estimates of the regression coefficients.
- D-optimality (determinant)
  - A popular criterion is D-optimality, which seeks to minimize |(X'X)^-1|, the determinant of the information matrix X'X of the design. This criterion results in maximizing the differential Shannon information content of the parameter estimates.
- E-optimality (eigenvalue)
  - Another design is E-optimality, which maximizes the minimum eigenvalue of the information matrix. The E-optimality criterion need not be differentiable at every point. Such E-optimal designs can be computed using methods of convex minimization that use subgradients rather than gradients at points of non-differentiability. Any non-differentiability need not be a serious problem, however: E-optimality problems are special cases of semidefinite-programming problems which have effective solution-methods, especially bundle methods and interior-point methods.
Other optimality-criteria are concerned with the variance of predictions:
- G-optimality
  - A popular criterion is G-optimality, which seeks to minimize the maximum entry in the diagonal of the hat matrix X(X'X)^-1X'. This has the effect of minimizing the maximum variance of the predicted values.
- I-optimality (integrated)
  - A second criterion on prediction variance is I-optimality, which seeks to minimize the mean squared prediction error.
- V-optimality (variance)
  - A third criterion on prediction variance is V-optimality, which seeks to minimize the average prediction variance.
Again, the above optimality-criteria are convex functions on domains of symmetric positive-semidefinite matrices: See an on-line textbook for practitioners, which has many illustrations and statistical applications:
- Boyd, Stephen and Vandenberghe, Lieven (2004). Convex Optimization. Cambridge University Press. http://www.stanford.edu/~boyd/cvxbook/. (book in pdf)
Boyd and Vandenberghe discuss optimal experimental designs on pages 384--396.
- Atkinson, A. C. and Donev, A. N. and Tobias, R. D. (2007). Optimum Experimental Designs, with SAS. Oxford University Press. pp. 511+xvi. ISBN 978-0-19-929660-6.
- Kiefer, Jack Carl. (1985). Brown, Lawrence D. and Olkin, Ingram and Sacks, Jerome and Wynn, Henry P.. ed. Jack Carl Kiefer Collected Papers III Design of Experiments. Springer-Verlag and the Institute of Mathematical Statistics. pp. 718+xxv. ISBN 0-387-96004-X.
- Pukelsheim, Friedrich (1993). Optimal Design of Experiments. Wiley. pp. 454+xxv. ISBN 0-471-61971-X. http://de.wikipedia.org/wiki/Pukelsheim.

[edit] References

Schervish, Mark J. (1995). "Section 2.3.1". Theory of Statistics. New York: Springer. ISBN 0387945466.

Leonard J. Savage (May 1976)). "On Rereading R. A. Fisher". The Annals of Statistics 4 (3): pp. 441-500. Stable URL: http://www.jstor.org/stable/2958221
- "In it [Fisher information], he [Fisher] was to some extent anticipated (Egeworth 1908--9 esp. 502, 507--8, 662, 677--8,82-5 and references he [Edgeworth] cites including Pearson and Filon 1898 [. . .])." [page 156 in Savage]

F. Y. Edgeworth (Sep. 1908). "On the Probable Errors of Frequency-Constants". Journal of the Royal Statistical Society 71 (3): pp. 499--512. Stable URL: http://www.jstor.org/stable/2339293

F. Y. Edgeworth (Dec. 1908). "On the Probable Errors of Frequency-Constants". Journal of the Royal Statistical Society 71 (4): pp. 651--678. Stable URL: http://www.jstor.org/stable/2339378

John W. Pratt (May 1976). "F. Y. Edgeworth and R. A. Fisher on the Efficiency of Maximum Likelihood Estimation". The Annals of Statistics 4 (3): pp. 501-514. Stable URL: http://www.jstor.org/stable/2958222

Stephen M. Stigler (1978). "Francis Ysidro Edgeworth, Statistician". Journal of the Royal Statistical Society. Series A (General) 141 (3). Stable URL: http://www.jstor.org/stable/2344804

Stephen Stigler. The History of Statistics: The Measurement of Uncertainty before 1900. Harvard University Press. ISBN 0-674-40340-1.

Stephen Stigler. Statistics on the Table: The History of Statistical Concepts and Methods. Harvard University Press. ISBN 0-674-83601-4.

Anders Hald (May 1999). "On the History of Maximum Likelihood in Relation to Inverse Probability and Least Squares". Statistical Science 14 (2). Stable URL: http://www.jstor.org/stable/2676741

Hald, A. (1998). A History of Mathematical Statistics from 1750 to 1930. New York: Wiley.

Van Trees, H. L. (1968). Detection, Estimation, and Modulation Theory, Part I. New York: Wiley. ISBN 0471095176.

Lehmann, E. L.; Casella, G. (1998). Theory of Point Estimation. Springer. pp. 2nd ed. ISBN 0-387-98502-6.

[edit] Further weblinks

Fisher4Cast: a Matlab, GUI-based Fisher information tool for research and teaching, primarily aimed at cosmological forecasting applications.

[0] Lehmann and Casella, p. 115

[1] The Fisher information was discussed by several early statisticians, notably F. Y. Edgeworth:

Leonard J. Savage (May 1976)). "On Rereading R. A. Fisher". The Annals of Statistics 4 (3): pp. 441-500. Stable URL: http://www.jstor.org/stable/2958221

"In it [Fisher information], he [Fisher] was to some extent anticipated (Egeworth 1908--9 esp. 502, 507--8, 662, 677--8,82-5 and references he [Edgeworth] cites including Pearson and Filon 1898 [. . .])." [page 156 in Savage]

F. Y. Edgeworth (Sep. 1908). "On the Probable Errors of Frequency-Constants". Journal of the Royal Statistical Society 71 (3): pp. 499--512. Stable URL: http://www.jstor.org/stable/2339293

F. Y. Edgeworth (Dec. 1908). "On the Probable Errors of Frequency-Constants". Journal of the Royal Statistical Society 71 (4): pp. 651--678. Stable URL: http://www.jstor.org/stable/2339378

John W. Pratt (May 1976). "F. Y. Edgeworth and R. A. Fisher on the Efficiency of Maximum Likelihood Estimation". The Annals of Statistics 4 (3): pp. 501-514. Stable URL: http://www.jstor.org/stable/2958222

Stephen M. Stigler (1978). "Francis Ysidro Edgeworth, Statistician". Journal of the Royal Statistical Society. Series A (General) 141 (3). Stable URL: http://www.jstor.org/stable/2344804

Stephen Stigler. The History of Statistics: The Measurement of Uncertainty before 1900. Harvard University Press. ISBN 0-674-40340-1.

Stephen Stigler. Statistics on the Table: The History of Statistical Concepts and Methods. Harvard University Press. ISBN 0-674-83601-4.

Anders Hald (May 1999). "On the History of Maximum Likelihood in Relation to Inverse Probability and Least Squares". Statistical Science 14 (2). Stable URL: http://www.jstor.org/stable/2676741

Hald, A. (1998). A History of Mathematical Statistics from 1750 to 1930. New York: Wiley.

[3] Leonard J. Savage (May 1976)). "On Rereading R. A. Fisher". The Annals of Statistics 4 (3): pp. 441-500. Stable URL: http://www.jstor.org/stable/2958221

"In it [Fisher information], he [Fisher] was to some extent anticipated (Egeworth 1908--9 esp. 502, 507--8, 662, 677--8,82-5 and references he [Edgeworth] cites including Pearson and Filon 1898 [. . .])." [page 156 in Savage]

[4] "In it [Fisher information], he [Fisher] was to some extent anticipated (Egeworth 1908--9 esp. 502, 507--8, 662, 677--8,82-5 and references he [Edgeworth] cites including Pearson and Filon 1898 [. . .])." [page 156 in Savage]

[5] F. Y. Edgeworth (Sep. 1908). "On the Probable Errors of Frequency-Constants". Journal of the Royal Statistical Society 71 (3): pp. 499--512. Stable URL: http://www.jstor.org/stable/2339293

[6] F. Y. Edgeworth (Dec. 1908). "On the Probable Errors of Frequency-Constants". Journal of the Royal Statistical Society 71 (4): pp. 651--678. Stable URL: http://www.jstor.org/stable/2339378

[7] John W. Pratt (May 1976). "F. Y. Edgeworth and R. A. Fisher on the Efficiency of Maximum Likelihood Estimation". The Annals of Statistics 4 (3): pp. 501-514. Stable URL: http://www.jstor.org/stable/2958222

[8] Stephen M. Stigler (1978). "Francis Ysidro Edgeworth, Statistician". Journal of the Royal Statistical Society. Series A (General) 141 (3). Stable URL: http://www.jstor.org/stable/2344804

[9] Stephen Stigler. The History of Statistics: The Measurement of Uncertainty before 1900. Harvard University Press. ISBN 0-674-40340-1.

[10] Stephen Stigler. Statistics on the Table: The History of Statistical Concepts and Methods. Harvard University Press. ISBN 0-674-83601-4.

[11] Anders Hald (May 1999). "On the History of Maximum Likelihood in Relation to Inverse Probability and Least Squares". Statistical Science 14 (2). Stable URL: http://www.jstor.org/stable/2676741

[12] Hald, A. (1998). A History of Mathematical Statistics from 1750 to 1930. New York: Wiley.

[2] Lehmann and Casella, eq. (2.5.16).

[3] Lehmann and Casella, eq. (2.5.11).

[4] Traditionally, statistians have evaluated estimators and designs by considering some summary statistic of the covariance matrix (of a mean-unbiased estimator), usually with positive real values (like the determinant or matrix trace). Working with positive real-numbers brings several advantages: If the estimator of a single parameter has a positive variance, then the variance and the Fisher information are both positive real numbers; hence they are members of the convex cone of nonnegative real numbers (whose nonzero members have reciprocals in this same cone).
For several parameters, the covariance-matrices and information-matrices are elements of the convex cone of nonnegative-definite symmetric matrices in a partially ordered vector space, under the Loewner (Löwner) order. This cone is closed under matrix-matrix addition, under matrix-inversion, and under the multiplication of positive real-numbers and matrices.
An exposition of matrix theory and the Loewner-order appears in Pukelsheim.

[5] The traditional criteria are the information-matrix's invariants; algebraically, the traditional optimality-criteria are functionals of the eigenvalues of the inverse-matrix of the covariance matrix.

A-optimality ("average" or trace)

One criterion is A-optimality, which seeks to minimize the trace of the inverse of the information matrix. This criterion results in minimizing the average variance of the estimates of the regression coefficients.

D-optimality (determinant)

A popular criterion is D-optimality, which seeks to minimize |(X'X)^-1|, the determinant of the information matrix X'X of the design. This criterion results in maximizing the differential Shannon information content of the parameter estimates.

E-optimality (eigenvalue)

Another design is E-optimality, which maximizes the minimum eigenvalue of the information matrix. The E-optimality criterion need not be differentiable at every point. Such E-optimal designs can be computed using methods of convex minimization that use subgradients rather than gradients at points of non-differentiability. Any non-differentiability need not be a serious problem, however: E-optimality problems are special cases of semidefinite-programming problems which have effective solution-methods, especially bundle methods and interior-point methods.

Other optimality-criteria are concerned with the variance of predictions:

G-optimality

A popular criterion is G-optimality, which seeks to minimize the maximum entry in the diagonal of the hat matrix X(X'X)^-1X'. This has the effect of minimizing the maximum variance of the predicted values.

I-optimality (integrated)

A second criterion on prediction variance is I-optimality, which seeks to minimize the mean squared prediction error.

V-optimality (variance)

A third criterion on prediction variance is V-optimality, which seeks to minimize the average prediction variance.

Again, the above optimality-criteria are convex functions on domains of symmetric positive-semidefinite matrices: See an on-line textbook for practitioners, which has many illustrations and statistical applications:

Boyd, Stephen and Vandenberghe, Lieven (2004). Convex Optimization. Cambridge University Press. http://www.stanford.edu/~boyd/cvxbook/. (book in pdf)

Boyd and Vandenberghe discuss optimal experimental designs on pages 384--396.

Atkinson, A. C. and Donev, A. N. and Tobias, R. D. (2007). Optimum Experimental Designs, with SAS. Oxford University Press. pp. 511+xvi. ISBN 978-0-19-929660-6.

Kiefer, Jack Carl. (1985). Brown, Lawrence D. and Olkin, Ingram and Sacks, Jerome and Wynn, Henry P.. ed. Jack Carl Kiefer Collected Papers III Design of Experiments. Springer-Verlag and the Institute of Mathematical Statistics. pp. 718+xxv. ISBN 0-387-96004-X.

Pukelsheim, Friedrich (1993). Optimal Design of Experiments. Wiley. pp. 454+xxv. ISBN 0-471-61971-X. http://de.wikipedia.org/wiki/Pukelsheim.

[17] A-optimality ("average" or trace)

One criterion is A-optimality, which seeks to minimize the trace of the inverse of the information matrix. This criterion results in minimizing the average variance of the estimates of the regression coefficients.

[18] One criterion is A-optimality, which seeks to minimize the trace of the inverse of the information matrix. This criterion results in minimizing the average variance of the estimates of the regression coefficients.

[19] D-optimality (determinant)

A popular criterion is D-optimality, which seeks to minimize |(X'X)^-1|, the determinant of the information matrix X'X of the design. This criterion results in maximizing the differential Shannon information content of the parameter estimates.

[20] A popular criterion is D-optimality, which seeks to minimize |(X'X)^-1|, the determinant of the information matrix X'X of the design. This criterion results in maximizing the differential Shannon information content of the parameter estimates.

[21] E-optimality (eigenvalue)

Another design is E-optimality, which maximizes the minimum eigenvalue of the information matrix. The E-optimality criterion need not be differentiable at every point. Such E-optimal designs can be computed using methods of convex minimization that use subgradients rather than gradients at points of non-differentiability. Any non-differentiability need not be a serious problem, however: E-optimality problems are special cases of semidefinite-programming problems which have effective solution-methods, especially bundle methods and interior-point methods.

[22] Another design is E-optimality, which maximizes the minimum eigenvalue of the information matrix. The E-optimality criterion need not be differentiable at every point. Such E-optimal designs can be computed using methods of convex minimization that use subgradients rather than gradients at points of non-differentiability. Any non-differentiability need not be a serious problem, however: E-optimality problems are special cases of semidefinite-programming problems which have effective solution-methods, especially bundle methods and interior-point methods.

[23] G-optimality

A popular criterion is G-optimality, which seeks to minimize the maximum entry in the diagonal of the hat matrix X(X'X)^-1X'. This has the effect of minimizing the maximum variance of the predicted values.

[24] A popular criterion is G-optimality, which seeks to minimize the maximum entry in the diagonal of the hat matrix X(X'X)^-1X'. This has the effect of minimizing the maximum variance of the predicted values.

[25] I-optimality (integrated)

A second criterion on prediction variance is I-optimality, which seeks to minimize the mean squared prediction error.

[26] A second criterion on prediction variance is I-optimality, which seeks to minimize the mean squared prediction error.

[27] V-optimality (variance)

A third criterion on prediction variance is V-optimality, which seeks to minimize the average prediction variance.

[28] A third criterion on prediction variance is V-optimality, which seeks to minimize the average prediction variance.

[29] Boyd, Stephen and Vandenberghe, Lieven (2004). Convex Optimization. Cambridge University Press. http://www.stanford.edu/~boyd/cvxbook/. (book in pdf)

[30] Atkinson, A. C. and Donev, A. N. and Tobias, R. D. (2007). Optimum Experimental Designs, with SAS. Oxford University Press. pp. 511+xvi. ISBN 978-0-19-929660-6.

[31] Kiefer, Jack Carl. (1985). Brown, Lawrence D. and Olkin, Ingram and Sacks, Jerome and Wynn, Henry P.. ed. Jack Carl Kiefer Collected Papers III Design of Experiments. Springer-Verlag and the Institute of Mathematical Statistics. pp. 718+xxv. ISBN 0-387-96004-X.

[32] Pukelsheim, Friedrich (1993). Optimal Design of Experiments. Wiley. pp. 454+xxv. ISBN 0-471-61971-X. http://de.wikipedia.org/wiki/Pukelsheim.

[1]

[2]

[3]

[4]

[5]

[6]

Fisher information

From Wikipedia, the free encyclopedia

Contents

[edit] Definition

[edit] Informal derivation

[edit] Single-parameter Bernoulli experiment

[edit] Matrix form

[edit] Orthogonal parameters

[edit] Multivariate normal distribution

[edit] Properties

[edit] Applications

[edit] Optimal design of experiments

[edit] See also

[edit] Notes

[edit] References

[edit] Further weblinks

Views

Personal tools

Navigation

Search

Interaction

Toolbox

Languages