Bayesian probability

From Wikipedia, the free encyclopedia
Jump to: navigation, search

Bayesian probability is one of the different interpretations of the concept of probability and belongs to the category of evidential probabilities. The Bayesian interpretation of probability can be seen as an extension of logic that enables reasoning with uncertain statements. To evaluate the probability of a hypothesis, the Bayesian probabilist specifies some prior probability, which is then updated in the light of new relevant data. The Bayesian interpretation provides a standard set of procedures and formulae to perform this calculation. Bayesian probability interprets the concept of probability as "a measure of a state of knowledge",[1] in contrast to interpreting it as a frequency or a physical property of a system.

"Bayesian" refers to the 18th century statistician Thomas Bayes (1702–1761), who provided an early, mathematically incomplete treatment of a problem in Bayesian inference.[2] Ironically, Bayes was a minor figure in the history of science, who had little or no impact on the early development of statistics; it was the French mathematician Pierre-Simon Laplace (1749–1827) who pioneered and popularized what is now called Bayesian probability.[3]

Broadly speaking, there are two views on Bayesian probability that interpret the state of knowledge concept in different ways. According to the objectivist view, the rules of Bayesian statistics can be justified by requirements of rationality and consistency and interpreted as an extension of logic.[1][4] According to the subjectivist view, the state of knowledge measures a "personal belief".[5] Many modern machine learning methods are based on objectivist Bayesian principles.[6] One of the crucial features of the Bayesian view is that a probability is assigned to a hypothesis, whereas under the frequentist view, a hypothesis is typically rejected or not rejected without directly assigning a probability.

Contents

[edit] Calculations with Bayesian probabilities

Main article: Bayes' theorem
Main article: Bayesian inference

Bayes' theorem is one of the main tools for manipulating probabilities of any kind; that is, it is applicable no matter what interpretation is being placed on the probabilities being manipulated. Bayesian inference is a formal approach to making statistical inferences in cases where some of the probabilities are interpreted as representing beliefs, or knowledge, rather than having a frequency-based interpretation. While "Bayesian inference" makes uses of Bayes' theorem, not all cases where Bayes' theorem is applied should be labelled as "Bayesian statistics" or "Bayesian inference".

The use of Bayes' theorem in Bayesian inference may be described as follows. Let H denote a hypothesis; that a certain statement of supposed fact is true, or that a statistical parameter takes a certain value. Before observing data from a given experiment, one starts with some belief about whether the hypothesis H is true, expressed in the form of a probability, usually called the prior probability. Bayes' theorem is used to determine what one's probability for the hypothesis should be, once the outcome D from the experiment is known. The phrase "should be" is important here, as Bayes' theorem is a condensation of the rules that anyone should apply to updating beliefs, provided that they are acting according to reasonable rules of requirements of rationality and consistency.[1][4] The probability of the hypothesis once the outcome from the experiment is known is called the posterior probability.

The posterior probability is proportional to the likelihood of the observed data, multiplied by the prior probability, and is given by Bayes' theorem. Thus

\operatorname{P}(H|D) = \frac{\operatorname{P}(D|H)\;\operatorname{P}(H)}{\operatorname{P}(D)},

where

The quantity \operatorname{P}(D) is the prior probability of witnessing the data D under all possible hypotheses, and it depends on the prior probabilities given to each of these other possible hypotheses. Given any exhaustive set of mutually exclusive hypotheses Hi,

\operatorname{P}(D) = \sum_i \operatorname{P}(D, H_i) = \sum_i \operatorname{P}(D|H_i)\operatorname{P}(H_i).\,

Here i can be considered to index alternative cases, of which exactly one is actually valid, and Hi is the hypothesis that case i is valid. Then \operatorname{P}(D, H_i) is then the probability that both case i is valid and that the data from the experiment turn out to be what was observed. Since the set of alternative cases is assumed to be mutually exclusive and exhaustive, the above formula is a case of the law of total probability. In many cases, \operatorname{P}(D), which is a normalizing constant, need not be evaluated. As a result, Bayes' formula is often simplified to:

\operatorname{P}(H|D) \propto \operatorname{P}(D|H)\;\operatorname{P}(H),

where \propto denotes proportionality.

[edit] Bayesian methodology

In general, Bayesian methods are characterized by the following concepts and procedures:

[edit] Objective and subjective Bayesian probabiliities

Broadly speaking, there are two views on Bayesian probability that interpret the 'state of knowledge' concept in different ways. For objectivists, the rules of Bayesian statistics can be justified by requirements of rationality and consistency[1][4]. Such requirements of rationality and consistency are also important for subjectivists, for which the state of knowledge corresponds to a 'personal belief' (rather than the objective state of knowledge in the world)[5]. For subjectivists however, rationality and consistency constrain the probabilities a subject may have, but allow for substantial variation within those constraints. The objective and subjective variants of Bayesian probability differ mainly in their interpretation and construction of the prior probability.

[edit] History

Main article: History of statistics#Bayesian statistics

The term Bayesian refers to Thomas Bayes (1702–1761), who proved a special case of what is now called Bayes' theorem. However, it was Pierre-Simon Laplace (1749–1827) who introduced a general version of the theorem and used it to approach problems in celestial mechanics, medical statistics, reliability, and jurisprudence [7]. Early Bayesian inference, which used uniform priors following Laplace's principle of insufficient reason, was called "inverse probability" (because it infers backwards from observations to parameters, or from effects to causes[8]). After the 1920s, "inverse probability" was largely supplanted by a collection of methods that came to be called frequentist statistics.[8]

In the 20th century, the ideas of Laplace were further developed in two different directions, giving rise to objective and subjective currents in Bayesian practice. In the objectivist stream, the statistical analysis depends on only the model assumed and the data analysed.[9]. No subjective decisions need to be involved. In contrast, "subjectivist" statisticians deny the possibility of fully objective analysis for the general case.

In the 1980s, there was a dramatic growth in research and applications of Bayesian methods, mostly attributed to the discovery of Markov chain Monte Carlo methods, which removed many of the computational problems, and an increasing interest in nonstandard, complex applications.[10] Despite growth of Bayesian research, most undergraduate teaching is still based on frequentist statistics.[11] Nonetheless, Bayesian methods are widely accepted and used, such as for example in the field of machine learning.[6]

[edit] Justification of Bayesian probabilities

The use of Bayesian probabilities as the basis of Bayesian inference has been supported by several arguments, such as the Cox axioms, the Dutch book argument, arguments based on decision theory and de Finetti's theorem.

[edit] Axiomatic approach

Richard T. Cox showed that[4] Bayesian updating follows from several axioms, including two functional equations and the controversial hypothesis that probability should be treated as a continuous function. Here "continuity" is equivalent to countable additivity, as proved in measure-theoretic probability books. The countable additivity requirement is rejected (e.g. for being non-falsifiable) by Bruno de Finetti, for example.

[edit] Dutch book approach

The Dutch book argument was proposed by de Finetti, and is based on betting. A Dutch book is made when a clever gambler places a set of bets that guarantee a profit, no matter what the outcome is of the bets. If a bookmaker follows the rules of the Bayesian calculus in the construction of his odds, a Dutch book cannot be made.

However, Ian Hacking noted that traditional Dutch book arguments did not specify Bayesian updating: they left open the possibility that non-Bayesian updating rules could avoid Dutch books. For example, Hacking writes[12] "And neither the Dutch book argument, nor any other in the personalist arsenal of proofs of the probability axioms, entails the dynamic assumption. Not one entails Bayesianism. So the personalist requires the dynamic assumption in order to be Bayesian. It is true that in consistency a personalist could abandon the Bayesian model of learning from experience. Salt could lose its savour."

In fact, there are non-Bayesian updating rules that also avoid Dutch books (as discussed in the literature on "probability kinematics" following the publication of Richard C. Jeffrey's rule). The additional hypotheses sufficient to (uniquely) specify Bayesian updating are substantial, complicated, and unsatisfactory.[13]

[edit] Decision theory approach

A decision-theoretic justification of the use of Bayesian inference (and hence of Bayesian probabilities) was given by Abraham Wald,[citation needed] who proved that every Bayesian procedure is admissible.[citation needed] Conversely, every admissible statistical procedure is either a Bayesian procedure or a limit of Bayesian procedures.[14]

[edit] Personal probabilities and objective methods for constructing priors

Following the work on expected utility theory of Ramsey and von Neumann, decision-theorists have accounted for rational behavior using a probability distribution for the agent. Johann Pfanzagl completed The Theory of Games and Economic Behavior by providing an axiomatization of subjective probability and utility, a task left uncompleted by von Neumann and Oskar Morgenstern: their original theory supposed that all the agents had the same probability distribution, as a convenience.[15] Pfanzagl's axiomatization was endorsed by Oskar Morgenstern: "Von Neumann and I have anticipated" the question whether probabilities "might, perhaps more typically, be subjective and have stated specifically that in the latter case axioms could be found from which could derive the desired numerical utility together with a number for the probabilities (cf. p. 19 of The Theory of Games and Economic Behavior). We did not carry this out; it was demonstrated by Pfanzagl ... with all the necessary rigor".[16]

Ramsey and Savage noted that the individual agent's probability distribution could be objectively studied in experiments. The role of judgment and disagreement in science has been recognized since Aristotle and even more clearly with Francis Bacon. The objectivity of science lies not in the psychology of individual scientists, but in the process of science and especially in statistical methods, as noted by C. S. Peirce.[citation needed] Recall that the objective methods for falsifying propositions about personal probabilities have been used for a half century, as noted previously. Procedures for testing hypotheses about probabilities (using finite samples) are due to Ramsey (1931) and de Finetti (1931, 1937, 1964, 1970). Both Bruno de Finetti and Frank P. Ramsey acknowledge[citation needed] their debts to pragmatic philosophy, particularly (for Ramsey) to Charles S. Peirce.

The "Ramsey test" for evaluating probability distributions is implementable in theory, and has kept experimental psychologists occupied for a half century.[17] This work demonstrates that Bayesian-probability propositions can be falsified, and so meet an empirical criterion of Charles S. Peirce, whose work inspired Ramsey. (This falsifiability-criterion was popularized by Karl Popper.[citation needed])

Modern work on the experimental evaluation of personal probabilities use the randomization, blinding, and Boolean-decision procedures of the Peirce-Jastrow experiment.[18] Since individuals act according to different probability judgements, these agents' probabilities are "personal" (but amenable to objective study).

Personal probabilities are problematic for science and for some applications where decision-makers lack the knowledge or time to specify an informed probability-distribution (on which they are prepared to act). To meet the needs of science and of human limitations, Bayesian statisticians have developed "objective" methods for specifying prior probabilities.

Indeed, some Bayesians have argued the prior state of knowledge defines the (unique) prior probability-distribution for "regular" statistical problems; cf. well-posed problems. Finding the right method for constructing such "objective" priors (for appropriate classes of regular problems) has been the quest of statistical theorists from Laplace to John Maynard Keynes, Harold Jeffreys, and Edwin Thompson Jaynes: These theorists and their successors have suggested several methods for constructing "objective" priors:

Each of these methods contributes useful priors for "regular" one-parameter problems, and each prior can handle some challenging statistical models (with "irregularity" or several parameters). Each of these methods has been useful in Bayesian practice. Indeed, methods for constructing "objective" (alternatively, "default" or "ignorance") priors have been developed by avowed subjective (or "personal") Bayesians like James Berger (Duke University) and José-Miguel Bernardo (Universitat de València), simply because[citation needed] such priors are needed for Bayesian practice, particularly in science. Each of these methods gives implausible priors for some problems, and so the quest for "the universal method for constructing priors" continues to attract statistical theorists.[citation needed]

Thus, the Bayesian statistican needs either to use informed priors (using relevant expertise or previous data) or to choose among the competing methods for constructing "objective" priors.

[edit] See also

[edit] Footnotes

  1. ^ a b c d ET. Jaynes. Probability Theory: The Logic of Science Cambridge University Press, (2003). ISBN 0-521-59271-2
  2. ^ Stephen M. Stigler (1986) The history of statistics. Harvard University press. pg 131.
  3. ^ Stephen M. Stigler (1986) The history of statistics. Harvard University press. pg 97-98, pg 131.
  4. ^ a b c d Richard T. Cox, Algebra of Probable Inference, The Johns Hopkins University Press, 2001
  5. ^ a b de Finetti, B. (1974) Theory of probability (2 vols.), J. Wiley & Sons, Inc., New York
  6. ^ a b Bishop, CM., Pattern Recognition and Machine Learning. Springer, 2007
  7. ^ Stephen M. Stigler (1986) The history of statistics. Harvard University press. Chapter 3.
  8. ^ a b Stephen. E. Fienberg, (2006) "When did Bayesian Inference become "Bayesian"? Bayesian Analysis, 1 (1), 1–40. See page 5.
  9. ^ JM. Bernardo (2005), "Reference analysis", Handbook of statistics, 25, 17–90
  10. ^ Wolpert, RL. (2004) A conversation with James O. Berger, Statistical science, 9, 205–218
  11. ^ José M. Bernardo (2006) A Bayesian mathematical statistics prior. ICOTS-7
  12. ^ Hacking (1967, Section 3, page 316), Hacking (1988, page 124)
  13. ^ van Frassen, B. (1989) Laws and Symmetries, Oxford University Press. ISBN 0198248601
  14. ^ Bickel & Doksum (2001, page 32)
  15. ^ Pfanzagl (1967, 1968
  16. ^ Morgenstern (1976, page 65)
  17. ^ Davidson et al. (1957)
  18. ^ Pierce & Jastrow (1885)

[edit] References

[edit] External links

Retrieved from "http://en.wikipedia.org/wiki/Bayesian_probability"
Personal tools
Namespaces
Variants
Actions
Navigation
Interaction
Toolbox
Print/export
Languages