Bayes' rule

This redirect is about the form of Bayes' theorem. For the decision rule, see Bayes estimator. For the use of Bayes factor in model selection, see Bayes factor.

In probability theory and applications, Bayes' rule relates the odds of event $A_{1}$ to event $A_{2}$, before (prior to) and after (posterior to) conditioning on another event $B$. The odds on $A_{1}$ to event $A_{2}$ is simply the ratio of the probabilities of the two events. The prior odds is the ratio of the unconditional or prior probabilities, the posterior odds is the ratio of conditional or posterior probabilities given the event $B$. The relationship is expressed in terms of the likelihood ratio or Bayes factor, $\Lambda$. By definition, this is the ratio of the conditional probabilities of the event $B$ given that $A_{1}$ is the case or that $A_{2}$ is the case, respectively. The rule simply states: posterior odds equals prior odds times Bayes factor.

Bayes' rule is an equivalent way to formulate Bayes' theorem. Bayes' rule may be preferred to Bayes' theorem in practice since it is intuitively simpler to understand, and because normalizing probabilities is sometimes unnecessary (one sometimes only needs to know ratios of probabilities) or easy to do afterwards, anyway. If we know the odds for and against $A$ we also know the probabilities of $A$. Bayes rule is widely used in statistics, science and engineering, for instance in model selection, probabilistic expert systems, probabilistic reasoning in courts of law, and so on.

Bayes's rule, as an elementary fact from the calculus of probability, tells us how unconditional and conditional probabilities are related whether we work with a frequentist interpretation of probability or a Bayesian interpretation of probability. Under the Bayesian interpretation it is frequently applied in the situation where $A_{1}$ and $A_{2}$ are competing hypotheses, and $B$ is some observed evidence. The rule shows how one's judgement on the whether $A_{1}$ or $A_{2}$ is true should be updated on observing the evidence $B$.

The rule

Single event

Given events $A_{1}$, $A_{2}$ and $B$, Bayes' rule states that the conditional odds of $A_{1}:A_{2}$ given $B$ are equal to the marginal odds of $A_{1}:A_{2}$ multiplied by the Bayes factor or likelihood ratio $\Lambda$:

$O(A_{1}:A_{2}|B)=\Lambda (A_{1}:A_{2}|B)\cdot O(A_{1}:A_{2}),$

where

$\Lambda (A_{1}:A_{2}|B)={\frac {P(B|A_{1})}{P(B|A_{2})}}.$

Here, the odds and conditional odds, also known as prior odds and posterior odds, are defined by

$O(A_{1}:A_{2})={\frac {P(A_{1})}{P(A_{2})}},$

$O(A_{1}:A_{2}|B)={\frac {P(A_{1}|B)}{P(A_{2}|B)}}.$

In the special case that $A_{1}=A$ and $A_{2}=\neg A$, one writes $O(A)=O(A:\neg A)$, and uses a similar abbreviation for the Bayes factor and for the conditional odds. The odds on $A$ is by definition the odds for and against $A$. Bayes rule can then be written in the abbreviated form

$O(A|B)=\Lambda (A|B)\cdot O(A),$

or in words: the posterior odds on $A$ equals the prior odds on $A$ times the Bayes factor for $A$ given information $B$.

Multiple events

Bayes' rule may be conditioned on an arbitrary number of events. For two events $B$ and $C$,

$O(A_{1}:A_{2}|B\cap C)=\Lambda (A_{1}:A_{2}|B\cap C)\cdot \Lambda (A_{1}:A_{2}|B)\cdot O(A_{1}:A_{2}),$

where

$\Lambda (A_{1}:A_{2}|B)={\frac {P(B|A_{1})}{P(B|A_{2})}},$

$\Lambda (A_{1}:A_{2}|B\cap C)={\frac {P(C|A_{1}\cap B)}{P(C|A_{2}\cap B)}}.$

In this special case, the equivalent notation is

$O(A|B,C)=\Lambda (A|B\cap C)\cdot \Lambda (B|A)\cdot O(A).$

Derivation

Consider two instances of Bayes' theorem:

$P(A_{1}|B)={\frac {1}{P(B)}}\cdot P(B|A_{1})\cdot P(A_{1}),$

$P(A_{2}|B)={\frac {1}{P(B)}}\cdot P(B|A_{2})\cdot P(A_{2}).$

Combining these gives

${\frac {P(A_{1}|B)}{P(A_{2}|B)}}={\frac {P(B|A_{1})}{P(B|A_{2})}}\cdot {\frac {P(A_{1})}{P(A_{2})}}.$

Now defining

$O(A_{1}:A_{2}|B)\triangleq {\frac {P(A_{1}|B)}{P(A_{2}|B)}}$

$O(A_{1}:A_{2})\triangleq {\frac {P(A_{1})}{P(A_{2})}}$

$\Lambda (A_{1}:A_{2}|B)\triangleq {\frac {P(B|A_{1})}{P(B|A_{2})}},$

this implies

$O(A_{1}:A_{2}|B)=\Lambda (A_{1}:A_{2}|B)\cdot O(A_{1}:A_{2}).$

A similar derivation applies for conditioning on multiple events, using the appropriate extension of Bayes' theorem

Examples

Frequentist example

Consider the drug testing example in the article on Bayes' theorem.

The same results may be obtained using Bayes' rule. The prior odds on an individual being a drug-user are 199 to 1 against, as $\textstyle 0.5\%={\frac {1}{200}}$ and $\textstyle 99.5\%={\frac {199}{200}}$. The Bayes factor when an individual tests positive is $\textstyle {\frac {0.99}{0.01}}=99:1$ in favour of being a drug-user: this is the ratio of the probability of a drug-user testing positive, to the probability of a non-drug user testing positive. The posterior odds on being a drug user are therefore $\textstyle 1\times 99:199\times 1=99:199$, which is very close to $\textstyle 100:200=1:2$. In round numbers, only one in three of those testing positive are actually drug-users.

Model selection

Main article: Bayesian model selection

External links

• The on-line textbook: Information Theory, Inference, and Learning Algorithms, by David J.C. MacKay, discusses Bayesian model comparison in Chapters 3 and 28.