Probability theory

Probability theory is a branch of mathematics concerned with analysis of random phenomena. The central objects of probability theory are random variables and stochastic processes: mathematical abstractions of non-deterministic events or measured quantities that may either be single occurrences or evolve over time in an apparently random fashion. As a mathematical foundation for statistics, probability theory is essential to most human activities that involve quantitative analysis of large sets of data. Methods of probability theory also apply to description of complex systems given only partial knowledge of their state, as in statistical mechanics. A great discovery of twentieth century physics was the probabilistic nature of physical phenomena at microscopic scales, described in quantum mechanics.

History

The mathematical theory of probability has its roots in attempts to analyse games of chance by Pierre de Fermat and Blaise Pascal in the seventeenth century. Although an individual coin toss or the roll of a die is random event, if repeated many times the sequence of random events will exhibit certain statistical patterns, which can be studied and predicted. Two representative mathematical results describing such patterns are the law of large numbers and the central limit theorem.

Initially, probability theory mainly considered discrete events, and its methods were mainly combinatorial. Eventually, analytical considerations compelled the incorporation of continuous variables into the theory. This culminated in modern probability theory, the foundations of which were laid by Andrey Nikolaevich Kolmogorov. Kolmogorov combined the notion of probability space, introduced by Richard von Mises, and measure theory and presented his axiom system for probability theory 1933. Fairly quickly this became the undisputed axiomatic basis for modern probability theory.

Treatment

Discrete probability theory

Discrete probability theory deals with events which occur in countable sample spaces.

Example: Throwing a dice, experiments with a deck of cards, random walk.

Classical definition: Initially the probability of an event to occur was defined as number of cases favorable for the event, over the number of total possible outcomes possible.

For example, if the event is occurrence of an even number when a die is rolled, the probability is given by ${\tfrac {3}{6}}={\tfrac {1}{2}}$, since 3 faces out of the 6 have even numbers.

Modern definition: The modern definition starts with a set called the sample space which relates to the set of all possible outcomes in classical sense, denoted by $\Omega =\left\{x_{1},x_{2},\dots \right\}$. It is then assumed that for each element $x\in \Omega \,$, an intrinsic "probability" value $f(x)\,$ is attached, which satisfies the following properties:

1. $f(x)\in [0,1]{\mbox{ for all }}x\in \Omega$
2. $\sum _{x\in \Omega }f(x)=1$

An event is defined as any subset $E\,$ of the sample space $\Omega \,$. The probability of the event $E\,$ defined as

$\Pr(E)=\sum _{x\in E}f(x)\,$

So, the probability of the entire sample space is 1, and the probability of the null event is 0.

The function $f(x)\,$ mapping a point in the sample space to the "probability" value is called a probability mass function abbreviated as pmf. The modern definition does not try to answer how probability mass functions are obtained; instead it builds a theory that assumes their existence.

Continuous probability theory

Continuous probability theory deals with events which occur in a continuous sample space.

If the sample space is the real numbers, then a function called the cumulative distribution function or cdf $F\,$ is assumed to exist, which gives $F(x)=\Pr(X\leq x)\,$.

The cdf must satisfy the following properties.

1. $F\,$ is a monotonically non-decreasing right-continuous function
2. $\lim _{x\rightarrow -\infty }F(x)=0$
3. $\lim _{x\rightarrow \infty }F(x)=1$

If $F\,$ is differentiable, then the random variable is said to have a probability density function or pdf or simply density $f(x)={\frac {dF(x)}{dx}}\,$.

For a set $E\subseteq \mathbb {R}$, the probability of the random variable being in $E\,$ is defined as

$\Pr(X\in E)=\int _{x\in E}dF(x)\,$

In case the density exists, then it can be written as

$\Pr(X\in E)=\int _{x\in E}f(x)\,dx$

These concepts can be generalized for multidimensional cases on $\mathbb {R} ^{n}$.

Measure theoretic probability theory

Certain distributions can be a mix of discrete and continuous distributions. For example, a sum of a discrete and continuous random variable will neither have a pmf nor a pdf. Other distributions may not even be a mix; for example the Cantor distribution which has no point mass and no density. The modern approach to probability theory solves these problems using measure theory and the definition of the probability space:

Given a sample space $X\,$ and a σ-algebra $\Sigma \,$ on it, a measure $\mu \,$ is called a probability measure if

1. $\mu \,$ is non-negative
2. $\mu (X)=1\,$

For any cdf there is a unique probability measure on the Borel sigma field, and vice versa. The measure corresponding to a cdf is said to be induced by the cdf. This measure coincides with the pmf for discrete variables, and pdf for continuous variables, making the measure theoretic approach free of fallacies.

The probability of a set $E\,$ in the sigma field $\Sigma \,$ is defined as

$\Pr(X\in E)=\int _{x\in E}dF(x)\,$

where the integration is with respect to the measure induced by $F\,$.

Along with providing better understanding, measure theoretic treatment also allows us to work on probabilities outside $\mathbb {R} ^{n}$. For example to study Brownian motion, probability is defined on a space of functions.

Probability distributions

Main article: Probability distributions

Certain random variables occur very often in probability theory due to many natural and physical processes. Their distributions therefore have gained special importance in probability theory. These probability distributions have been given special names to easily identify and study them.

Some fundamental discrete distributions are the discrete uniform, Bernoulli, binomial, negative binomial, Poisson and geometric distributions.

Some fundamental continuous distributions are thecontinuous uniform, normal, exponential, gamma and beta distributions.

Convergence of random variables

Main article: Convergence of random variables

In probability theory, there are several notions of convergence for random variables. They are listed below in the order of strength, i.e. any later notion convergence in the list implies convergence according to all the preceding notions.

Convergence in distribution: As the name implies, a sequence of random variables $X_{1},X_{2},\dots ,\,$ converges to the random variable $X\,$ in distribution if their respective cumulative distribution functions $F_{1},F_{2},\dots \,$ converge to the cumulative distribution function $F\,$ of $X\,$, wherever $F\,$ is continuous.

Weak convergence: The sequence of random variables $X_{1},X_{2},\dots \,$ is said to converge towards the random variable $X\,$ weakly if $\lim _{n\rightarrow \infty }P\left(\left|X_{n}-X\right|\geq \varepsilon \right)=0$ for every ε > 0. Weak convergence is also called convergence in probability.

Strong convergence: The sequence of random variables $X_{1},X_{2},\dots \,$ is said to converge towards the random variable $X\,$ strongly if $\Pr(\lim _{n\rightarrow \infty }X_{n}=X)=1.$ Strong convergence is also known as almost sure convergence.

Intuitively, strong convergence is a technically stronger version of the weak convergence, in both cases the random variables $X_{1},X_{2},\dots \,$ show an increasing correlation with $X\,$. However in case of convergence in distribution, the realized values of the random variables do not need to converge, and any possible correlation among them is immaterial.

Law of large numbers

Main article: Law of large numbers

If a fair coin is tossed, we know that roughly half of the time it will turn up heads, and the other half it will turn up tails. It also seems that the more we toss it, the more likely it is that the ratio of heads:tails will approach 1:1. Modern probability allows us to formally arrive at the same result, dubbed the law of large numbers. This result is remarkable because it was nowhere assumed while building the theory and is completely an offshoot of the theory. Linking theoretically-derived probabilities to their actual frequency of occurrence in the real world, this result is considered as a pillar in the history of statistical theory.[1]

The strong law of large numbers (SLLN) states that if an event of probability p is observed repeatedly during independent experiments, the ratio of the observed frequency of that event to the total number of repetitions converges towards p strongly in probability.

In other words, if $X_{1},X_{2},...\,$ are independent Bernoulli random variables taking values 1 with probability p and 0 with probability 1-p, then the sequence of random numbers ${\frac {\sum X_{n}}{n}}\,$ converges to p almost surely, i.e.

$\Pr \left(\lim _{n\rightarrow \infty }{\frac {\sum X_{n}}{n}}=p\right)=1.\,$

Central limit theorem

Main article: Central limit theorem

The central limit theorem is the reason for the ubiquitous occurrence of the normal distribution in nature, for which it is one of the most celebrated theorems in probability and statistics.

The theorem states that the average of many independent and identically distributed random variables tends towards a normal distribution irrespective of which distribution the original random variables follow. Formally, let $X_{1},X_{2},\dots \,$ be independent random variables with means $\mu _{1},\mu _{2},\dots \,$, and variances $\sigma _{1}^{2},\sigma _{2}^{2},\dots ,\,$ then the sequence of random variables

$Z_{n}={\frac {\sum _{i=1}^{n}(X_{i}-\mu _{i})}{\sqrt {\sum _{i=1}^{n}\sigma _{i}^{2}}}}$

converges in distribution to a standard normal random variable.

See also

• glossary of probability and statistics
• list of probability topics
• list of statistical topics
• list of publications in statistics
• probabilistic logic
• predictive modelling
• fuzzy measure theory
• expected value
• likelihood function
• variance
• statistical independence
• notation in probability

Bibliography

• Pierre Simon de Laplace (1812) Analytical Theory of Probability

The first major treatise blending calculus with probability theory, originally in French: Théorie Analytique des Probabilités.

• Andrei Nikolajevich Kolmogorov (1950) Foundations of the Theory of Probability

The modern measure-theoretic foundation of probability theory; the original German version (Grundbegriffe der Wahrscheinlichkeitrechnung) appeared in 1933.

• Harold Jeffreys (1939) The Theory of Probability

An empiricist, Bayesian approach to the foundations of probability theory.

• Edward Nelson (1987) Radically Elementary Probability Theory

Discrete foundations of probability theory, based on nonstandard analysis and internal set theory. downloadable. http://www.math.princeton.edu/~nelson/books.html

• Patrick Billingsley: Probability and Measure, John Wiley and Sons, New York, Toronto, London, 1979.

• Henk Tijms (2004) Understanding Probability

A lively introduction to probability theory for the beginner, Cambridge Univ. Press.

• Gut, Allan (2005). Probability: A Graduate Course. Springer. ISBN 0387228330.