In probability theory, Hoeffding's inequality provides an upper bound on the probability that the sum of random variables deviates from its expected value. Hoeffding's inequality was proved by Wassily Hoeffding in 1963.
Hoeffding's inequality is a special case of the Azuma–Hoeffding inequality, and it is more general than the Bernstein inequality, proved by Sergei Bernstein in 1923. They are also special cases of McDiarmid's inequality.
Special case of Bernoulli random variables
Hoeffding's inequality can be applied to the important special case of identically distributed Bernoulli random variables, and this is how the inequality is often used in combinatorics and computer science. We consider a coin that shows heads with probability and tails with probability . We toss the coin times. The expected number of times the coin comes up heads is . Furthermore, the probability that the coin comes up heads at most times can be exactly quantified by the following expression:
In the case that for some , Hoeffding's inequality bounds this probability by a term that is exponentially small in :
Similarly, in the case that for some , Hoeffding's inequality bounds the probability that we see at least more tosses that show heads than we would expect:
Hence Hoeffding's inequality implies that the number of heads that we see is concentrated around its mean, with exponentially small tail.
We define the empirical mean of these variables
Theorem 2 of Hoeffding (1963) proves the inequalities
which are valid for positive values of t. Here is the expected value of . The inequalities can be also stated in terms of the sum
of the random variables:
Note that the inequalities also hold when the have been obtained using sampling without replacement; in this case the random variables are not independent anymore. A proof of this statement can be found in Hoeffding's paper. For slightly better bounds in the case of sampling without replacement, see for instance the paper by Serfling (1974).
- Serfling, Robert J. (1974). "Probability Inequalities for the Sum in Sampling without Replacement". The Annals of Statistics 2 (1): 39–48. doi:10.1214/aos/1176342611.
- Hoeffding, Wassily (March 1963). "Probability inequalities for sums of bounded random variables". Journal of the American Statistical Association 58 (301): 13–30. JSTOR 2282952.