Concentration inequality

From Wikipedia, the free encyclopedia
Jump to navigation Jump to search

In probability theory, concentration inequalities provide bounds on how a random variable deviates from some value (typically, its expected value). The laws of large numbers of classical probability theory states that sums of independent random variables are, under very mild conditions, close to their expectation with a large probability. Such sums are the most basic examples of random variables concentrated around their mean. Recent results show that such behavior is shared by other functions of independent random variables.

Concentration inequalities can be sorted according to how much information about the random variable is needed in order to use them.

Markov's inequality[edit]

Let be a random variable that is non-negative (almost surely). Then, for every constant ,

Note the following extension to Markov's inequality: if is a strictly increasing and non-negative function, then

Chebyshev's inequality[edit]

Chebyshev's inequality requires the following information on a random variable :

  • The expected value is finite.
  • The variance is finite.

Then, for every constant ,

or equivalently,

where is the standard deviation of .

Chebyshev's inequality can be seen as a special case of the generalized Markov's inequality applied to the random variable with .

Chernoff bounds[edit]

The generic Chernoff bound[1]:63–65 requires only the moment generating function of , defined as: , provided it exists. Based on Markov's inequality, for every :

and for every :

There are various Chernoff bounds for different distributions and different values of the parameter . See [2]:5-7 for a compilation of more concentration inequalities.

Bounds on sums of independent variables[edit]

Let be independent random variables such that, for all i:

almost surely.

Let be their sum, its expected value and its variance:

It is often interesting to bound the difference between the sum and its expected value. Several inequalities can be used.

1. Hoeffding's inequality says that:

2. The random variable is a special case of a martingale, and . Hence, Azuma's inequality can also be used and it yields a similar bound:

This is a generalization of Hoeffding's since it can handle other types of martingales, as well as supermartingales and submartingales.

3. The sum function, , is a special case of a function of n variables. This function changes in a bounded way: if variable i is changed, the value of f changes by at most . Hence, McDiarmid's inequality can also be used and it yields a similar bound:

This is a different generalization of Hoeffding's since it can handle other functions besides the sum function, as long as they change in a bounded way.

4. Bennett's inequality offers some improvement over Hoeffding's when the variances of the summands are small compared to their almost-sure bounds C. It says that:


5. The first of Bernstein's inequalities says that:

This is a generalization of Hoeffding's since it can handle not only independent variables but also weakly-dependent variables.

6. Chernoff bounds have a particularly simple form in the case of sum of independent variables, since .

For example,[3] suppose the variables satisfy , for . Then we have lower tail inequality:

If satisfies , we have upper tail inequality:

If are i.i.d., and is the variance of , a typical version of Chernoff inequality is:

7. Similar bounds can be found in: Rademacher distribution#Bounds on sums

Efron–Stein inequality[edit]

The Efron–Stein inequality (or influence inequality, or MG bound on variance) bounds the variance of a general function.

Suppose that , are independent with and having the same distribution for all .

Let Then

Dvoretzky–Kiefer–Wolfowitz inequality[edit]

The Dvoretzky–Kiefer–Wolfowitz inequality bounds the difference between the real and the empirical cumulative distribution function.

Given a natural number , let be real-valued independent and identically distributed random variables with cumulative distribution function F(·). Let denote the associated empirical distribution function defined by

So is the probability that a single random variable is smaller than , and is the average number of random variables that are smaller than .



  1. ^ Mitzenmacher, Michael; Upfal, Eli (2005). Probability and Computing: Randomized Algorithms and Probabilistic Analysis. Cambridge University Press. ISBN 0-521-83540-2.
  2. ^ Slagle, N.P. (2012). "One Hundred Statistics and Probability Inequalities" (PDF).
  3. ^ Chung, Fan; Lu, Linyuan (2010). "Old and new concentration inequalities" (PDF). Complex Graphs and Networks. American Mathematical Society. Retrieved August 14, 2018.

External links[edit]

Karthik Sridharan, "A Gentle Introduction to Concentration Inequalities"  —Cornell University