Sub-Gaussian distribution

In probability theory, a sub-Gaussian distribution, the distribution of a sub-Gaussian random variable, is a probability distribution with strong tail decay. More specifically, the tails of a sub-Gaussian distribution are dominated by (i.e. decay at least as fast as) the tails of a Gaussian. This property gives sub-Gaussian distributions their name.

Formally, the probability distribution of a random variable $X$ is called sub-Gaussian if there is a positive constant C such that for every $t\geq 0$,

${\textstyle \operatorname {P} (|X|\geq t)\leq 2\exp {(-t^{2}/C^{2})}}$.

Alternatively, a random variable is considered sub-Gaussian if its distribution function is upper bounded (up to a constant) by the distribution function of a Gaussian. Specifically, we say that $X$ is sub-Gaussian if for all $s\geq 0$ we have that:

$P(|X|\geq s)\leq cP(|Z|\geq s),$

where $c\geq 0$ is constant and $Z$ is a mean zero Gaussian random variable.

Definitions

The sub-Gaussian norm of $X$, denoted as $\Vert X\Vert _{gauss}$, is defined by$$\Vert X\Vert _{gauss}=\inf \left\{c>0:\operatorname {E} \left[\exp {\left({\frac {X^{2}}{c^{2}}}\right)}\right]\leq 2\right\},$$which is the Orlicz norm of $X$ generated by the Orlicz function $\Phi (u)=e^{u^{2}}-1.$ By condition $(2)$ below, sub-Gaussian random variables can be characterized as those random variables with finite sub-Gaussian norm.

Sub-Gaussian properties

Let $X$ be a random variable. The following conditions are equivalent:

1. $\operatorname {P} (|X|\geq t)\leq 2\exp {(-t^{2}/K_{1}^{2})}$ for all $t\geq 0$, where $K_{1}$ is a positive constant;
2. $\operatorname {E} [\exp {(X^{2}/K_{2}^{2})}]\leq 2$, where $K_{2}$ is a positive constant;
3. $\operatorname {E} |X|^{p}\leq 2K_{3}^{p}\Gamma \left({\frac {p}{2}}+1\right)$ for all $p\geq 1$, where $K_{3}$ is a positive constant.

Proof. $(1)\implies (3)$ By the layer cake representation,$${\begin{aligned}\operatorname {E} |X|^{p}&=\int _{0}^{\infty }\operatorname {P} (|X|^{p}\geq t)dt\\&=\int _{0}^{\infty }pt^{p-1}\operatorname {P} (|X|\geq t)dt\\&\leq 2\int _{0}^{\infty }pt^{p-1}\exp \left(-{\frac {t^{2}}{K_{1}^{2}}}\right)dt\\\end{aligned}}$$After a change of variables $u=t^{2}/K_{1}^{2}$, we find that$${\begin{aligned}\operatorname {E} |X|^{p}&\leq 2K_{1}^{p}{\frac {p}{2}}\int _{0}^{\infty }u^{{\frac {p}{2}}-1}e^{-u}du\\&=2K_{1}^{p}{\frac {p}{2}}\Gamma \left({\frac {p}{2}}\right)\\&=2K_{1}^{p}\Gamma \left({\frac {p}{2}}+1\right).\end{aligned}}$$

$(3)\implies (2)$ Using the Taylor series for $e^{x}$:$$e^{x}=1+\sum _{p=1}^{\infty }{\frac {x^{p}}{p!}},$$ we obtain that$${\begin{aligned}\operatorname {E} [\exp {(\lambda X^{2})}]&=1+\sum _{p=1}^{\infty }{\frac {\lambda ^{p}\operatorname {E} {[X^{2p}]}}{p!}}\\&\leq 1+\sum _{p=1}^{\infty }{\frac {2\lambda ^{p}K_{3}^{2p}\Gamma \left(p+1\right)}{p!}}\\&=1+2\sum _{p=1}^{\infty }\lambda ^{p}K_{3}^{2p}\\&=2\sum _{p=0}^{\infty }\lambda ^{p}K_{3}^{2p}-1\\&={\frac {2}{1-\lambda K_{3}^{2}}}-1\quad {\text{for }}\lambda K_{3}^{2}<1,\end{aligned}}$$which is less than or equal to $2$ for $\lambda \leq {\frac {1}{3K_{3}^{2}}}$. Take $K_{2}\geq 3^{\frac {1}{2}}K_{3}$, then$$\operatorname {E} [\exp {(X^{2}/K_{2}^{2})}]\leq 2.$$

$(2)\implies (1)$ By Markov's inequality,$$\operatorname {P} (|X|\geq t)=\operatorname {P} \left(\exp \left({\frac {X^{2}}{K_{2}^{2}}}\right)\geq \exp \left({\frac {t^{2}}{K_{2}^{2}}}\right)\right)\leq {\frac {\operatorname {E} [\exp {(X^{2}/K_{2}^{2})}]}{\exp \left({\frac {t^{2}}{K_{2}^{2}}}\right)}}\leq 2\exp \left(-{\frac {t^{2}}{K_{2}^{2}}}\right).$$

More equivalent definitions

The following properties are equivalent:

• The distribution of $X$ is sub-Gaussian.
• Laplace transform condition: for some B, b > 0, $\operatorname {E} e^{\lambda (X-\operatorname {E} [X])}\leq Be^{\lambda ^{2}b}$ holds for all $\lambda$.
• Moment condition: for some K > 0, $\operatorname {E} |X|^{p}\leq K^{p}p^{p/2}$ for all $p\geq 1$.
• Moment generating function condition: for some $L>0$, $\operatorname {E} [\exp(\lambda ^{2}X^{2})]\leq \exp(L^{2}\lambda ^{2})$ for all $\lambda$ such that $|\lambda |\leq {\frac {1}{L}}$.
• Union bound condition: for some c > 0, $\ \operatorname {E} [\max\{|X_{1}-\operatorname {E} [X]|,\ldots ,|X_{n}-\operatorname {E} [X]|\}]\leq c{\sqrt {\log n}}$ for all n > c, where $X_{1},\ldots ,X_{n}$ are i.i.d copies of X.

Examples

A standard normal random variable $X\sim N(0,1)$ is a sub-Gaussian random variable.

Let $X$ be a random variable with symmetric Bernoulli distribution (or Rademacher distribution). That is, $X$ takes values $-1$ and $1$ with probabilities $1/2$ each. Since $X^{2}=1$, it follows that $$\Vert X\Vert _{gauss}=\inf \left\{c>0:\operatorname {E} \left[\exp {\left({\frac {X^{2}}{c^{2}}}\right)}\right]\leq 2\right\}=\inf \left\{c>0:\operatorname {E} \left[\exp {\left({\frac {1}{c^{2}}}\right)}\right]\leq 2\right\}={\frac {1}{\sqrt {\ln 2}}},$$and hence $X$ is a sub-Gaussian random variable.

Maximum of Sub-Gaussian Random Variables

Consider a finite collection of subgaussian random variables, X₁, ..., X_n, with corresponding subgaussian parameters $\sigma$. The random variable M_n = max(X₁, ..., X_n) represents the maximum of this collection. The expectation $M_{n}$ can be bounded above by ${\sqrt {2\sigma ^{2}\log n}}$. Note that no independence assumption is needed to form this bound.

See also

• Platykurtic distribution

References

• Kahane, J.P. (1960). "Propriétés locales des fonctions à séries de Fourier aléatoires". Studia Mathematica. 19: 1–25. doi:10.4064/sm-19-1-1-25.
• Buldygin, V.V.; Kozachenko, Yu.V. (1980). "Sub-Gaussian random variables". Ukrainian Mathematical Journal. 32 (6): 483–489. doi:10.1007/BF01087176.
• Ledoux, Michel; Talagrand, Michel (1991). Probability in Banach Spaces. Springer-Verlag.
• Stromberg, K.R. (1994). Probability for Analysts. Chapman & Hall/CRC.
• Litvak, A.E.; Pajor, A.; Rudelson, M.; Tomczak-Jaegermann, N. (2005). "Smallest singular value of random matrices and geometry of random polytopes" (PDF). Advances in Mathematics. 195 (2): 491–523. doi:10.1016/j.aim.2004.08.004.
• Rudelson, Mark; Vershynin, Roman (2010). "Non-asymptotic theory of random matrices: extreme singular values". Proceedings of the International Congress of Mathematicians 2010. pp. 1576–1602. arXiv:1003.2990. doi:10.1142/9789814324359_0111.
• Rivasplata, O. (2012). "Subgaussian random variables: An expository note" (PDF). Unpublished.
• Vershynin, R. (2018). "High-dimensional probability: An introduction with applications in data science" (PDF). Volume 47 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge.
• Zajkowskim, K. (2020). "On norms in some class of exponential type Orlicz spaces of random variables". Positivity. An International Mathematics Journal Devoted to Theory and Applications of Positivity. 24(5): 1231--1240. arXiv:1709.02970. doi.org/10.1007/s11117-019-00729-6.