Dirichlet distribution

Dirichlet

Probability density function
Parameters	$K >= 2$ number of categories (integer) $\alpha_1, \ldots, \alpha_K$ concentration parameters, where $\alpha_i > 0$
Support	$x_1, \ldots, x_K$ where $x_i \in [0,1]$ and $\sum x_i = 1$
pdf	$\frac{1}{\mathrm{B}(\boldsymbol\alpha)} \prod_{i=1}^K x_i^{\alpha_i - 1}$ where $\mathrm{B}(\boldsymbol\alpha) = \frac{\prod_{i=1}^K \Gamma(\alpha_i)}{\Gamma\bigl(\sum_{i=1}^K \alpha_i\bigr)}$ where $\boldsymbol\alpha=(\alpha_1,\ldots,\alpha_K)$
Mean	$\operatorname{E}[X_i] = \frac{\alpha_i}{\sum_k \alpha_k}$ $\operatorname{E}[\ln X_i] = \psi(\alpha_i)-\psi(\textstyle\sum_k \alpha_k)$ (see digamma function)
Mode	$x_i = \frac{\alpha_i - 1}{\sum_{i=1}^K\alpha_i - K}, \quad \alpha_i > 1.$
Variance	$\mathrm{Var}[X_i] = \frac{\alpha_i (\alpha_0-\alpha_i)}{\alpha_0^2 (\alpha_0+1)},$ where $\alpha_0 = \sum_{i=1}^k\alpha_i$ $\mathrm{Cov}[X_i,X_j] = \frac{- \alpha_i \alpha_j}{\alpha_0^2 (\alpha_0+1)}~~(i\neq j)$
Entropy	see text

In probability and statistics, the Dirichlet distribution (after Peter Gustav Lejeune Dirichlet), often denoted $\operatorname{Dir}(\boldsymbol\alpha)$, is a family of continuous multivariate probability distributions parametrized by a vector $\boldsymbol\alpha$ of positive reals. It is the multivariate generalization of the beta distribution. Dirichlet distributions are very often used as prior distributions in Bayesian statistics, and in fact the Dirichlet distribution is the conjugate prior of the categorical distribution and multinomial distribution. That is, its probability density function returns the belief that the probabilities of K rival events are $x_i$ given that each event has been observed $\alpha_i-1$ times.

The infinite-dimensional generalization of the Dirichlet distribution is the Dirichlet process.

Probability density function

Illustrating how the log of the density function changes when K=3 as we change the vector α from α=(0.3, 0.3, 0.3) to (2.0, 2.0, 2.0), keeping all the individual $\alpha_i$'s equal to each other.

The Dirichlet distribution of order K ≥ 2 with parameters α₁, ..., α_K > 0 has a probability density function with respect to Lebesgue measure on the Euclidean space R^K-1 given by

$f(x_1,\dots, x_{K-1}; \alpha_1,\dots, \alpha_K) = \frac{1}{\mathrm{B}(\alpha)} \prod_{i=1}^K x_i^{\alpha_i - 1}$

for all x₁, ..., x_K–1 > 0 satisfying x₁ + ... + x_K–1 < 1, and where x_K = 1 – x₁ – ... – x_K–1. The density is zero outside this open (K − 1)-dimensional simplex.

The normalizing constant is the multinomial Beta function, which can be expressed in terms of the gamma function:

$\mathrm{B}(\alpha) = \frac{\prod_{i=1}^K \Gamma(\alpha_i)}{\Gamma\bigl(\sum_{i=1}^K \alpha_i\bigr)},\qquad\alpha=(\alpha_1,\dots,\alpha_K).$

Support

The support of the Dirichlet distribution is the set of $K$-dimensional vectors $\boldsymbol x$ whose entries are real numbers in the interval $(0,1)$; furthermore, $||\boldsymbol x||_1=1$, i.e. the sum of the coordinates is 1. These can be viewed as the probabilities of a K-way categorical event. Another way to express this is that the domain of the Dirichlet distribution is itself a probability distribution, specifically a $K$-dimensional discrete distribution. Note that the technical term for the set of points in the support of a $K$-dimensional Dirichlet distribution is the open standard $K-1$-simplex, which is a generalization of a triangle, embedded in the next-higher dimension. For example, with $K=3$, the support looks like an equilateral triangle embedded in a downward-angle fashion in three-dimensional space, with vertices at $(1,0,0), (0,1,0)$ and $(0,0,1)$, i.e. touching each of the coordinate axes at a point 1 unit away from the origin.

Special cases

A very common special case is the symmetric Dirichlet distribution, where all of the elements making up the parameter vector $\boldsymbol\alpha$ have the same value. Symmetric Dirichlet distributions are often used when a Dirichlet prior is called for, since there typically is no prior knowledge favoring one component over another. Since all elements of the parameter vector have the same value, the distribution alternatively can be parametrized by a single scalar value $\alpha$, called the concentration parameter^{[citation needed]}. The density function then simplifies to

$f(x_1,\dots, x_{K-1}; \alpha) = \frac{\Gamma(\alpha K)}{\Gamma(\alpha)^K} \prod_{i=1}^K x_i^{\alpha - 1}.$

When $\alpha=1$, the symmetric Dirichlet distribution is equivalent to a uniform distribution over the open standard $K-1$-simplex, i.e. it is uniform over all points in its support. Values of the concentration parameter above 1 prefer variates that are dense, evenly distributed distributions, i.e. all the values within a single sample are similar to each other. Values of the concentration parameter below 1 prefer sparse distributions, i.e. most of the values within a single sample will be close to 0, and the vast majority of the mass will be concentrated in a few of the values.

More generally, parameter vector is sometimes written as the product $\alpha \boldsymbol n$ of a (scalar) concentration parameter $\alpha$ and a (vector) base measure $\boldsymbol n=(n_1,\dots,n_K)$ where $\boldsymbol n$ lies within the K-1 simplex (i.e.: its coordinates $n_i$ sum to one). The concentration parameter in this case is larger by a factor of K than the concentration parameter for a symmetric Dirichlet distribution described above. This construction ties in with concept of a base measure when discussing Dirichlet processes and is often used in the topic modelling literature.

^ If we define the concentration parameter as the sum of the Dirichlet parameters for each dimension, the Dirichlet distribution is uniform with a concentration parameter is K, the dimension of the distribution.

Properties

Moments

Let $X = (X_1, \ldots, X_K)\sim\operatorname{Dir}(\alpha)$, meaning that the first K – 1 components have the above density and

$X_K=1-X_1-\cdots-X_{K-1}.$

Define $\textstyle\alpha_0 = \sum_{i=1}^K\alpha_i$. Then

$\mathrm{E}[X_i] = \frac{\alpha_i}{\alpha_0},$

$\mathrm{Var}[X_i] = \frac{\alpha_i (\alpha_0-\alpha_i)}{\alpha_0^2 (\alpha_0+1)}.$

Furthermore, if $i\neq j$

$\mathrm{Cov}[X_i,X_j] = \frac{- \alpha_i \alpha_j}{\alpha_0^2 (\alpha_0+1)}.$

(Note that the matrix so defined is singular.).

Mode

The mode of the distribution is^{[citation needed]} the vector (x₁, ..., x_K) with

$x_i = \frac{\alpha_i - 1}{\alpha_0 - K}, \quad \alpha_i > 1.$

Marginal distributions

The marginal distributions are beta distributions:

$X_i \sim \operatorname{Beta}(\alpha_i, \left(\sum_{j=1}^K{\alpha_j}\right) - \alpha_i).$

Conjugate to categorical/multinomial

The Dirichlet distribution is the conjugate prior distribution of the categorical distribution (a generic discrete probability distribution with a given number of possible outcomes) and multinomial distribution (the distribution over observed counts of each possible category in a set of categorically distributed observations). This means that if a data point has either a categorical or multinomial distribution, and the prior distribution of the data point's parameter (the vector of probabilities that generates the data point) is distributed as a Dirichlet, then the posterior distribution of the parameter is also a Dirichlet. Intuitively, in such a case, starting from what we know about the parameter prior to observing the data point, we then can update our knowledge based on the data point and end up with a new distribution of the same form as the old one. This means that we can successively update our knowledge of a parameter by incorporating new observations one at a time, without running into mathematical difficulties.

Formally, this can be expressed as follows. Given a model

$\begin{array}{lclcl} \boldsymbol\alpha &=& (\alpha_1, \ldots, \alpha_K) &=& \text{concentration hyperparameter} \\ \mathbf{p}\mid\boldsymbol\alpha &=& (p_1, \ldots, p_K) &\sim& \operatorname{Dir}(K, \boldsymbol\alpha) \\ \mathbb{X}\mid\mathbf{p} &=& (\mathbf{x}_1, \ldots, \mathbf{x}_N) &\sim& \operatorname{Cat}(K,\mathbf{p}) \end{array}$

then the following holds:

$\begin{array}{lclcl} \mathbf{c} &=& (c_1, \ldots, c_K) &=& \text{number of occurrences of category }i \\ \mathbf{p} \mid \mathbb{X},\boldsymbol\alpha &\sim& \operatorname{Dir}(K,\mathbf{c}+\boldsymbol\alpha) &=& \operatorname{Dir}(K,c_1+\alpha_1,\ldots,c_K+\alpha_K) \end{array}$

This relationship is used in Bayesian statistics to estimate the underlying parameter p of a categorical distribution given a collection of N samples. Intuitively, we can view the hyperprior vector α as pseudocounts, i.e. as representing the number of observations in each category that we have already seen. Then we simply add in the counts for all the new observations (the vector c) in order to derive the posterior distribution.

In Bayesian mixture models and other hierarchical Bayesian models with mixture components, Dirichlet distributions are commonly used as the prior distributions for the categorical variables appearing in the models. See the section on applications below for more information.

Relation to Dirichlet-multinomial distribution

In a model where a Dirichlet prior distribution is placed over a set of categorical-valued observations, the marginal joint distribution of the observations (i.e. the joint distribution of the observations, with the prior parameter marginalized out) is a Dirichlet-multinomial distribution. This distribution plays an important role in hierarchical Bayesian models, because when doing inference over such models using methods such as Gibbs sampling or variational Bayes, Dirichlet prior distributions are often marginalized out. See the article on this distribution for more details.

Entropy

If X is a Dir(α) random variable, then the exponential family differential identities can be used to get an analytic expression for the expectation of $\log(X_i)$ and its associated covariance matrix:^{[citation needed]}

$\operatorname{E}[\log(X_i)] = \psi(\alpha_i)-\psi(\alpha_0)$

and

$\operatorname{Cov}[\log(X_i),\log(X_j)] = \psi'(\alpha_i) \delta_{ij} - \psi'(\alpha_0)$

where $\psi$ is the digamma function, $\psi'$ is the trigamma function, and $\delta_{ij}$ is the Kronecker delta. The formula for $\operatorname{E}[\log(X_i)]$ yields the following formula for the information entropy of X:^{[citation needed]}

$H(X) = \log \mathrm{B}(\alpha) + (\alpha_0-K)\psi(\alpha_0) - \sum_{j=1}^K (\alpha_j-1)\psi(\alpha_j)$

Aggregation

If $X = (X_1, \ldots, X_K)\sim\operatorname{Dir}(\alpha_1,\ldots,\alpha_K)$ then, if the random variables with subscripts i and j are dropped from the vector and replaced by their sum,

$X' = (X_1, \ldots, X_i + X_j, \ldots, X_K)\sim\operatorname{Dir}(\alpha_1,\ldots,\alpha_i+\alpha_j,\ldots,\alpha_K).$

This aggregation property may be used to derive the marginal distribution of $X_i$ mentioned above.

Neutrality

Main article: Neutral vector

If $X = (X_1, \ldots, X_K)\sim\operatorname{Dir}(\alpha)$, then the vector $X$ is said to be neutral in the sense that $X_K$ is independent of $X^{(-K)}$ where

$X^{(-K)}=\left(\frac{X_1}{1-X_K},\frac{X_2}{1-X_K},\ldots,\frac{X_{K-1}}{1-X_K}\right),$

and similarly for removing any of $X_2,\ldots,X_{K-1}$. Observe that any permutation of $X$ is also neutral (a property not possessed by samples drawn from a generalized Dirichlet distribution.)

Related distributions

If, for $i\in\{1,2,\ldots,K\},$

$Y_i\sim\operatorname{Gamma}(\textrm{shape}=\alpha_i,\textrm{scale}=\theta)\text{ independently},$

then

$V=\sum_{i=1}^K Y_i\sim\operatorname{Gamma}(\textrm{shape}=\sum_{i=1}^K\alpha_i,\textrm{scale}=\theta),$

and

$X = (X_1,\ldots,X_K) = (Y_1/V,\ldots,Y_K/V)\sim \operatorname{Dir}(\alpha_1,\ldots,\alpha_K).$

Although the X_is are not independent from one another, they can be seen to be generated from a set of $K$ independent gamma random variables (see for proof). Unfortunately, since the sum $V$ is lost in forming X, it is not possible to recover the original gamma random variables from these values alone. Nevertheless, because independent random variables are simpler to work with, this reparametrization can still be useful for proofs about properties of the Dirichlet distribution.

Applications

Dirichlet distributions are most commonly used as the prior distribution of categorical variables or multinomial variables in Bayesian mixture models and other hierarchical Bayesian models. (Note that in many fields, such as in natural language processing, categorical variables are often imprecisely called "multinomial variables". Such a usage is liable to cause confusion, just as if Bernoulli distributions and binomial distributions were commonly conflated.)

Inference over hierarchical Bayesian models is often done using Gibbs sampling, and in such a case, instances of the Dirichlet distribution are typically marginalized out of the model by integrating out the Dirichlet random variable. This causes the various categorical variables drawn from the same Dirichlet random variable to become correlated, and the joint distribution over them assumes a Dirichlet-multinomial distribution, conditioned on the hyperparameters of the Dirichlet distribution (the concentration parameters). One of the reasons for doing this is that Gibbs sampling of the Dirichlet-multinomial distribution is extremely easy; see that article for more information.

Random number generation

Gamma distribution

With a source of Gamma-distributed random variates, one can easily sample a random vector $x=(x_1, \ldots, x_K)$ from the K-dimensional Dirichlet distribution with parameters $(\alpha_1, \ldots, \alpha_K)$ . First, draw K independent random samples $y_1, \ldots, y_K$ from Gamma distributions each with density

$\textrm{Gamma}(\alpha_i, 1) = \frac{y_i^{\alpha_i-1} \; e^{-y_i}}{\Gamma (\alpha_i)}, \!$

and then set

$x_i = y_i/\sum_{j=1}^K y_j. \!$

Below is example python code to draw the sample:

params = [a1, a2, ..., ak]
sample = [random.gammavariate(a,1) for a in params]
sample = [v/sum(sample) for v in sample]

Marginal beta distributions

A less efficient algorithm relies on the univariate marginal and conditional distributions being beta and proceeds as follows. Simulate $x_1$ from a $\textrm{Beta}(\alpha_1, \sum_{i=2}^K \alpha_i)$ distribution. Then simulate $x_2, \ldots, x_{K-1}$ in order, as follows. For $j=2, \ldots, K-1$, simulate $\phi_j$ from a $\textrm{Beta}(\alpha_j, \sum_{i=j+1}^K \alpha_i)$ distribution, and let $x_j=(1-\sum_{i=1}^{j-1} x_i)\phi_j$. Finally, set $x_K=1-\sum_{i=1}^{K-1} x_i$.

Below is example python code to draw the sample:

params = [a1, a2, ..., ak]
xs = [random.betavariate(params[0], sum(params[1:]))]
for j in range(1,len(params)-1):
    phi = random.betavariate(params[j], sum(params[j+1:]))
    xs.append((1-sum(xs)) * phi)
xs.append(1-sum(xs))

Intuitive interpretations of the parameters

The concentration parameter

Dirichlet distributions are very often used as prior distributions in Bayesian inference. The simplest and perhaps most common type of Dirichlet prior is the symmetric Dirichlet distribution, where all parameters are equal. This corresponds to the case where you have no prior information to favor one component over any other. As described above, the single value α to which all parameters are set is called the concentration parameter. If the sample space of the Dirichlet distribution is interpreted as a discrete probability distribution, then intuitively the concentration parameter can be thought of as determining how "concentrated" the probability mass of a sample from a Dirichlet distribution is likely to be. With a value much less than 1, the mass will be highly concentrated in a few components, and all the rest will have almost no mass. With a value much greater than 1, the mass will be dispersed almost equally among all the components. See the article on the concentration parameter for further discussion.

String cutting

One example use of the Dirichlet distribution is if one wanted to cut strings (each of initial length 1.0) into K pieces with different lengths, where each piece had a designated average length, but allowing some variation in the relative sizes of the pieces. The α/α₀ values specify the mean lengths of the cut pieces of string resulting from the distribution. The variance around this mean varies inversely with α₀.

Pólya's urn

Consider an urn containing balls of K different colors. Initially, the urn contains α₁ balls of color 1, α₂ balls of color 2, and so on. Now perform N draws from the urn, where after each draw, the ball is placed back into the urn with an additional ball of the same color. In the limit as N approaches infinity, the proportions of different colored balls in the urn will be distributed as Dir(α₁,...,α_K).

For a formal proof, note that the proportions of the different colored balls form a bounded [0,1]^K-valued martingale, hence by the martingale convergence theorem, these proportions converge almost surely and in mean to a limiting random vector. To see that this limiting vector has the above Dirichlet distribution, check that all mixed moments agree.

Note that each draw from the urn modifies the probability of drawing a ball of any one color from the urn in the future. This modification diminishes with the number of draws, since the relative effect of adding a new ball to the urn diminishes as the urn accumulates increasing numbers of balls. This "diminishing returns" effect can also help explain how small α values yield Dirichlet distributions with most of the probability mass concentrated around a single point on the simplex.

See also

• Generalized Dirichlet distribution
• Latent Dirichlet allocation
• Dirichlet process

External links

• Hazewinkel, Michiel, ed. (2001), "Dirichlet distribution", Encyclopedia of Mathematics, Springer, ISBN 978-1-55608-010-4 
• Dirichlet Distribution
• How to estimate the parameters of the Dirichlet distribution using expectation-maximization (EM)
• Luc Devroye. "Non-Uniform Random Variate Generation". Retrieved May 2012. 
• Dirichlet Random Measures, Method of Construction via Compound Poisson Random Variables, and Exchangeability Properties of the resulting Gamma Distribution