Categorical distribution

In probability theory and statistics, a categorical distribution (occasionally termed the "discrete distribution",^{[citation needed]} which properly refers to a general class of distributions) is a probability distribution that describes the result of a random event that can take on one of K possible outcomes, with the probability of each outcome separately specified. There is not necessarily an underlying ordering of these outcomes, but numerical labels are attached for convenience in describing the distribution, often in the range 1 to K. Note that the K-dimensional categorical distribution is the most general distribution over a K-way event; any other discrete distribution over a size-K sample space is a special case. The parameters specifying the probabilities of each possible outcome are constrained only by the fact that each must be in the range 0 to 1, and all must sum to 1.

Note that, in some fields, such as natural language processing, the categorical and multinomial distributions are conflated, and it is common to speak of a "multinomial distribution" when a categorical distribution is actually meant. This stems from the fact that it is sometimes convenient to express the outcome of a categorical distribution as a "1-of-K" vector (a vector with one element containing a 1 and all other elements containing a 0) rather than as an integer in the range 1 to K; in this form, a categorical distribution is equivalent to a multinomial distribution for a single observation (see below).

Introduction

A categorical distribution is a discrete probability distribution whose sample space is the set of n individually identified items. It is the generalization of the Bernoulli distribution for a categorical random variable.

In one formulation of the distribution, the sample space is taken to be a finite sequence of integers. The items might be encoded as {0, 1, ..., n-1} or {1, 2, ..., n} for example: the latter is used here, although the former is the used for the Bernoulli distribution. In this case, the probability mass function f is:

$f(x_i; \boldsymbol{p} ) = p_i ,$

where p_i represents the probability of seeing element i and $\textstyle{\sum_{i=1}^n p_i = 1}$.

In another formulation, the categorical distribution is a special case of the multinomial distribution in which the parameter n of the multinomial distribution is fixed at 1. In this formulation, the sample space can be considered to be the set of 1-of-N encoded (also known as 1-of-K encoded) random vectors x of dimension n having the property that exactly one element has the value 1 and the others have the value 0. The probability mass function f in this formulation is:

$f( \mathbf{x} ; \boldsymbol{p} ) = \prod_{i=1}^n p_i^{x_i} ,$

where p_i represents the probability of seeing element i and $\textstyle{\sum_i p_i = 1}$. This is the formulation adopted by Bishop .

Properties

The possible probabilities for the categorical distribution with n = 3 are the 2-simplex x₁ + x₂ + x₃ = 1, embedded in 3-space.

• The distribution is completely given by the probabilities associated with each number k: p_k = P(X = x_k), k = 1,...,n, where $\textstyle{\sum_k p_k = 1}$. The possible probabilities are exactly the standard (n − 1)-dimensional simplex; for n = 2 this reduces to the possible probabilities of the Bernoulli distribution being the 1-simplex, $p+q=1, 0 \leq p \leq 1$.

• The distribution is a special case of a "multivariate Bernoulli distribution" in which exactly one of the n 0-1 variables takes the value one.

• $\mathbb{E} \left[ \mathbf{x} \right] = \boldsymbol{p}$

• Let $\boldsymbol{X}$ be the realisation from a categorical distribution. Define the random vector Y as composed of the elements:

$Y_j=I(\boldsymbol{X}=x_j),$

where I is the indicator function. Then Y has a distribution which is a special case of the multinomial distribution with parameter n = 1. The sum of n independent and identically distributed such random variables Y constructed from a categorical distribution with parameter $\boldsymbol{p}$ is multinomially distributed with parameters n and $\boldsymbol{p}$

• The sufficient statistic from n independent observations is the set of counts (or, equivalently, proportion) of observations in each category, where the total number of trials (=n) is fixed.

• The conjugate prior is the Dirichlet distribution.

• The indicator function of an observation, x_k, is Bernoulli distributed with parameter p_k.

See also

• Categorical data
• Categorical variable

Related distributions

• Dirichlet distribution
• Multinomial distribution
• Bernoulli distribution