In information theory, the cross-entropy between two probability distributions
and
over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized for an estimated probability distribution
, rather than the true distribution
.
Definition
The cross-entropy of the distribution
relative to a distribution
over a given set is defined as follows:
,
where
is the expected value operator with respect to the distribution
.
The definition may be formulated using the Kullback–Leibler divergence
, divergence of
from
(also known as the relative entropy of
with respect to
).
![{\displaystyle H(p,q)=H(p)+D_{\mathrm {KL} }(p\parallel q),}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a65abe2412b6d51037923018665417e34b5dfbe0)
where
is the entropy of
.
For discrete probability distributions
and
with the same support
this means
![{\displaystyle H(p,q)=-\sum _{x\in {\mathcal {X}}}p(x)\,\log q(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c6b895514e10a3ce88773852cba1cb1e248ed763) | | (Eq.1) |
The situation for continuous distributions is analogous. We have to assume that
and
are absolutely continuous with respect to some reference measure
(usually
is a Lebesgue measure on a Borel σ-algebra). Let
and
be probability density functions of
and
with respect to
. Then
![{\displaystyle -\int _{\mathcal {X}}P(x)\,\log Q(x)\,dr(x)=\operatorname {E} _{p}[-\log Q]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/5f76631e4d31793fd2d2b3bb42796166b04fa4b2)
and therefore
![{\displaystyle H(p,q)=-\int _{\mathcal {X}}P(x)\,\log Q(x)\,dr(x)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c96018b6306901426015782f04705e4549590c55) | | (Eq.2) |
NB: The notation
is also used for a different concept, the joint entropy of
and
.
Motivation
In information theory, the Kraft–McMillan theorem establishes that any directly decodable coding scheme for coding a message to identify one value
out of a set of possibilities
can be seen as representing an implicit probability distribution
over
, where
is the length of the code for
in bits. Therefore, cross-entropy can be interpreted as the expected message-length per datum when a wrong distribution
is assumed while the data actually follows a distribution
. That is why the expectation is taken over the true probability distribution
and not
. Indeed the expected message-length under the true distribution
is
![{\displaystyle \operatorname {E} _{p}[\ell ]=-\operatorname {E} _{p}\left[{\frac {\ln {q(x)}}{\ln(2)}}\right]=-\operatorname {E} _{p}\left[\log _{2}{q(x)}\right]=-\sum _{x_{i}}p(x_{i})\,\log _{2}q(x_{i})=-\sum _{x}p(x)\,\log _{2}q(x)=H(p,q).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/3f5427081785e1c503a5e3c49969c38d8d90c21a)
Estimation
There are many situations where cross-entropy needs to be measured but the distribution of
is unknown. An example is language modeling, where a model is created based on a training set
, and then its cross-entropy is measured on a test set to assess how accurate the model is in predicting the test data. In this example,
is the true distribution of words in any corpus, and
is the distribution of words as predicted by the model. Since the true distribution is unknown, cross-entropy cannot be directly calculated. In these cases, an estimate of cross-entropy is calculated using the following formula:
![{\displaystyle H(T,q)=-\sum _{i=1}^{N}{\frac {1}{N}}\log _{2}q(x_{i})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bb11eae1b2b1120c2bcccf741a51c2511c0cbffe)
where
is the size of the test set, and
is the probability of event
estimated from the training set. In other words,
is the probability estimate of the model that the i-th word of the text is
. The sum is averaged over the
words of the test. This is a Monte Carlo estimate of the true cross-entropy, where the test set is treated as samples from
[citation needed].
Relation to maximum likelihood
In classification problems we want to estimate the probability of different outcomes. Let the estimated probability of outcome
be
with to-be-optimized parameters
and let the frequency (empirical probability) of outcome
in the training set be
.
Given N conditionally independent samples in the training set, then the likelihood of the parameters
of the model
on the training set is
![{\displaystyle {\mathcal {L}}(\theta )=\prod _{i\in X}({\mbox{est. probability of }}i)^{{\mbox{number of occurrences of }}i}=\prod _{i}q_{\theta }(X=i)^{Np(X=i)}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d452937ed5a5bdfdac361ad273d0c55868f0cdd3)
so the log-likelihood, divided by
is
![{\displaystyle {\frac {1}{N}}\log({\mathcal {L}}(\theta ))={\frac {1}{N}}\log \prod _{i}q_{\theta }(X=i)^{Np(X=i)}=\sum _{i}p(X=i)\log q_{\theta }(X=i)=-H(p,q)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/9601b9fef9a3e4c9bb43553b1b3a1d523c0f3dfa)
so that maximizing the likelihood with respect to the parameters
is the same as minimizing the cross-entropy.[citation needed]
Cross-entropy minimization
Cross-entropy minimization is frequently used in optimization and rare-event probability estimation. When comparing a distribution
against a fixed reference distribution
, cross-entropy and KL divergence are identical up to an additive constant (since
is fixed): According to the Gibbs' inequality, both take on their minimal values when
, which is
for KL divergence, and
for cross-entropy. In the engineering literature, the principle of minimizing KL divergence (Kullback's "Principle of Minimum Discrimination Information") is often called the Principle of Minimum Cross-Entropy (MCE), or Minxent.
However, as discussed in the article Kullback–Leibler divergence, sometimes the distribution
is the fixed prior reference distribution, and the distribution
is optimised to be as close to
as possible, subject to some constraint. In this case the two minimisations are not equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be
, rather than
.
Cross-entropy loss function and logistic regression
Cross-entropy can be used to define a loss function in machine learning and optimization. The true probability
is the true label, and the given distribution
is the predicted value of the current model. This is also known as the log loss (or logarithmic loss[1] or logistic loss);[2] the terms "log loss" and "cross-entropy loss" are used interchangeably.[3]
More specifically, consider a binary regression model which can be used to classify observations into two possible classes (often simply labelled
and
). The output of the model for a given observation, given a vector of input features
, can be interpreted as a probability, which serves as the basis for classifying the observation. In logistic regression, the probability is modeled using the logistic function
where
is some function of the input vector
, commonly just a linear function. The probability of the output
is given by
![{\displaystyle q_{y=1}={\hat {y}}\equiv g(\mathbf {w} \cdot \mathbf {x} )={\frac {1}{1+e^{-\mathbf {w} \cdot \mathbf {x} }}},}](https://wikimedia.org/api/rest_v1/media/math/render/svg/3d8f848dfb1d02497994d04b429b1f91330744b9)
where the vector of weights
is optimized through some appropriate algorithm such as gradient descent. Similarly, the complementary probability of finding the output
is simply given by
![{\displaystyle q_{y=0}=1-{\hat {y}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ee6e9334a2dc1ab9ab3339399272e55713a4e9c0)
Having set up our notation,
and
, we can use cross-entropy to get a measure of dissimilarity between
and
:
![{\displaystyle H(p,q)\ =\ -\sum _{i}p_{i}\log q_{i}\ =\ -y\log {\hat {y}}-(1-y)\log(1-{\hat {y}})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1f3f3acfb5549feb520216532a40082193c05ccc)
Logistic regression typically optimizes the log loss for all the observations on which it is trained, which is the same as optimizing the average cross-entropy in the sample. For example, suppose we have
samples with each sample indexed by
. The average of the loss function is then given by:
![{\displaystyle {\begin{aligned}J(\mathbf {w} )\ &=\ {\frac {1}{N}}\sum _{n=1}^{N}H(p_{n},q_{n})\ =\ -{\frac {1}{N}}\sum _{n=1}^{N}\ {\bigg [}y_{n}\log {\hat {y}}_{n}+(1-y_{n})\log(1-{\hat {y}}_{n}){\bigg ]}\,,\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/80f87a71d3a616a0939f5360cec24d702d2593a2)
where
, with
the logistic function as before.
The logistic loss is sometimes called cross-entropy loss. It is also known as log loss (In this case, the binary label is often denoted by {−1,+1}).[4]
Remark: The gradient of the cross-entropy loss for logistic regression is the same as the gradient of the squared error loss for linear regression. That is, define
![{\displaystyle X^{T}={\begin{pmatrix}1&x_{11}&\dots &x_{1p}\\1&x_{21}&\cdots &x_{2p}\\\vdots &\vdots &&\vdots \\1&x_{n1}&\cdots &x_{np}\\\end{pmatrix}}\in \mathbb {R} ^{n\times (p+1)}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/61c77c082b6f7b1ec835bde8679ef2b9032c6f2f)
![{\displaystyle {\hat {y_{i}}}={\hat {f}}(x_{i1},\dots ,x_{ip})={\frac {1}{1+\exp(-\beta _{0}-\beta _{1}x_{i1}-\dots -\beta _{p}x_{ip})}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b01e073319478690c12f7cb9b2578aa7f5bee571)
![{\displaystyle L({\overrightarrow {\beta }})=-\sum _{i=1}^{N}[y^{i}\log {\hat {y}}^{i}+(1-y^{i})\log(1-{\hat {y}}^{i})]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/596a23552927b2ee6c05bfdd176e474834b0c1b1)
Then we have the result
![{\displaystyle {\frac {\partial }{\partial {\overrightarrow {\beta }}}}L({\overrightarrow {\beta }})=X^{T}({\hat {Y}}-Y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/93f9238495a91585ffa646e32c2b80a8a1a815ff)
The proof is as follows. For any
, we have
![{\displaystyle {\frac {\partial }{\partial \beta _{0}}}\ln {\frac {1}{1+e^{-\beta _{0}+k_{0}}}}={\frac {e^{-\beta _{0}+k_{0}}}{1+e^{-\beta _{0}+k_{0}}}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d97553e3d273415fcd9629e2057f3f2030b2d301)
![{\displaystyle {\frac {\partial }{\partial \beta _{0}}}\ln \left(1-{\frac {1}{1+e^{-\beta _{0}+k_{0}}}}\right)={\frac {-1}{1+e^{-\beta _{0}+k_{0}}}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/caa173d300d2f202b787c964ec7ef9ff149bcf90)
![{\displaystyle {\begin{aligned}{\frac {\partial }{\partial \beta _{0}}}L({\overrightarrow {\beta }})&=-\sum _{i=1}^{N}\left[{\frac {y^{i}\cdot e^{-\beta _{0}+k_{0}}}{1+e^{-\beta _{0}+k_{0}}}}-(1-y^{i}){\frac {1}{1+e^{-\beta _{0}+k_{0}}}}\right]\\&=-\sum _{i=1}^{N}[y^{i}-{\hat {y}}^{i}]=\sum _{i=1}^{N}({\hat {y}}^{i}-y^{i})\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1c02e7f3551dd635964e2089931a939e1b1ba2a5)
![{\displaystyle {\frac {\partial }{\partial \beta _{1}}}\ln {\frac {1}{1+e^{-\beta _{1}x_{i1}+k_{1}}}}={\frac {x_{i1}e^{k_{1}}}{e^{\beta _{1}x_{i1}}+e^{k_{1}}}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/3704a36e32d60f7236d7947f18e2ef635b9f9481)
![{\displaystyle {\frac {\partial }{\partial \beta _{1}}}\ln \left[1-{\frac {1}{1+e^{-\beta _{1}x_{i1}+k_{1}}}}\right]={\frac {-x_{i1}e^{\beta _{1}x_{i1}}}{e^{\beta _{1}x_{i1}}+e^{k_{1}}}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0b33c567b55bc690ac5dd5b5c2e2d8b9d870c05c)
![{\displaystyle {\frac {\partial }{\partial \beta _{1}}}L({\overrightarrow {\beta }})=-\sum _{i=1}^{N}x_{i1}(y^{i}-{\hat {y}}^{i})=\sum _{i=1}^{N}x_{i1}({\hat {y}}^{i}-y^{i})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/5bd40cc4d7176b2b421f2257d3d02e03fff6f952)
In a similar way, we eventually obtain the desired result.
See also
References
- ^ The Mathematics of Information Coding, Extraction and Distribution, by George Cybenko, Dianne P. O'Leary, Jorma Rissanen, 1999, p. 82
- ^ Probability for Machine Learning: Discover How To Harness Uncertainty With Python, Jason Brownlee, 2019, p. 220: "Logistic loss refers to the loss function commonly used to optimize a logistic regression model. It may also be referred to as logarithmic loss (which is confusing) or simply log loss."
- ^ sklearn.metrics.log_loss
- ^ Murphy, Kevin (2012). Machine Learning: A Probabilistic Perspective. MIT. ISBN 978-0262018029.
External links