# Entropy rate

(Redirected from Source information rate)

In the mathematical theory of probability, the entropy rate or source information rate of a stochastic process is, informally, the time density of the average information in a stochastic process. For stochastic processes with a countable index, the entropy rate H(X) is the limit of the joint entropy of n members of the process Xk divided by n, as n tends to infinity:

${\displaystyle H(X)=\lim _{n\to \infty }{\frac {1}{n}}H(X_{1},X_{2},\dots X_{n})}$

when the limit exists. An alternative, related quantity is:

${\displaystyle H'(X)=\lim _{n\to \infty }H(X_{n}|X_{n-1},X_{n-2},\dots X_{1})}$

For strongly stationary stochastic processes, ${\displaystyle H(X)=H'(X)}$. The entropy rate can be thought of as a general property of stochastic sources; this is the asymptotic equipartition property.

## Entropy rates for Markov chains

Since a stochastic process defined by a Markov chain that is irreducible, aperiodic and positive recurrent has a stationary distribution, the entropy rate is independent of the initial distribution.

For example, for such a Markov chain Yk defined on a countable number of states, given the transition matrix Pij, H(Y) is given by:

${\displaystyle \displaystyle H(Y)=-\sum _{ij}\mu _{i}P_{ij}\log P_{ij}}$

where μi is the asymptotic distribution of the chain.

A simple consequence of this definition is that an i.i.d. stochastic process has an entropy rate that is the same as the entropy of any individual member of the process.