Directed information

Directed information, $I(X^{n}\to Y^{n})$, is a measure of information theory and it measures the amount of information that flows from the process $X^{n}$ to $Y^{n}$, where $X^{n}$ denotes the vector $X_{1},X_{2},...,X_{n}$ and $Y^{n}$ denotes $Y_{1},Y_{2},...,Y_{n}$. The term "directed information" was coined by James Massey and is defined as

$I(X^{n}\to Y^{n})=\sum _{i=1}^{n}I(X^{i};Y_{i}|Y^{i-1})$,

where $I(X^{i};Y_{i}|Y^{i-1})$ is the conditional mutual information.

Note that if $n=1$, directed information becomes mutual information $I(X;Y)$. Directed information has many applications in problems where causality plays an important role such as capacity of channel with feedback, capacity of discrete memoryless networks with feedback, gambling with causal side information, compression with causal side information, and in real-time control communication settings.