Venn diagram of information theoretic measures for three variables , , and , represented by the lower left, lower right, and upper circles, respectively. The conditional mutual informations , and are represented by the yellow, cyan, and magenta regions, respectively.
A more general definition of conditional mutual information, applicable to random variables with continuous or other arbitrary distributions, will depend on the concept of regular conditional probability. (See also.)
Let be a probability space, and let the random variables , , and each be defined as a Borel-measurable function from to some state space endowed with a topological structure.
Consider the Borel measure (on the σ-algebra generated by the open sets) in the state space of each random variable defined by assigning each Borel set the -measure of its preimage in . This is called the pushforward measure The support of a random variable is defined to be the topological support of this measure, i.e.
In an expression such as and need not necessarily be restricted to representing individual random variables, but could also represent the joint distribution of any collection of random variables defined on the same probability space. As is common in probability theory, we may use the comma to denote such a joint distribution, e.g. Hence the use of the semicolon (or occasionally a colon or even a wedge ) to separate the principal arguments of the mutual information symbol. (No such distinction is necessary in the symbol for joint entropy, since the joint entropy of any number of random variables is the same as the entropy of their joint distribution.)
for discrete, jointly distributed random variables , and . This result has been used as a basic building block for proving other inequalities in information theory, in particular, those known as Shannon-type inequalities.
Conditioning on a third random variable may either increase or decrease the mutual information: that is, the difference , called the interaction information, may be positive, negative, or zero. This is the case even when random variables are pairwise independent. Such is the case when:
in which case , and are pairwise independent and in particular , but
The conditional mutual information can be used to inductively define a multivariate mutual information in a set- or measure-theoretic sense in the context of information diagrams. In this sense we define the multivariate mutual information as follows:
This definition is identical to that of interaction information except for a change in sign in the case of an odd number of random variables. A complication is that this multivariate mutual information (as well as the interaction information) can be positive, negative, or zero, which makes this quantity difficult to interpret intuitively. In fact, for random variables, there are degrees of freedom for how they might be correlated in an information-theoretic sense, corresponding to each non-empty subset of these variables. These degrees of freedom are bounded by various Shannon- and non-Shannon-type inequalities in information theory.
^D. Leao, Jr. et al. Regular conditional probability, disintegration of probability and Radon spaces. Proyecciones. Vol. 23, No. 1, pp. 15–29, May 2004, Universidad Católica del Norte, Antofagasta, Chile PDF