Markov chains on a measurable state space

A Markov chain on a measurable state space is a discrete-time-homogenous Markov chain with a measurable space as state space.

History

The definition of Markov chains has evolved during the 20th century. In 1953 the term Markov chain was used for stochastic processes with discrete or continuous index set, living on a countable or finite state space, see Doob or Chung. Since the late 20th century it became more popular to consider a Markov chain as a stochastic process with discrete index set, living on a measurable state space.

Definition

Denote with $(E,\Sigma )$ a measurable space and with $p$ a Markov kernel with source and target $(E,\Sigma )$. A stochastic process $(X_{n})_{n\in \mathbb {N} }$ on $(\Omega ,{\mathcal {F}},\mathbb {P} )$ is called a time homogeneous Markov chain with Markov kernel $p$ and start distribution $\mu$ if

$\mathbb {P} [X_{0}\in A_{0},X_{1}\in A_{1},\dots ,X_{n}\in A_{n}]=\int _{A_{0}}\dots \int _{A_{n-1}}p(y_{n-1},A_{n})\,p(y_{n-2},dy_{n-1})\dots p(y_{0},dy_{1})\,\mu (dy_{0})$

is satisfied for any $n\in \mathbb {N} ,\,A_{0},\dots ,A_{n}\in \Sigma$. One can construct for any Markov kernel and any probability measure an associated Markov chain.

Remark about Markov kernel integration

For any measure $\mu \colon \Sigma \to [0,\infty ]$ we denote for $\mu$-integrable function $f\colon E\to \mathbb {R} \cup \{\infty ,-\infty \}$ the Lebesgue integral as $\int _{E}f(x)\,\mu (dx)$. For the measure $\nu _{x}\colon \Sigma \to [0,\infty ]$ defined by $\nu _{x}(A):=p(x,A)$ we used the following notation:

$\int _{E}f(y)\,p(x,dy):=\int _{E}f(y)\,\nu _{x}(dy).$

Basic properties

Starting in a single point

If $\mu$ is a Dirac measure in $x$, we denote for a Markov kernel $p$ with starting distribution $\mu$ the associated Markov chain as $(X_{n})_{n\in \mathbb {N} }$ on $(\Omega ,{\mathcal {F}},\mathbb {P} _{x})$ and the expectation value

$\mathbb {E} _{x}[X]=\int _{\Omega }X(\omega )\,\mathbb {P} _{x}(d\omega )$

for a $\mathbb {P} _{x}$-integrable function $X$. By definition, we have then $\mathbb {P} _{x}[X_{0}=x]=1$.

We have for any measurable function $f\colon E\to [0,\infty ]$ the following relation :

$\int _{E}f(y)\,p(x,dy)=\mathbb {E} _{x}[f(X_{1})].$

Family of Markov kernels

For a Markov kernel $p$ with starting distribution $\mu$ one can introduce a family of Markov kernels $(p_{n})_{n\in \mathbb {N} }$ by

$p_{n+1}(x,A):=\int _{E}p_{n}(y,A)\,p(x,dy)$

for $n\in \mathbb {N} ,\,n\geq 1$ and $p_{1}:=p$. For the associated Markov chain $(X_{n})_{n\in \mathbb {N} }$ according to $p$ and $\mu$ one obtains

$\mathbb {P} [X_{0}\in A,\,X_{n}\in B]=\int _{A}p_{n}(x,B)\,\mu (dx)$.

Construction of Markov kernels

For a given Markov kernel $p$ and probability measure $\mu$ one finds an associated time-homogenous Markov chain $(X_{n})_{n\in \mathbb {N} }$ on $(\Omega ,{\mathcal {F}},\mathbb {P} )$. If only the time-homogenous Markov chain $(X_{n})_{n\in \mathbb {N} }$ on $(\Omega ,{\mathcal {F}},\mathbb {P} )$ is given, then one can obtain the Markov kernel $p$ only $\mu$-almost everywhere . This means that one can rebuild the Markov kernel only up to the start distribution. For example, if $\mu$ is a Dirac measure in $x$, then we can rebuild from $(X_{n})_{n\in \mathbb {N} }$ on $(\Omega ,{\mathcal {F}},\mathbb {P} _{x})$ the Markov kernel p only at point x.

One can embed any two random variables $X,Y\colon \Omega \to E$ into the beginning of a Markov chain, i.e. there exists a Markov chain $(X_{n})_{n\in \mathbb {N} }$ on $(\Omega ,{\mathcal {F}},\mathbb {P} _{\mu })$ with $\mathbb {P} [X\in A,\,Y\in B]=\mathbb {P} _{\mu }[X_{0}\in A,\,X_{1}\in B]$ for any $A,B\in \Sigma$.

Stationary measure

A probability measure $\mu$ is called stationary measure of a Markov kernel $p$ if

$\int _{A}\mu (dx)=\int _{E}p(x,A)\,\mu (dx)$

holds for any $A\in \Sigma$. If $(X_{n})_{n\in \mathbb {N} }$ on $(\Omega ,{\mathcal {F}},\mathbb {P} )$ denotes the Markov chain according to a Markov kernel $p$ with stationary measure $\mu$, then all $X_{n}$ have the same probability distribution, namely:

$\mathbb {P} [X_{n}\in A]=\mu (A)$

for any $A\in \Sigma$.

Reversibility

A Markov kernel $p$ is called reversible according to a probability measure $\mu$ if

$\int _{A}p(x,B)\,\mu (dx)=\int _{B}p(x,A)\,\mu (dx)$ holds for any $A,B\in \Sigma$.

Replacing $A=E$ shows that if $p$ is reversible according to $\mu$, then $\mu$ must be a stationary measure of $p$.