Markov chains

Home | Assessment | Notes | Worksheets | Tutorials

Consider the set $$Y=\{x\in \{0,1{\}}^{\mathbb{N}}:x(n)=1\Rightarrow x(n+1)=0\}$$ of sequences in which every 1 is followed by a zero. This is certainly invariant for the shift map: if $y\in Y$ then $T(y)$ belongs to $Y$ as well.

To study the dynamics of $T$ on $Y$ using ergodic theory we want a measure on $Y$ that is $T$ invariant. For each $0\le p\le 1$ we have the $(1-p,p)$ coin measure ${\mu }_p$ on $\{0,1{\}}^{\mathbb{N}}$ and each is $T$ invariant. However, only one - the point measure ${\mu }_0$ - gives positive measure to $Y$. For example, since the number $w_n$ of cylinders of length $n$ that intersects $Y$ satisfies the recurrence $$w_1=2w_2=3w_{n+2}=w_{n+1}+w_n$$ for all $n\in \mathbb{N}$ we have $${\mu }_{1/2}(Y)\le {\displaystyle \frac{w_n}{2^n}}\to 0$$ as $n\to \mathrm{\infty }$. How can we equip $Y$ with a probability measure $\mu$ that is $T$ invariant?

Walks on directed graphs

The coin measure were inappropriate because they see all cylinders: they assign $[01]$ and $[11]$ positive measure whereas only one of those cylinders intersects $Y$. We can think of the points in $Y$ as the result of all possible infinite walks on the directed graph with vertices $0$, $1$ and directed edges as follows.

• An edge from $0$ to $0$
• An edge from $0$ to $1$
• An edge from $1$ to $0$

To any endless journey on the graph one associates a sequence in $Y$ by recording the labels associated with the visited vertices. As the vertex labelled 1 cannot be visited consecutively, and as there are not other restrictions, we get all sequences in $Y$ this way.

Let us assign probabilities to each traversal. We encode these in a matrix $$\left[\begin{array}{cc}q(0,0)& q(0,1)\\ q(1,0)& q(1,1)\end{array}\right]$$ with $0\le q(i,j)\le 1$ representing the probability that one moves in a single step from vertex $i$ to vertex $j$. We must have $$q(0,0)+q(0,1)=1q(1,0)+q(1,1)=1$$ and we must also have $q(1,1)=0$ as we forbid consecutive ones. To entirely determine the measure we also need the probability of the starting location. Fix $0\le p(i)\le 1$ with $$p(0)+p(1)=1$$ where $p(i)$ is the probability that one begins at vertex $i$. With this information - values for all $p(i)$ and all $q(i,j)$ - we define $$\nu ([{ϵ}_1\cdots {ϵ}_r])=p({ϵ}_1)\prod _{i=1}^{r-1}q({ϵ}_i,{ϵ}_{i+1})$$ on all cylinder sets $[{ϵ}_1\cdots {ϵ}_r]$. For example $$\nu ([01])=p(0)q(0,1)$$ is the probability of beginning at 0 multiplied by the probability that the first step is from 0 to 1.

Note that if $\mu$ is to be an invariant measure then we must have $$p(i)=\mu ([i])=\mu (T^{-1}[i])=p(0)q(0,i)+p(1)q(1,i)$$ which is to say $$\left[\begin{array}{cc}p(0)& p(1)\end{array}\right]\left[\begin{array}{cc}q(0,0)& q(0,1)\\ q(1,0)& q(1,1)\end{array}\right]=\left[\begin{array}{cc}p(0)& p(1)\end{array}\right]$$ holds so we assume this of our parameters.

We will take for granted that the above formula defines a measure $\nu$ on $\{0,1{\}}^{\mathbb{N}}$. Let us verify that it is $T$ invariant. Recall that it suffices to check $\nu (C)$ and $\nu (T^{-1}C)$ agree for all cylinder sets $C$ because such sets form a π system. Write $C=[{ϵ}_1\cdots {ϵ}_r]$. We calculate $$\begin{array}{rl}\nu (T^{-1}C)& =\nu ([0{ϵ}_1\cdots {ϵ}_r])+\nu ([1{ϵ}_1\cdots {ϵ}_r])\\ & =(p(0)q(0,{ϵ}_1)+p(1)q(1,{ϵ}_1))\prod _{i=1}^{r-1}q({ϵ}_i,{ϵ}_{i+1})\\ & =p({ϵ}_1)\prod _{i=1}^{r-1}q({ϵ}_i,{ϵ}_{i+1})=\nu (C)\end{array}$$ so $\nu$ is $T$ invariant.

We have total freedom in the parameters $p(0)$ and $q(0,0)$. Fixing their values then determines $p(1)$ and $q(0,1)$ by the laws of total probability. What is the best way to choose their values? Absent any other information about the dynamics, or any other quantity that we might be interested in, it is often reasonable to choose the values that maximize the entropy.

Proposition

For the above measure the quantity $$-p(0)q(0,0)\log q(0,0)-p(0)q(0,1)\log q(0,1)$$ is the entropy of $T$.

Proof:

As $\xi =([0],[1])$ is a generator for the Borel σ algebra on $X$ the limit $$\underset{N\to \mathrm{\infty }}{lim}{\displaystyle \frac{1}{N}}\mathsf{H}\left(\underset{n=0}{\overset{N-1}{\bigvee }}{T}^{-n}\xi \right)$$ is the entropy $\mathsf{H}(T)$ we want to calculate. We have $$\begin{array}{rl}& -\mathsf{H}\left(\underset{n=0}{\overset{N-1}{\bigvee }}{T}^{-n}\xi \right)\\ =& \sum _{ϵ\in \{0,1{\}}^N}\mu ([{ϵ}_1\cdots {ϵ}_r])\log \mu ([{ϵ}_1\cdots {ϵ}_r])\\ =& \sum _{ϵ\in \{0,1{\}}^{\mathbb{N}}}p({ϵ}_1)q({ϵ}_1,{ϵ}_2)\cdots q({ϵ}_{N-1},{ϵ}_N)\log p({ϵ}_1)q({ϵ}_1,{ϵ}_2)\cdots q({ϵ}_{N-1},{ϵ}_N)\\ =& \sum _{i\in \{0,1\}}p(i)\log p(i)+(N-1)\sum _{i,j\in \{0,1\}}p(i)q(i,j)\log q(i,j)\end{array}$$ by writing a sum of logarithms and then using repeatedly both $$p(i)=p(0)q(0,i)+p(1)q(1,i)q(i,0)+q(i,1)=1$$ for $i=1,2$. Dividing by $N$ and taking the limit as $N\to \mathrm{\infty }$ gives the desired result as $q(1,1)=0$ and $q(1,0)=1$ in our special case. ▮

The Parry measure

We would like to maximize $$-p(0)q(0,0)\log q(0,0)-p(0)q(0,1)\log q(0,1)$$ for values in $[0,1]$ subject to $$\begin{array}{c}p(0)+p(1)=1q(0,0)+q(0,1)=1\\ p(0)q(0,0)+p(1)=p(0)p(0)q(0,1)=p(1)\end{array}$$ which is not so simple an optimization problem.

If we attempt to be as unbiased as possible in our walk on the graph, choosing which edge to traverse from vertex 0 each time by a fair coin toss then we can assert $$q(0,0)=\frac{1}{2}=q(0,1)$$ which then forces $p(0)=\frac{2}{3}$ and $p(1)=\frac{1}{3}$. For these particular choices we get an entropy value of $$\mathsf{H}(T)=-2\cdot {\displaystyle \frac{1}{3}}{\displaystyle \frac{1}{2}}\log {\displaystyle \frac{1}{2}}={\displaystyle \frac{2}{3}}\log 2\approx 0.200686664$$ but it is not clear that this is maximal.

Theorem (Parry)

Let $B$ be a $k\times k$ matrix with entries from $\{0,1\}$. Suppose there is $r\in \mathbb{N}$ with all entries of $B^r$ positive. Let $\lambda$ be the largest positive eigenvalue of $B$. Fix left and right eigenvectors $u$ and $v$ of $B$ respectively with $$u(1)v(1)+\cdots +u(k)v(k)=1$$ such that both have an eigenvalue of $\lambda$. With $$p(i)=u(i)v(i)$$ and $$q(i,j)={\displaystyle \frac{B(i,j)}{\lambda }}{\displaystyle \frac{v(j)}{v(i)}}$$ the corresponding Markov measure maximizes the entropy for $T$ on the set $$Y=\{x\in \{1,\dots ,k{\}}^{\mathbb{N}}:B(x(n),x(n+1))=1\text{ for all }n\in \mathbb{N}\}$$ where transitions are determined by $B$.

Proof:

This is Theorem 8.10 in Walters. ▮

The resulting measure is the Parry measure on the Markov chain. For our example we have $$B=\left[\begin{array}{cc}1& 1\\ 1& 0\end{array}\right]$$ with eigenvalues $${\displaystyle \frac{1-\sqrt{5}}{2}}{\displaystyle \frac{1+\sqrt{5}}{2}}$$ and the latter must be $\lambda$. The vectors $$U=\left[\begin{array}{cc}1+\sqrt{5}& 2\end{array}\right]V=\left[\begin{array}{c}1+\sqrt{5}\\ 2\end{array}\right]$$ are left and right eigenvectors respectively of $B$ with eingevalue $\lambda$. Since the ratio of the eigenvectors is unchanged by scaling we conclude that $$q(0,0)={\displaystyle \frac{B(0,0)}{\lambda }}{\displaystyle \frac{V(0)}{V(0)}}={\displaystyle \frac{1}{\lambda }}$$ and that $$q(0,1)={\displaystyle \frac{B(0,1)}{\lambda }}{\displaystyle \frac{V(1)}{V(0)}}={\displaystyle \frac{1}{{\lambda }^2}}$$ are the values that will maximize the entropy. As $$p(0)q(0,1)=p(1)=1-p(0)$$ we conclude that $$p(0)={\displaystyle \frac{{\lambda }^2}{1+{\lambda }^2}}p(1)={\displaystyle \frac{1}{1+{\lambda }^2}}$$ giving an entropy value of $$\mathsf{H}(T)=\log \lambda \approx 0.20898764$$ which is indeed larger than entropy one gets from our naive guess $q(0,0)=\frac{1}{2}$. In fact, the entropy of the Parry measure is always $\log \lambda$.