The law of large numbers

Home | Assessment | Notes | Worksheets | Tutorials

In this section we will prove the strong law of large numbers.

Theorem (Strong law of large numbers)

For all $0\le p\le 1$ the set $$\{x\in \{0,1{\}}^{\mathbb{N}}:\underset{N\to \mathrm{\infty }}{lim}{\displaystyle \frac{|\{1\le n\le N:x(n)=1\}|}{N}}=p\}$$ has full measure with respect to ${\mu }_p$.

Fix $0\le p\le 1$. Define a function $Y_n:\{0,1{\}}^{\mathbb{N}}\to \mathbb{R}$ by $$Y_n(x)=x(n)-p$$ for each $n\in \mathbb{N}$. If $x\in \{0,1{\}}^{\mathbb{N}}$ satisfies $$\underset{N\to \mathrm{\infty }}{lim}{\displaystyle \frac{Y_1(x)+\cdots +Y_N(x)}{N}}=0$$ then we have $$\underset{N\to \mathrm{\infty }}{lim}{\displaystyle \frac{x(1)+\cdots +x(N)}{N}}=p$$ so it suffices to work with the $Y_n$. We could work throughout with function $Z_n(x)=x(n)$ but it is slightly more convenient - and often done - to work with the functions $Y_n$ instead. The main advantage of $Y_n$ is that is has zero integral. We can check this directly because $$Y_n=1_{T^{n-1}[1]}-p$$ is a simple function and therefore has integral $${\mu }_p(T^{n-1}([1]))-p=p-p=0$$ as claimed.

We want to show that the set $$B=\{x\in \{0,1{\}}^{\mathbb{N}}:\underset{N\to \mathrm{\infty }}{lim}{\displaystyle \frac{Y_1(x)+\cdots +Y_N(x)}{N}}\ne 0\}$$ has zero measure. First note that $$B=\bigcup _{r\in \mathbb{N}}\bigcap _{M\in \mathbb{N}}\bigcup _{N\ge M}\{x\in \{0,1{\}}^{\mathbb{N}}:\left|{\displaystyle \frac{Y_1(x)+\cdots +Y_N(x)}{N}}\right|\ge {\displaystyle \frac{1}{r}}\}$$ so we must show for every $r\in \mathbb{N}$ that $$\bigcap _{M\in \mathbb{N}}\bigcup _{N\ge M}\{x\in \{0,1{\}}^{\mathbb{N}}:\left|{\displaystyle \frac{Y_1(x)+\cdots +Y_N(x)}{N}}\right|\ge {\displaystyle \frac{1}{r}}\}$$ has zero measure with respect to ${\mu }_p$. The following lemma will give us a way to do this.

Lemma (Borel-Cantelli lemma)

Let $(X,\mathcal{B},\nu )$ be a measure space with $\nu (X)=1$. For any sequence $A_1,A_2,\dots$ in $\mathcal{B}$ the convergence $$\sum _{n=1}^{\mathrm{\infty }}\nu (A_n)<\mathrm{\infty }$$ implies $$\bigcap _{M\in \mathbb{N}}\bigcup _{N\ge M}{A}_N$$ has zero measure.

Proof:

By subadditivity we can calculate that $$\nu \left(\bigcup _{N\ge M}{A}_N\right)\le \sum _{N=M}^{\mathrm{\infty }}\nu (A_N)\to 0$$ as $N\to \mathrm{\infty }$. Since $$M\mapsto \bigcup _{N\ge M}{A}_N$$ is a decreasings sequence of sets and $\nu (X)=1$ the result follows from continuity of measures.▮

Returning to the strong law of large numbers, it suffices by the Borel-Cantelli lemma to prove that $$N\mapsto {\nu }_p\left(\{x\in \{0,1{\}}^{\mathbb{N}}:\left|{\displaystyle \frac{Y_1(x)+\cdots +Y_N(x)}{N}}\right|\ge {\displaystyle \frac{1}{r}}\}\right)$$ is summable. Fix $r\in \mathbb{N}$ and $N\in \mathbb{N}$.

Lemma (Markov's inequality)

Fix a measure space $(X,\mathcal{B},\nu )$ with $\nu (X)=1$ and fix a measurable function $f:X\to [0,\mathrm{\infty })$. Then $$\nu (\{x\in X:f(x)\ge s\})\le {\displaystyle \frac{1}{s}}\int f\mathsf{d}\nu$$ for every $s\ge 0$.

Proof:

This is nothing other than the fact that $$0\le s\cdot 1_{\{x\in X:f(x)\ge s\}}\le f$$ and monotonicity of the integral.▮

Applying Markov's inequality to our set gives us $$\begin{array}{rl}& {\mu }_p\left(\{x\in \{0,1{\}}^{\mathbb{N}}:\left|{\displaystyle \frac{Y_1(x)+\cdots +Y_N(x)}{N}}\right|\ge {\displaystyle \frac{1}{r}}\}\right)\\ \le & r\int \left|{\displaystyle \frac{Y_1+\cdots +Y_N}{N}}\right|\mathsf{d}{\mu }_p\\ \le & {\displaystyle \frac{r}{N}}\sum _{n=1}^N\int |Y_n|\mathsf{d}{\mu }_p\end{array}$$ for all $N\in \mathbb{N}$. However $$|Y_n|=(1-p)\cdot 1_{T^{n-1}[1]}+p\cdot 1_{T^{n-1}[0]}$$ so our upper bound above is just $${\displaystyle \frac{r}{N}}\sum _{n=1}^{N}2p(1-p)$$ which is not summable in $N$.

To do better, we note that $$\left|{\displaystyle \frac{Y_1(x)+\cdots +Y_N(x)}{N}}\right|\ge {\displaystyle \frac{1}{r}}\iff {\left|{\displaystyle \frac{Y_1(x)+\cdots +Y_N(x)}{N}}\right|}^4\ge {\displaystyle \frac{1}{r^4}}$$ and we may therefore also take $$\begin{array}{rl}& {\mu }_p\left(\{x\in \{0,1{\}}^{\mathbb{N}}:\left|{\displaystyle \frac{Y_1(x)+\cdots +Y_N(x)}{N}}\right|\ge {\displaystyle \frac{1}{r}}\}\right)\\ \le & r^4\int {\left|{\displaystyle \frac{Y_1+\cdots +Y_N}{N}}\right|}^4\mathsf{d}{\mu }_p\\ =& {\displaystyle \frac{r^4}{N^4}}\int |Y_1+\cdots +Y_N{|}^4\mathsf{d}{\mu }_p\end{array}$$ from Markov's inequality. It remains for us to calculate the integral $$\int |Y_1+\cdots +Y_N{|}^4=\sum _{a,b,c,d=1}^N\int Y_a{Y}_b{Y}_c{Y}_d\mathsf{d}{\mu }_p$$ which we do by checking various possibilities.

First we note that $$\int Y_a{Y}_b{Y}_c{Y}_d\mathsf{d}{\mu }_p=0$$ will be zero whenever some member of the tuple $(a,b,c,d)$ appears only once. This is because ${\mu }_p$ is such that $$\int Y_a^{\alpha }{Y}_b^{\beta }{Y}_c^{\gamma }{Y}_d^{\delta }\mathsf{d}{\mu }_p=\int Y_a^{\alpha }\mathsf{d}{\mu }_p\int Y_b^{\beta }{\mu }_p\int Y_c^{\gamma }{\mu }_p\int Y_d^{\delta }\mathsf{d}{\mu }_p$$ whenver $\alpha ,\beta ,\gamma ,\delta$ belong to $\mathbb{N}$ and $a,b,c,d$ are pairwise distinct.

Thus $$\sum _{a,b,c,d=1}^N\int Y_a{Y}_b{Y}_c{Y}_d\mathsf{d}{\mu }_p=\sum _{a,b=1}^N\int Y_a^2{Y}_b^2\mathsf{d}{\mu }_p$$ and from $|Y_a|\le 1$ we get $$|\sum _{a,b,c,d=1}^N\int Y_a{Y}_b{Y}_c{Y}_d\mathsf{d}{\mu }_p|\le N^2$$ for all $N\in \mathbb{N}$. To summarize, Markov's inequality gives $$\begin{array}{rl}& {\mu }_p\left(\{x\in \{0,1{\}}^{\mathbb{N}}:\left|{\displaystyle \frac{Y_1(x)+\cdots +Y_N(x)}{N}}\right|\ge {\displaystyle \frac{1}{r}}\}\right)\\ \le & r^4\int {\left|{\displaystyle \frac{Y_1+\cdots +Y_N}{N}}\right|}^4\mathsf{d}{\mu }_p\\ \le & {\displaystyle \frac{r^4}{N^4}}\int \sum _{a,b,c,d=1}^N{Y}_a{Y}_b{Y}_c{Y}_d\mathsf{d}{\mu }_p\\ \le & {\displaystyle \frac{r^4}{N^2}}\end{array}$$ which is summable in $N$. We conclude from the Borel-Cantelli lemma that $$\bigcap _{M\in \mathbb{N}}\bigcup _{N\ge M}\{x\in \{0,1{\}}^{\mathbb{N}}:\left|{\displaystyle \frac{Y_1(x)+\cdots +Y_N(x)}{N}}\right|\ge {\displaystyle \frac{1}{r}}\}$$ has zero measure. Since $r\in \mathbb{N}$ was arbitrary, we are done.