AMH - 3__probability.3

Definition

A sequence of random variables $X_1, X_2, X_3, \dots$ is is the random variables $X_1, X_2, X_3, \dots$ are independent and all have exactly the same distribution.

Independent, identically distributed sequences are common enough that we shorten the property to i.i.d.

For example, a sequence of random variables $X_1, X_2, X_3, \dots$ which are indepenent and have Poisson distributions is not necessarily i.i.d. If all the random variables have Poisson( $3$ ) distributions, then the sequence is i.i.d.

There is an open question regarding the existence of such sequences. We will take for granted here that such sequences to exist and the proof that they do will be reserved for a more advanced probability course.

Strong Law Of Large Numbers

Let $X_1, X_2, X_3, \dots$ be a sequence of i.i.d. random variabes. The states that $\lim_{n \rightarrow \infty} \frac{\sum_{i=1}^n X_i}{n} = E[X_1]$ with probability $1.$

The strong law of large numbers holds even when $E[X_1]$ is $\infty$ or $-\infty.$ Also, $E[X_1] = E[X_n]$ for any $X_n$ in the sequence since the sequence is i.i.d.

The significance of the result should not be overlooked. Notice that the limit is a limit of random values. As more and more of the random values are accounted for, their average converges to the mean of one of the variables in the sequence, which is not random. Dividing by $n$ eliminates the randomness.

The proof of the strong law of large numbers is reserved for a more advanced probability course.

Example

Let $X_1, X_2, X_3, \dots$ be i.i.d. with pmf $p(4) = \frac{1}{2},$ $p(10) = \frac{1}{2}.$ Then $E[X_1] = \frac{1}{2} \cdot 4 + \frac{1}{2} \cdot 10 = 7$ For a particular outcome in the sample space, $\omega \in \Omega,$ the sequence $X_1(\omega), X_2(\omega), X_3(\omega), \dots$ is a sequence of $4$ 's and $10$ 's. With probability $1,$ the running average of that sequence will converge to $7.$

Suppose the first $10$ terms of the sequence have the values $10, 10, 4, 10, 10, 4, 10, 4, 10, 4.$ Then the first $10$ terms in the running average are $\begin{align} & \frac{10}{1} = 10 \\ & \frac{10+10}{2} = 10 \\ & \frac{10+10+4}{3} = 8 \\ & \frac{10+10+4+10}{4} = 8.5 \\ & \frac{10+10+4+10+10}{5} = 8.8 \\ & \frac{10+10+4+10+10+4}{6} = 8 \\ & \frac{10+10+4+10+10+4+10}{7} \approx 8.29 \\ & \frac{10+10+4+10+10+4+10+4}{8} = 7.75 \\ & \frac{10+10+4+10+10+4+10+4+10}{9} = 8 \\ & \frac{10+10+4+10+10+4+10+4+10+4}{10} = 7.6 \\ \end{align}$ As the sequence continues, the running average will converge to the mean, $7,$ with probability $1.$

Check your understanding:

1. Let $X_1, X_2, \dots$ be i.i.d. random variables with mean $2.$ Compute $\lim_{n \rightarrow \infty} \frac{1}{n-1}\sum_{i=1}^n X_i$

Unanswered

Solution: By the strong law of large numbers, $\lim_{n \rightarrow \infty} \frac{1}{n} \sum_{i=1}^n X_i = 2.$ By properties of limits, $\begin{align} \lim_{n \rightarrow \infty} \frac{1}{n-1} \sum_{i=1}^n X_i & = \lim_{n \rightarrow \infty} \frac{n}{n} \cdot \frac{1}{n-1}\sum_{i=1}^n X_i \\ & = \left(\lim_{n \rightarrow \infty} \frac{n}{n-1}\right)\left(\lim_{n \rightarrow \infty}\frac{1}{n}\sum_{i=1}^n X_i\right) \\ & = 1 \cdot 2 \\ & = 2 \end{align}$

2. A survey is given that asks people to record the number of silbings they have. The results of the first $10$ responses are $2, 0, 1, 0, 8, 3, 0, 2, 4, 1$ If we assume the responses are i.i.d., what is the expected value of the next response?

Unanswered

Solution: Since we are assuming the random variables are i.i.d., the best guess for the expected value of the random variables is the sum of the variables we have so far. $\frac{2+0+1+0+8+3+0+2+4+1}{10} = 2.1$