A sequence of random variables \(X_1, X_2, X_3, \dots\) is independent, identically distributed is the random variables \(X_1, X_2, X_3, \dots\) are independent and all have exactly the same distribution.
Independent, identically distributed sequences are common enough that we shorten the property to i.i.d.
For example, a sequence of random variables \(X_1, X_2, X_3, \dots\) which are indepenent and have Poisson distributions is not necessarily i.i.d. If all the random variables have Poisson(\(3\)) distributions, then the sequence is i.i.d.
There is an open question regarding the existence of such sequences. We will take for granted here that such sequences to exist and the proof that they do will be reserved for a more advanced probability course.
Let \(X_1, X_2, X_3, \dots\) be a sequence of i.i.d. random variabes. The strong law of large numbers states that \[\lim_{n \rightarrow \infty} \frac{\sum_{i=1}^n X_i}{n} = E[X_1]\] with probability \(1.\)
The strong law of large numbers holds even when \(E[X_1]\) is \(\infty\) or \(-\infty.\) Also, \(E[X_1] = E[X_n]\) for any \(X_n\) in the sequence since the sequence is i.i.d.
The significance of the result should not be overlooked. Notice that the limit is a limit of random values. As more and more of the random values are accounted for, their average converges to the mean of one of the variables in the sequence, which is not random. Dividing by \(n\) eliminates the randomness.
The proof of the strong law of large numbers is reserved for a more advanced probability course.
Let \(X_1, X_2, X_3, \dots\) be i.i.d. with pmf \(p(4) = \frac{1}{2},\) \(p(10) = \frac{1}{2}.\) Then
\[E[X_1] = \frac{1}{2} \cdot 4 + \frac{1}{2} \cdot 10 = 7\]
For a particular outcome in the sample space, \(\omega \in \Omega,\) the sequence \(X_1(\omega), X_2(\omega), X_3(\omega), \dots\) is a sequence of \(4\)'s and \(10\)'s. With probability \(1,\) the running average of that sequence will converge to \(7.\)
Suppose the first \(10\) terms of the sequence have the values \(10, 10, 4, 10, 10, 4, 10, 4, 10, 4.\) Then the first \(10\) terms in the running average are
\begin{align}
& \frac{10}{1} = 10 \\
& \frac{10+10}{2} = 10 \\
& \frac{10+10+4}{3} = 8 \\
& \frac{10+10+4+10}{4} = 8.5 \\
& \frac{10+10+4+10+10}{5} = 8.8 \\
& \frac{10+10+4+10+10+4}{6} = 8 \\
& \frac{10+10+4+10+10+4+10}{7} \approx 8.29 \\
& \frac{10+10+4+10+10+4+10+4}{8} = 7.75 \\
& \frac{10+10+4+10+10+4+10+4+10}{9} = 8 \\
& \frac{10+10+4+10+10+4+10+4+10+4}{10} = 7.6 \\
\end{align}
As the sequence continues, the running average will converge to the mean, \(7,\) with probability \(1.\)
Check your understanding:
1. Let \(X_1, X_2, \dots\) be i.i.d. random variables with mean \(2.\) Compute
\[\lim_{n \rightarrow \infty} \frac{1}{n-1}\sum_{i=1}^n X_i\]
Unanswered
2. A survey is given that asks people to record the number of silbings they have. The results of the first \(10\) responses are
\[2, 0, 1, 0, 8, 3, 0, 2, 4, 1\]
If we assume the responses are i.i.d., what is the expected value of the next response?
Unanswered