The Law of Large Numbers

After much confusion about what topic to post, I have decided to go with one of most basic and well known laws in statistics, the Law of Large Numbers. The motivation for this is an interesting read from Stephen Stigler's book titled, "The History of Statistics".

In Chapter 2 of this book, the author mentions that -"the consideration of games of chance led to the first mathematical treatment of the quantification of uncertainty". By the way, this is the 17th Century we are talking about. The early works focused on the quantifying uncertainty a priori, i.e., given that we have an unbiased coin, what is the probability of getting a heads? The answer is of course, p = 0.5, since an unbiased coin makes heads and tails equally likely!

However the a posteriori question of determining the bias on the coin given our observed or empirical data was addressed by one of the most renowned families in the history of mathematical sciences, the Bernoulli family.

Jacob Bernoulli (shown above) (source: Wikipedia), one of the greatest mathematicians of the 17th century, is responsible for the law of large numbers. It was found in his work, Ars Conjectandi, which was published posthumously by his nephew Nicholas Bernoulli.

Bernoulli proved that if X is the total number of observed successes for a total of N observations and if p is the unknown proportion or the "probability" of success, then for any given small {\epsilon} and any given large positive number c, N can be specified so that,  

The second inequality can be derived from the first inequality using simple algebra. We don't need to bother ourselves with the derivations of these inequalities. Here, {\epsilon} and c can be thought of as parameters that control the amount of accuracy desired in estimation and the idea is that N can be chosen as large as required to fulfill the above inequality.

Note that X/N denotes our estimate for the probability of success, while p denotes the actual or true probability of success. The term |(X/N) - p| denotes how far our estimate is from its true value. Hence an {\epsilon} = 1/50 implies that we want our estimate (X/N) to fall within 1/50 of its true value (p). The chance that our estimate *does not* fall within 1/50 of its true value is controlled by c. Thus for highly accurate estimate it is not difficult to see that we need to specify {\epsilon} as a small value and c as a large value. In this case, N needs to be specified as large as required for the above inequality to hold. This is the Bernoulli's weak law of large numbers.

A stronger version of this law (not credited to Bernoulli, I think) states that for a sequence of pairwise independent, identically distributed random variables say {X_{n}}, if S_{n} is the sum of all {X} from 1 to n,

if E(X) < infinity, then

The law simply states that as we get more information, i.e., as n tends to infinity, our estimate for the mean of the sample denoted by LHS of the above equation converges to the true mean or expectation of X denoted by the RHS. An elementary proof of the above result can be found here. Note that the independent and identically distributed random variable assumption is extremely important to apply this law in practice.

Before I end this post, I would recommend looking at the nice intuitive experiments for illustrating the law of large numbers given on this wikipage.

So there you have it......one of the simplest and perhaps one of the most intuitive laws in statistics.