The Law of Large Numbers

After much confusion about what topic to post, I have decided to go with one of most basic and well known laws in statistics, the Law of Large Numbers. The motivation for this is an interesting read from Stephen Stigler's book titled, "The History of Statistics".

In Chapter 2 of this book, the author mentions that -"the consideration of games of chance led to the first mathematical treatment of the quantification of uncertainty". By the way, this is the 17th Century we are talking about. The early works focused on the quantifying uncertainty a priori, i.e., given that we have an unbiased coin, what is the probability of getting a heads? The answer is of course, p = 0.5, since an unbiased coin makes heads and tails equally likely!

However the a posteriori question of determining the bias on the coin given our observed or empirical data was addressed by one of the most renowned families in the history of mathematical sciences, the Bernoulli family.

Jacob Bernoulli (shown above) (source: Wikipedia), one of the greatest mathematicians of the 17th century, is responsible for the law of large numbers. It was found in his work, Ars Conjectandi, which was published posthumously by his nephew Nicholas Bernoulli.

Bernoulli proved that if X is the total number of observed successes for a total of N observations and if p is the unknown proportion or the "probability" of success, then for any given small {\epsilon} and any given large positive number c, N can be specified so that,  

The second inequality can be derived from the first inequality using simple algebra. We don't need to bother ourselves with the derivations of these inequalities. Here, {\epsilon} and c can be thought of as parameters that control the amount of accuracy desired in estimation and the idea is that N can be chosen as large as required to fulfill the above inequality.

Note that X/N denotes our estimate for the probability of success, while p denotes the actual or true probability of success. The term |(X/N) - p| denotes how far our estimate is from its true value. Hence an {\epsilon} = 1/50 implies that we want our estimate (X/N) to fall within 1/50 of its true value (p). The chance that our estimate *does not* fall within 1/50 of its true value is controlled by c. Thus for highly accurate estimate it is not difficult to see that we need to specify {\epsilon} as a small value and c as a large value. In this case, N needs to be specified as large as required for the above inequality to hold. This is the Bernoulli's weak law of large numbers.

A stronger version of this law (not credited to Bernoulli, I think) states that for a sequence of pairwise independent, identically distributed random variables say {X_{n}}, if S_{n} is the sum of all {X} from 1 to n,

if E(X) < infinity, then

The law simply states that as we get more information, i.e., as n tends to infinity, our estimate for the mean of the sample denoted by LHS of the above equation converges to the true mean or expectation of X denoted by the RHS. An elementary proof of the above result can be found here. Note that the independent and identically distributed random variable assumption is extremely important to apply this law in practice.

Before I end this post, I would recommend looking at the nice intuitive experiments for illustrating the law of large numbers given on this wikipage.

So there you have it......one of the simplest and perhaps one of the most intuitive laws in statistics.

Learning the Bayes-ics

The Indian movie buff may have his Rajnikanth, or perhaps you have a stack of Chuck Norris jokes. But the statistician will always revere Thomas Bayes (Gauss and the owner of this blog are free to disagree). And there's no joking about it.

The discerning reader would have picked up on my choice of the word "revere". As it turns out, in spite of being famous for his statistical prowess, Thomas Bayes was actually a Presbyterian minister! And his most famous theorem is (this might be a shocker), Bayes' Theorem :

P(A|B) = P(B|A)*P(A)/P(B)

where P(A|B) is the conditional probability of the event A given that the event B has already occurred. Despite its simplicity, the theorem is extremely useful. This is because, quite often we will be interested in the value of P(A|B), but it is more viable to measure P(B|A). Also, Bayes' theorem can throw up some pretty counter-intuitive results: the famous Taxicab problem - if there are 2 kinds of taxis: Green and Blue, a witness reports seeing a Blue taxi involved in an accident and we know that the witness' testimony is reliable 80% of the time then it must be the Blue taxi, right?!). In fact, Bayes' theorem has been shown to fool the gut instinct of seasoned professionals (see this for an interesting read). But enough of vague talk - let me make this concrete with an example(and this will just prove how counter-intuitive this can be):

Let's say Amit has just given an HIV test and it has sadly come back as positive. Is he doomed? Is there no hope?! Bayes to the rescue - let's examine this mathematically! 1% of the world population suffers from HIV, so Amit had a 0.01 "prior probability" of being HIV+ right from the get go. The test he just gave correctly identifies HIV+ people 96% of the time, but misdiagnoses a non HIV+ person as positive 8% of the time. Now that we have this information, we can apply Bayes' Theorem to get a probabilistic estimate on whether Amit really is HIV+.

A = Person is HIV+
B = Test returns positive

We know,
P(A) = 0.01
P(B|A) = 0.96
P(B| ~A) = 0.08

P(B) = P(B|A)*P(A) + P(B|~A)*P(~A) =  0.0888

We want to know P(A|B) - the probability that a person is HIV+ given that the test came back positive.
Well,
P(A|B) = 0.96 * 0.01/0.088 = 0.108

Shocking, right! This means, there's only a 10.8% chance that Amit actually has HIV, even though the test has come back as positive! Just goes to show the power of this theorem, and why it's one of my favorites.

Welcome to the Statland!

Hello all,

I am a graduating Masters Student in Computer Science from the University of Colorado, Boulder (CU). I would be pursuing my Ph.D. starting Fall 2012 in CU. One of my research interests is statistical inference, which is why I have decided to blog on this topic.

I have started this blog for purely selfish reasons! I would be attempting to explain some concepts in statistics that I find interesting. I believe that in trying to do so my understanding of these concepts would improve. Apart from the desire to hone my own skills in this field, I hope that my posts are of some help to you guys as well. 

This journey through the statistical world on which I welcome you aboard, would consist of text explanations, experiments, code, citations of technical papers, etc. 

Comments and/or criticisms would be highly appreciated! :)

PS: I am thankful to my friend Saurajit Mukherjee, who is also going to post in this blog to help me out!