5
October 2013Type I and Type II Errors | Smoke Detector and The Boy Who Cried Wolf
Every cook knows how to avoid Type I Error: just remove the batteries. Let's also learn how to reduce the chances of Type II errors.
Why Type I and Type II Errors Matter
A/B testing is an essential component of large-scale online services today. So essential, that every online business worth mentioning has been doing it for the last 10 years.
A/B testing is also used in email marketing by all major online retailers. The Obama for America data science team received a lot of press coverage for leveraging data science, especially A/B testing during the presidential campaign.
Here is an interesting article on this topic along with a data science bootcamp that teaches a/b testing and statistical analysis.
If you have been involved in anything related to A/B testing (online experimentation) on UI, relevance or email marketing, chances are that you have heard of Type I and Type II errors. The usage of these terms is very common but a good understanding of them are not.
I have seen illustrations as simple as this.
Examples of Type I and Type II Errors
I intend to share two great examples I recently read that will help you remember this very important concept in hypothesis testing.
TYPE I ERROR: An alarm without a fire.
TYPE II ERROR: A fire without an alarm.
Every cook knows how to avoid Type I Error – just remove the batteries. Unfortunately, this increases the incidences of Type II error.
Reducing the chances of Type II error would mean making the alarm hypersensitive, which in turn would increase the chances of Type I error.
Another way to remember this is by recalling the story of the Boy Who Cried Wolf.
Null Hypothesis: There is no wolf.
Alternate Hypothesis: There is a wolf.
Villagers believing the boy when there was no wolf (Rejecting null hypothesis incorrectly): Type I Error
Villagers not believing the boy when there actually was a wolf (Rejecting alternate hypothesis incorrectly): Type II Error
Tailpiece
The purpose of the post is not to explain type I and type II errors. If this is the first time you are hearing about these terms, here is the Wikipedia entry: Type I and Type II Error.
