Think what your loyalty provides
TFS: Recognizing Anchors

TFS: How to avoid bad samples

One of the most common errors in my profession is making a decision based on a small sample, a too-small sample. So how do we know if the sample is too small? We can make the decision rationally, like a scientist, using System 2 thinking. Or we can go with our gut, which says no sample is too small. Any data is better than no data. 

Fortunes have been lost with that kind of thinking. Small samples are naturally more extreme than large samples. (Kahneman refers to this as the Law of Small Numbers.) Bad data is all around us. Random variation produces patterns that mislead us. Confidance bias encourages us to grasp at straws in the hope of proving our favorite theory. But remember, "grasping at straws" means trying to pull yourself up on something that can't hold your weight. 

Tq130402tvWhen I was in business school, the best course I took was statistics. What I learned is that I don't understand probability theory. I listened to the professor and reviewed the material, but it just didn't ever sound true. Now I know that is because System 1 finds statistical facts unsatisfying. Our brain seeks causes and "natural variance" is the equivalent of "no cause." 

Another related problem is the expense of large, reliable samples. In business, we have to keep our costs down. Scientists are plagued by this problem as well. The temptation to pretend a sample is reliable can be overwhelming. 

Solution: Think like a scientist even if you can't afford a reliable sample. Remember that variance is not a cop-out excuse, it's a real force of nature. Planes do not get off the ground by denying gravity...they deal with it. 

Comments