In Behave, his book about the science of human choice, Robert Sapolsky cautions that the statistics he cites only emerge when you study very large groups of people and are not useful for predicting the behavior of individuals.
I have thought a lot about this idea because it is something that we often get wrong in our use of statistics. We have all gotten the message about the fallacy of generalizing from small samples. For example, you cannot determine if a coin is fair by flipping it just twice, and you cannot accurately guess the average age of the U.S. by asking 10 people how old they are. This is basically the law of large numbers.
What we don't often consider, though, is that the law of large numbers works both ways. You cannot generalize about an individual member of a large group only from what you know about that group's average. Yet, we do it almost constantly. We assume that an employee is going to prioritize work-life balance because she is part of "Gen Z." We think that a customer has a 25% chance of cancelling because that is the historical average for their cohort.
The problem with these assumptions is that averages work by noise cancellation. Your noise-cancelling headphones deliver distraction-free music not by blocking outside noise, but by counteracting it. When a car horn honks, it creates a spike in upward sound energy. Your headphones generate a matching dip downward. When the two waves combine, they cancel each other out. So, the average customer cancellation rate is the effect of a customer cancelling early counteracted by a customer cancelling late. Study a large enough dataset and you are left with the underlying "signal." But — and this is really important — it may take a LOT of data to get the signal to be meaningful. So, when you look at an individual customer, you do not know if you are looking at signal or noise. This is especially true if your dataset is noisy and/or small.
The point of all of this is that executives are too often offered assessments such as, "This deal has a 25% probability of closing because it is in the 'Evaluation' funnel stage." Or, "Based on historical averages, we can assume that half of the customers in this category will churn." Or even, "Based on industry survey results, we are recommending changes to our website." If you hear group statistics applied in this way, you need to be skeptical. These numbers may describe what happens across thousands of observations, but they say little about the outcome of one specific deal, this quarter's customers, or your unique website. Without context, group statistics are weak predictors of individual cases — they are a starting point for judgment, not the conclusion. We will discuss the best ways to use this data in later blog posts.
← Back to Writing