So the second issue I want to talk about is sample size and want to begin with the psychology of sample size, essentially. And this motivates our use of data to increase samples and provide some caution on our use of small intuitive samples. The first is the example, this is from a classic research study where participants were asked the following vignette. Your firm has two plants, one large and one small, which mass produce a standard computer chip. Other than the amount they produce, the two plants are identical in all essential regards. Both use the same technology to produce the same product. When properly functioning this technology produces 1% defective items. Whenever the number of defective items from one day's production exceeds 2%, a special note is made in the quality control log to flag the problem. At the end of the quarter, which plant would you expect to have more flagged days in the quality control log? So, two identical plants, big and small, how many fall outside the quality control acceptable limits? Everything else about the plant is the same, it's just that one is larger than the other. Does the small plant have more of these deviations, does the large plant have more, or are they the same? Because, after all, they're identical in every other respect. So, if you ask people this about 22% say the smaller plant, 30% the larger plant, and then about half say the same. Because in fact, all the technologies are the same, why should have be any different. What's the right answer? The right answer is the small plant of course, because from small samples, you get greater variation and what we're talking about here is variation. These are production outcomes that fall outside the quality control limits. Those are variations from expectation. Small samples necessarily are more varied and people don't appreciate that. And this is a major problem for performance evaluation because we are apt to draw a too strong a conclusion from small samples. Let's look at this in a little more detail. Sample means converge to population means as the sample size increases. If you've had a statistics course you might recognize this as the central limit theorem. So because this happens as we get larger and larger samples the converse is exactly true that the smaller the sample, the more likely you see more extreme values. So there is a couple example questions. What are you more likely to see, a .400 batting average in baseball, which is a very good batting average, far higher than we typically see. Do you see it at the beginning of the season around May 1st or towards the end of the season, September 1st? Well of course, you're far more likely to see such an extreme batting average from a small sample. The kind of sample you could have at the beginning of the season than you would later. What about hospitals? Where are you likely to see dramatically higher percentage of boys than girls, or vice versa, born on any given day? A small community hospital, which might have 4-5 births a day? Or a large city hospital which might have 100 births a day. Where are you more likely to see the population mean vary in that daily sample? The small hospital, of course, could see large variations. You might have all girls one day, all boys another day in a small hospital. There's no way you have that in a larger hospital. We emphasize this because it should give us great caution in what we say from small samples, but importantly it shows that we don't have that caution. We tend to believe that what we see in a small sample is representative of the underlying population. We don't appreciate that a small sample can bounce around quite a bit. And not in fact be representative of the underlying population. This means that we should be very careful to not follow our intuition on what it means when we only have a small sample. This is a bias acknowledged in the literature called, the law of small numbers, which is silly of course, there is no law of small numbers, it's not like a law of the large numbers. The small numbers in fact, what makes them interesting is that there is no law, they bounce around. People believe small samples closely share the properties of the underlying population, but in fact they don't. This means they too readily infer population's properties, the average from the samples. They neglect the role variability plays, chance plays, in these small samples. What are the implications? What are the implications in your organization? This is a fundamental issue in performance evaluation, because often, maybe even typically, performance evaluation is on small samples. The smaller the sample, the more we have to be cautious what kind of inference we draw from it. So this is a place where data can make a big difference, this is a place where by using more sources, more history, using data as opposed to just relying on our intuition we had the chance to collect larger samples. But I don't want to say data alone is sufficient, in fact it's interesting to note that in academia, especially in psychology but now it's expanding to other disciplines as well. We are discovering that we have been drawing too strong an inference from even our academic research. There's somewhat of a revolution going on in psychology, now being picked up in other disciplines, that the main theme of which is, we've been drawing too strong an inference from the samples we've been using, so this isn't just academia saying be careful out there, you non-academics. It's even academics that have had to learn that we too readily draw inferences from samples that aren't really sufficient. The lesson is almost always get more data, get bigger samples, be careful about the inferences you draw from these small numbers, and know that we have a bias towards jumping too quickly based on the small samples.
