Extrapolating from Small Samples

Loading...

Do curso por University of Pennsylvania

People Analytics

classificações

University of Pennsylvania

People Analytics

classificações

Curso 3 de 5 em Specialization Análises Empresariais

People analytics is a data-driven approach to managing people at work. For the first time in history, business leaders can make decisions about their people based on deep analysis of data rather than the traditional methods of personal relationships, decision making based on experience, and risk avoidance. In this brand new course, three of Wharton’s top professors, all pioneers in the field of people analytics, will explore the state-of-the-art techniques used to recruit and retain great people, and demonstrate how these techniques are used at cutting-edge companies. They’ll explain how data and sophisticated analysis is brought to bear on people-related issues, such as recruiting, performance evaluation, leadership, hiring and promotion, job design, compensation, and collaboration. By the end of this course, you’ll understand how and when hard data is used to make soft-skill decisions about hiring and talent development, so that you can position yourself as a strategic partner in your company’s talent management decisions. Organizations flourish when the people who work in them flourish. Analytics can help make both happen. This course in People Analytics is designed to help you flourish in your career, too.

Na lição

Introduction to People Analytics, and Performance Evaluation

In this module, you'll meet Professors Massey, Bidwell, and Haas, cover the structore and scope of the course, and dive into the first topic: Performance Evaluation. Performance evaluation plays an influential role in our work lives, whether it is used to reward or punish and/or to gather feedback. Yet its fundamental challenge is that the measures we used to evaluate performance are imperfect: we can't infer how hard or smart an employee is working based solely on outcomes. In this module, you’ll learn the four key issues in measuring performance: regression to the mean, sample size, signal independence, and process vs. outcome, and see them at work in current companies, including an extended example from the NFL. By the end of this module, you’ll understand how to separate skill from luck and learn to read noisy performance measures, so that you can go into your next performance evaluation sensitive to the role of chance, knowing your environment, and aware of the four most common biases, so that you can make more informed data-driven decisions about your company's most valuable asset: its employees.

Introduction to People Analytics8:45

Goals for the Course1:40

Course Outline and Overview3:50

People Analytics in Practice4:55

Performance Evaluation: the Challenge of Noisy Data6:46

Chance vs. Skill: the NFL Draft22:08

Finding Persistence: Regression to the Mean11:56

Extrapolating from Small Samples5:53

The Wisdom of Crowds: Signal Independence5:56

Process vs. Outcome7:47

Summary of Performance Evaluation3:36

Conheça os instrutores

Cade Massey
Practice Professor
The Wharton School
Martine Haas
Associate Professor of Management
The Wharton School
Matthew Bidwell
Associate Professor of Management
The Wharton School

So the second issue I want to talk about is sample size and

want to begin with the psychology of sample size, essentially.

And this motivates our use of data to increase samples and

provide some caution on our use of small intuitive samples.

The first is the example, this is from a classic research study

where participants were asked the following vignette.

Your firm has two plants, one large and

one small, which mass produce a standard computer chip.

Other than the amount they produce,

the two plants are identical in all essential regards.

Both use the same technology to produce the same product.

When properly functioning this technology produces 1% defective items.

Whenever the number of defective items from one day's production exceeds 2%,

a special note is made in the quality control log to flag the problem.

At the end of the quarter,

which plant would you expect to have more flagged days in the quality control log?

So, two identical plants, big and

small, how many fall outside the quality control acceptable limits?

Everything else about the plant is the same,

it's just that one is larger than the other.

Does the small plant have more of these deviations,

does the large plant have more, or are they the same?

Because, after all, they're identical in every other respect.

So, if you ask people this about 22% say the smaller plant,

30% the larger plant, and then about half say the same.

Because in fact, all the technologies are the same,

why should have be any different.

What's the right answer?

The right answer is the small plant of course, because from small samples,

you get greater variation and what we're talking about here is variation.

These are production outcomes that fall outside the quality control limits.

Those are variations from expectation.

Small samples necessarily are more varied and people don't appreciate that.

And this is a major problem for performance evaluation

because we are apt to draw a too strong a conclusion from small samples.

Let's look at this in a little more detail.

Sample means converge to population means as the sample size increases.

If you've had a statistics course you might recognize this as the central

limit theorem.

So because this happens as we get larger and

larger samples the converse is exactly true that the smaller the sample,

the more likely you see more extreme values.

So there is a couple example questions.

What are you more likely to see, a .400 batting average in baseball,

which is a very good batting average, far higher than we typically see.

Do you see it at the beginning of the season around May 1st or

towards the end of the season, September 1st?

Well of course, you're far more likely to see such an extreme batting average

from a small sample.

The kind of sample you could have at the beginning of the season than

you would later.

What about hospitals?

Where are you likely to see dramatically higher percentage of boys than girls, or

vice versa, born on any given day?

A small community hospital, which might have 4-5 births a day?

Or a large city hospital which might have 100 births a day.

Where are you more likely to see the population mean vary in that daily sample?

The small hospital, of course, could see large variations.

You might have all girls one day, all boys another day in a small hospital.

There's no way you have that in a larger hospital.

We emphasize this because it should give us great caution in what we

say from small samples, but importantly it shows that we don't have that caution.

We tend to believe that what we see in a small sample is representative

of the underlying population.

We don't appreciate that a small sample can bounce around quite a bit.

And not in fact be representative of the underlying population.

This means that we should be very careful to not follow our intuition

on what it means when we only have a small sample.

This is a bias acknowledged in the literature called,

the law of small numbers, which is silly of course,

there is no law of small numbers, it's not like a law of the large numbers.

The small numbers in fact, what makes them interesting is that there is no law,

they bounce around.

People believe small samples closely share the properties

of the underlying population, but in fact they don't.

This means they too readily infer population's properties,

the average from the samples.

They neglect the role variability plays, chance plays, in these small samples.

What are the implications? What are the implications in

your organization?

This is a fundamental issue in performance evaluation, because often,

maybe even typically, performance evaluation is on small samples.

The smaller the sample,

the more we have to be cautious what kind of inference we draw from it.

So this is a place where data can make a big difference,

this is a place where by using more sources,

more history, using data as opposed to

just relying on our intuition we had the chance to collect larger samples.

But I don't want to say data alone is sufficient,

in fact it's interesting to note that in academia,

especially in psychology but now it's expanding to other disciplines as well.

We are discovering that we have been drawing too strong an inference from even

our academic research.

There's somewhat of a revolution going on in psychology,

now being picked up in other disciplines, that the main theme of which is,

we've been drawing too strong an inference from the samples we've been using,

so this isn't just academia saying be careful out there, you non-academics.

It's even academics that have had to learn that we too readily draw inferences

from samples that aren't really sufficient.

The lesson is almost always get more data, get bigger samples,

be careful about the inferences you draw from these small numbers, and

know that we have a bias towards jumping too quickly based on the small samples.

Explore nosso catálogo

Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.

O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.

© 2018 Coursera Inc. Todos os direitos reservados.

Baixar na App Store

Baixar no Google Play