In the previous lecture, I motivated the reason why we might care about statistics

by analysis that went wrong.

So in that analysis, as you'll recall,

they used genomic measurements to try to target chemotherapeutics or

try to decide which chemotherapies would apply best to which people.

And so that actually boiled down to two specific

reasons why that analysis went wrong.

And so one is lack of transparency.

So right from the start the data and

code used to perform that analysis were not made available.

In other words, the paper was published and people were looking at the paper and

trying to do the analysis themselves, and they couldn't get a hold of the raw data

or the process data, and they couldn't get a hold of the actual computer code or

the statistical code that was used to perform that analysis.

So this is related to the idea of reproducibility.

Can you actually re-perform the analysis that someone else created in their

original paper?

And so for the analysis that was done in the Duke example that I talked about.

There was actually not reproducible code.

The, it was not available.

You couldn't get a hold of it.

Similarly, there was a lack of cooperation.

And so this is actually not true in general, but

it was in this particular case, not only were the code and data not available, but

the people in charge of the code and data, the principal investigator and

the people that were the lead authors on the study were very reluctant to

hand the data over to statisticians, and other people to take a look at.

Now in every data analysis,

it's invariable that there are always some problems.

There's always some little issue that maybe people didn't notice.

But if the data and code aren't available, and not only that but the people that

performed the analysis aren't cooperative, and aren't sharing that data and code.

But it can take a very long time to discover if there's any

problems in a data analysis that are serious.

Like for example in the analysis that was done

in the case of the genomic signatures.

And so

the second thing that people would notice is that there's a lack of expertise.

And so the lack of expertise in this case dealt with specifically with statistics.

So one of the things that they used were very silly prediction rules.

So these are prediction rules where they defined probabilities in ways

that people wouldn't necessarily not only are they not right, but they sort of

are recognizably silly, so here's an example where I'm showing a probability

formula where there's a minus one-fourth in the formula sort of out of nowhere.

And so their prediction rules were based on these probability definitions

that not only weren't right, but were kind of silly.

And so that relates to a lack of statistical expertise.

The person who's actually developing the model haven't gone through

an actually done statistics class or perform enough analysis to know.

When they were doing something that wasn't not only right, but sort of silly to,

to even look at.

Another thing is that they have major study design problems.

So we'll talk about this in future lecture.

But they have things that we call batrifacts basically

they have run samples on different days.

And those different samples related to whether they would have one

particular outcome better than the other.

So it's called a confounder which we'll talk about.

But these study design problems at the very beginning before even

performing analysis, they'd set it up in a way they sort of set themselves up to

fail in the sense that the experimental design wasn't in place,

in a way that would allow them to do the analysis that they were hoping to do.

And so finally the predictions weren't locked down.

So in this case what, what happened was they had these prediction rules.

And the prediction rules stated, stated when you should apply

which chemotherapy to which person on the basis of the genomic measurements.

But because the prediction rules had a random component to them,

if you predicted on one day the probability of being assigned

to one treatment might be one number.

And on another day, you ran the exact same algorithm on the exact same code and you

would get a totally different prediction on which chemotherapy they should get.

And so this wasn't due to changes in the data or

changes in the statistical algorithm.

It was due just to changes in the day that the algorithm was running.

And obviously if you're running a clinical trial,

you don't people assigned to therapies based on sort of random chance.

So these are all issues where there was lack of statistical expertise, and

it turns out that the analysis was ultimately totally reproducible.

The statisticians at MD Anderson were able to chase down all of the details,

were able to put out all of the code, and all of the data that was originally used

to perform the paper, so it was a totally reproducible analysis.

The problem is that the data, the data analysis was just wrong.

And the reason why the data analysis was wrong was that there was a severe lack of

statistical expertise among the people that were performing the study.

So I hope I motivated that having the statistical expertise can help you avoid

before the problem even started, at the experimental design level,

at the level of creating correct statistical procedures.

Some of the problems that led to the scandal at Duke.

What Went Wrong?

Introduction to Genomic Technologies

Conheça os instrutores