0:01
In the previous lecture, I motivated the reason why we might care about statistics
by analysis that went wrong.
So in that analysis, as you'll recall,
they used genomic measurements to try to target chemotherapeutics or
try to decide which chemotherapies would apply best to which people.
And so that actually boiled down to two specific
reasons why that analysis went wrong.
And so one is lack of transparency.
So right from the start the data and
code used to perform that analysis were not made available.
In other words, the paper was published and people were looking at the paper and
trying to do the analysis themselves, and they couldn't get a hold of the raw data
or the process data, and they couldn't get a hold of the actual computer code or
the statistical code that was used to perform that analysis.
So this is related to the idea of reproducibility.
Can you actually re-perform the analysis that someone else created in their
original paper?
And so for the analysis that was done in the Duke example that I talked about.
There was actually not reproducible code.
The, it was not available.
You couldn't get a hold of it.
Similarly, there was a lack of cooperation.
And so this is actually not true in general, but
it was in this particular case, not only were the code and data not available, but
the people in charge of the code and data, the principal investigator and
the people that were the lead authors on the study were very reluctant to
hand the data over to statisticians, and other people to take a look at.
Now in every data analysis,
it's invariable that there are always some problems.
There's always some little issue that maybe people didn't notice.
But if the data and code aren't available, and not only that but the people that
performed the analysis aren't cooperative, and aren't sharing that data and code.
But it can take a very long time to discover if there's any
problems in a data analysis that are serious.
Like for example in the analysis that was done
in the case of the genomic signatures.
And so
the second thing that people would notice is that there's a lack of expertise.
And so the lack of expertise in this case dealt with specifically with statistics.
So one of the things that they used were very silly prediction rules.
So these are prediction rules where they defined probabilities in ways
that people wouldn't necessarily not only are they not right, but they sort of
are recognizably silly, so here's an example where I'm showing a probability
formula where there's a minus one-fourth in the formula sort of out of nowhere.
And so their prediction rules were based on these probability definitions
that not only weren't right, but were kind of silly.
And so that relates to a lack of statistical expertise.
The person who's actually developing the model haven't gone through
an actually done statistics class or perform enough analysis to know.
When they were doing something that wasn't not only right, but sort of silly to,
to even look at.
2:35
Another thing is that they have major study design problems.
So we'll talk about this in future lecture.
But they have things that we call batrifacts basically
they have run samples on different days.
And those different samples related to whether they would have one
particular outcome better than the other.
So it's called a confounder which we'll talk about.
But these study design problems at the very beginning before even
performing analysis, they'd set it up in a way they sort of set themselves up to
fail in the sense that the experimental design wasn't in place,
in a way that would allow them to do the analysis that they were hoping to do.
And so finally the predictions weren't locked down.
So in this case what, what happened was they had these prediction rules.
And the prediction rules stated, stated when you should apply
which chemotherapy to which person on the basis of the genomic measurements.
But because the prediction rules had a random component to them,
if you predicted on one day the probability of being assigned
to one treatment might be one number.
And on another day, you ran the exact same algorithm on the exact same code and you
would get a totally different prediction on which chemotherapy they should get.
And so this wasn't due to changes in the data or
changes in the statistical algorithm.
It was due just to changes in the day that the algorithm was running.
And obviously if you're running a clinical trial,
you don't people assigned to therapies based on sort of random chance.
So these are all issues where there was lack of statistical expertise, and
it turns out that the analysis was ultimately totally reproducible.
The statisticians at MD Anderson were able to chase down all of the details,
were able to put out all of the code, and all of the data that was originally used
to perform the paper, so it was a totally reproducible analysis.
The problem is that the data, the data analysis was just wrong.
And the reason why the data analysis was wrong was that there was a severe lack of
statistical expertise among the people that were performing the study.
So I hope I motivated that having the statistical expertise can help you avoid
before the problem even started, at the experimental design level,
at the level of creating correct statistical procedures.
Some of the problems that led to the scandal at Duke.