As we've seen in many of the analyses we've talked about throughout this class,

there are a large number of steps that are involved in

doing a statistical genomics project from pre-processing and normalization,

to statistical modeling, to post hoc analyses of the results that you get.

So I wanted to talk a little bit about Researcher degrees of freedom.

This is an idea that was originally proposed in psychology, and there was this

paper that said, basically, undisclosed flexibility in data collection and

analysis allows for presenting anything as statistically significant.

And, so, what are they talking about here?

They're talking about how there's a large number of steps in the sort of

data analytic pipeline.

They go from experimental design, all the way from the raw data to

the summary statistics, and then finally there's a p-value at the end.

Now usually when people are talking about statistical significance,

they talk about p-values or multiple testing corrected p-values, and

often a lot depends on that p-value being

sort of small enough that a journal will publish the paper, or something like that.

And so that dependence is going down a little bit over time,

but originally there's been a lot of sort of focus on that.

But there's been a lot of sort of steps underneath that

process before you get to a p--value that could change what the p-value is.

So, for example, if you throw out a particular outlier, or if you normalize

the data a little bit differently, you might get different results.

And so, there's lots of different ways you can analyze data.

And the danger here is that, when they were talking about it in this paper,

they were sort of talking about a nefarious case where you

keep doing everything you can until you get a p-value that's significant, but

you could imagine doing this just sort of by accident.

You make a large number of choices when doing a genomic data analysis,

and once you've made those choices, you get some result.

And maybe you don't like that result so you redo the analysis.

So one thing that you have to be very careful about when doing

genomic analysis is redoing the analysis too many times.

It makes sense when there's new updated software or there's sort of new biological

or scientific knowledge that's been brought to bear to redo the analysis.

But if you keep redoing it over and over again you sort of fall into this trip.

And so, you can imagine how that would happen with different teams.

So, this comes from sort of a recent analysis.

This is an analysis in genomics, but

it kind of illustrates the point that 29 different research teams were asked to

see if referees were more likely to give red cards to dark-skinned players.

And so each team analyzed the data a little bit differently.

And here you can see the dots represent the different effect sizes that they

estimated for the different studies, and so you can see that they're all different.

And then the sort of confidence intervals,

or the sort of confidence uncertainty intervals, for

each of these different estimates are also different from each other.

And so, while they're comfortingly sort of similar for

many of the estimates here in the middle, you can get quite big variability

just by changing the way that you analyze the data.

And so, you have to be careful to make sure that you don't do this over and

over and over again until you find just the one case where you get a large

estimate of the effect, even if it's probably

not necessarily due to anything other than the way that you analyze the data.

And so, the difficult thing about thinking about that is if

you do a different analysis, particularly if you adjust for

different covariants, you actually are answering different questions.

So the a question is going to be conditional on what your sort of model is.

Ans so if you have whole bunch of extra covariants in the model,

then you're asking, is there a difference in gene expression after I account for

all of these other variables?

That's a very different question than, is there just a gene expression difference

overall, which might mean something totally different.

And so you have to be a little bit careful about this idea researcher degrees of

freedom as related to knowing what question it is that you're answering.

And so this whole idea was sort of summarizing in this paper by Andrew Gelman

and Eric Loken when they talk about The garden of forking paths.

What they mean by that is basically that you start off doing an analysis where you

just haven't seen the data, and maybe you have an analysis plan in mind.

Then once you collect the data you realize,

oh that there's a problem of a particular type.

This happens all the time in genomic data.

And then you start making decisions based on the data that you've observed, and

once you start doing that you start playing into this researcher degrees of

freedom idea.

You're basically changing the way that you're analyzing the data based on

the data, and you can end up with a little bit of trouble.

So the key is to be thinking ahead right from the beginning, how am I going to

analyze these data, what decisions am I going to make before looking at the data,

so that you're not sort of driven by those,

and sort of end up chasing a false positive.

So the key take home message here,

have a specific hypothesis that you're looking for.

So with genomic data there's this sort of tendency to just sort of do discovery for

the sake of doing discovery without a specific hypothesis.

And that can often lead towards this sort of garden of forking paths or

these researchers degrees of freedom.

Another thing that you can do is pre-specify your analysis plan,

that even if it's just internally to you, say like

this is the way we're going to analyze the data and we're going to stick to it.

And then even if you end up adapting it later,

it's good to just analyze the data once exactly how you planned on analyzing it,

even if it has problems, just so you know what would have happened, and

see if there's big differences and why those differences might be.