This course focuses on the concepts and tools behind reporting modern data analyses in a reproducible manner. Reproducible research is the idea that data analyses, and more generally, scientific claims, are published with their data and software code so that others may verify the findings and build upon them. The need for reproducibility is increasing dramatically as data analyses become more complex, involving larger datasets and more sophisticated computations. Reproducibility allows for people to focus on the actual content of a data analysis, rather than on superficial details reported in a written summary. In addition, reproducibility makes an analysis more useful to others because the data and code that actually conducted the analysis are available. This course will focus on literate statistical analysis tools which allow one to publish data analyses in a single document that allows others to easily execute the same analysis to obtain the same results.
Evidence-based Data Analysis (part 5)
Loading...
Do curso por Johns Hopkins University
Reproducible Research
2662 classificações
Na lição
Week 3: Reproducible Research Checklist & Evidence-based Data Analysis
This week covers what one could call a basic check list for ensuring that a data analysis is reproducible. While it's not absolutely sufficient to follow the check list, it provides a necessary minimum standard that would be applicable to almost any area of analysis.
Conheça os instrutores
Roger D. Peng, PhDAssociate Professor, Biostatistics
Bloomberg School of Public Health
Jeff Leek, PhDAssociate Professor, Biostatistics
Bloomberg School of Public Health
Brian Caffo, PhDProfessor, Biostatistics
Bloomberg School of Public Health
So the, my only thinking is that we should
just take this principle and apply it everywhere, right?
As much as possible to create analytic pipelines from evidence-based components.
So and to standardize it.
And so jeff has a, another contribution to this.
Is, he calls this the deterministic statistical machine.
The idea that you have a algorithm or pipeline that produces
a deterministic result given a set of data and a minimal,
minimal set of inputs.
And so once this evidence based pipeline
is established, you don't mess with it, alright?
So I call this, kind of, transparent box analysis, right?
So it's a box, so you don't mess with
it, it's like Al Gore's lock box, but it's transparent
because you know exactly what's happening in there, alright, so
it's not a black box, you can see what's happening.
But the point is, you don't mess with it.
So, one of the goals of this is
to reduce the so-called researcher degrees of freedom,
where people can tinker with every single tuning parameter
in the whole pipeline, which is, kind of, growing exponentially.
And to kind of get whatever they want out of the analysis.
So I think a lot, this is kind of analogous to
what something ideal with what it
is [INAUDIBLE] pre-specifies clinical trial protocol.
So when you run a clinical trail, you have to say
everything you're going to do, including how you're going to analyze the data.
And wouldn't it be nice if someone had just written this for you?
Well, if you use a
standardized kind of pipeline someone has written it for you.
And you just, you can pre-specify exactly what
you can do for him for a given situation.
So, the basic structure of this the cartoon of course will depend
on what you are actually, what question you're actually trying to ask.
But you have some data set some minimal input metadata perhaps
and then there's a series of modules that it goes through.
Which can be quite long in the reprocessing and sort of statistical
methods involved in all these pieces here.
and, and you can imagine that there'd be like a benchmark set of
data sets or data set that is used to evaluate various types of Statistical
methods and then you can produce some sort of report which perhaps can
be cut and pasted into a methods sections or posted in some public depository.
I just wanted to do a quick case study for what I mean by this is one of the
things I do a lot is estimate acute effects of avian air pollution exposure.
So so these are cor-relational types of studies a lot of this
stuff kind of originated from the 70's and 80's so it's not.
We start doing this yesterday, we have a long history of this kind of research.
The basic question, are short-term changes in pollution
associated with short-term changes in a population health outcome?
These are usually conducted at the level of the community.
And we have a lot, and there's a lot
of statistical research that has gone, some of it done
by myself, that has gone into this type of analysis.
So this is what the data typically look like.
We have, death counts, this is in New York City.
We're going for a couple years here.
And then you have particulate matter levels down here at the bottom.
Now you can see there's not as much particulate matter data
as there is mortality data, because there's a lot of missing data.
So, essentially you just want to say, is
this top thing correlated with the bottom thing.
So question is, can we encode
everything that we found in the statistical
and epidemiological research into a single package?
The answer is yes.
Time series studies like this don't have a huge range of variation.
And so they typically involve similar types of data.
You know, it might be hospitalization instead of
mortality or what not, but it's often very similar.
And so can we create kind of
a deterministic statistical machine for this area?
So the basic
kind of pipeline that looks, it's a very simple pipeline.
This is not a very complicated analysis for the most part.
You want to check the data, see if there
are any outliers, high leverage types of points.
Pollution data is often skewed, so you want to check for that.
Look for overdispersion.
Do you want to fill in the missing data? The answer is absolutely no.
There's been a lot of work on that.
It doesn't, it doesn't turn out well.
The big question really here is model selection, so one
of the things that we have to worry about is
called unmeasured confounding in these types of time series studies.
And you don't measure a lot of things that vary over time.
So I guess this is like your batch effects.
And so you have to, there are various approaches
to estimating of how you adjust these unmeasured confounders.
We use semiparametric progression methods to do this, so.
Estimating the degrees of freedom, this is, has the most profound effect
on any kind of association that you might estimate, so this is critical.
So, but, you know, there is lots
of research on how to do this.
There's been a couple of papers One is mine.
One of the various approaches to e, estimating
this number of degrees of freedom and there, and
you can settle on a, you know, one or
two approaches that are That are better than others.
So we can just implement that.
Other aspects of the model tend not to be that important.
Again, whether you adjust for temperature and weather and other
things it kind of doesn't really matter how you do that.
There's other things that you're typically interested
in, multiple lag analysis and sensitivity analysis.
So you want to see, you know, you can
select a model here, but if you want to see
if you can move the model back and forth
a little bit Does your association change dramatically, right?
So those are the typical things that you want to see, in this kind of analysis.
And when I review, you know, one paper a month, of, of
a time series study these are the things that I always ask for.
So. >> Roger.
>> Yeah.
>> Is there a 15 second response to why.
Computation is so bad in the setting? >> Oh, because the data
are missing systematically so the, the, the pollution data is very difficult to
collect, and so they typically only measure it only once every six days.
So there's five days missing for every six to eight days.
One observation for every six days.
So, you can try to imp-to inpute it but you just
add a lot of noise for a little bit of savings advice.
So my thinking, I, I feel like this is, you know, you can't have one obviously.
There's lots of various data analysis.
Different problems require different expertise.
I can't handle everything, obviously.
So the idea is you can create a curated library of these machines.
providing, kind of, the standardized, state
of the art, analysis type, well, just
state of the art in can have different meanings in different areas, obviously.
Time series analysis solutions are a very well developed
area, there's not a lot of research, statistical research,
going on, a lot of the questions have been settled.
So, we could have a, kind of, archive of these kinds of data analysis.
An archive for data analysis.
What analogy is this kind of Cochrane Collaboration which, I don't know if
any of you are familiar with but
it's basically an archive of evidence-based medicine.
So you can go to the Cochrane
Collaboration search for things like, you can just
ask, you can search for, you know, can you use vitamin C to treat asthma.
And so here, in the
summary it says. You know, we reviewed evidence.
So, they, they just basically did meta-analyses.
They reviewed evidence from nine clinical trials of Vitamin C.
[COUGH] In general, the evidence, the, the trails were very small.
It's not possible to recommend either the use or the avoidance of Vitamin C.
In asthma. Okay so.
So, we don't have to worry about vitamin C.
That's not going to help us, it's not going to hurt us.
So, something like, so this, this is
the basis of a lot of evidence-based interventions,
evidence-based treatments.
So something like this could be done for data analysis I think.
So, we can have a packages that encode the analyses
pipelines Curated by experts, you know, knowledgeable in the field.
These don't have to be perfect solutions but it the idea is that, they'd
be, they'd be pretty good solutions to kind of cover most of the bases.
And then so, that's kind of all
I wanted to say.
I, I think, so, I, I, I kind of did a bait and switch.
I didn't really talk about reproducible research.
But I do think it's important.
I, except for the fact I don't think
it Really fundamentally addresses the utmost important question.
Whether you can trust the analysis.
And so I think, so the I think, so
what I call this evidence based data analysis which
some people don't like that phrase but I just
come up with organized standardized best practices for given
scientific areas and questions.
And it's not that I don't think that they, I don't think they exist.
Especially it's really professional kind of
organizing and getting together and honest.
I think this is something that could give reviewers a
powerful tool and without dramatically increasing the burden on them.
Like you know, requiring them to reproduce the analysis would I think that we should
put more effort into kind of addressing
the up stream aspects of, of scientific research.
Explore nosso catálogo
Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.
O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.
© 2018 Coursera Inc. Todos os direitos reservados.
