This course covers the essential exploratory techniques for summarizing data. These techniques are typically applied before formal modeling commences and can help inform the development of more complex statistical models. Exploratory techniques are also important for eliminating or sharpening potential hypotheses about the world that can be addressed by the data. We will cover in detail the plotting systems in R as well as some of the basic principles of constructing data graphics. We will also cover some of the common multivariate statistical techniques used to visualize high-dimensional data.
Dimension Reduction (part 2)
Loading...
Do curso por Johns Hopkins University
Exploratory Data Analysis
4333 classificações
Na lição
Week 3
Welcome to Week 3 of Exploratory Data Analysis. This week covers some of the workhorse statistical methods for exploratory analysis. These methods include clustering and dimension reduction techniques that allow you to make graphical displays of very high dimensional data (many many variables). We also cover novel ways to specify colors in R so that you can use color as an important and useful dimension when making data graphics. All of this material is covered in chapters 9-12 of my book Exploratory Data Analysis with R.
Conheça os instrutores
Roger D. Peng, PhDAssociate Professor, Biostatistics
Bloomberg School of Public Health
Jeff Leek, PhDAssociate Professor, Biostatistics
Bloomberg School of Public Health
Brian Caffo, PhDProfessor, Biostatistics
Bloomberg School of Public Health
So if you look at, if we break down the
components of the singular value decomposition we can take, look at
the u, the kind of the u and the v
matrix of the left singular values and the right singular values.
So you, if you look at the original data matrix here on
the left, so that's the image plot here that, that we saw before.
And then on the, in the middle plot here, I plot the, the left, the left,
the first, sorry, the first left singular vector.
And it kind of roughly shows basically the mean of all the, of that kind of data set.
So if you look, if you plot, if you plot it across the rows, you can see that the
first singular vector is kind of, has a negative value.
For rows 40 through about 18 or so or 17.
And then it has a positive value for the remaining rows.
So that shows the clear separation in the means of the two sets of of rows, right.
And if you look at the the, the, the
right sing, the first right singular vector you can see
that this also shows the change in the mean between
the the first five columns and the second five columns.
And so the nice thing
about the singular value decomposition here is that
it immediately, it immediately picked up on the shift
in the means, both kind of in the
row, from the row dimension and the column dimension.
Without you having to tell it anything.
It was totally unsupervised, and it just kind of picked it up automatically.
So remember, we made a plot that was very, very similar to this before, but that
was be, when we knew that there was this kind of pattern in the data set.
So here the, the singular value decomposition
picked up the pattern right away.
And, and, and in the first kind
of singular vet, left and right singular vectors.
Another kind of component of the singular value
decomposition is, is known as the variance explained.
And this comes from the singular values that are in the d matrix.
Remember the d matrix, is really a diagonal matrix.
It only has elements that are on the diagonal of that matrix.
And you can think of each singular value as representing the percent of
the total variation in your data
set that's explained by that particular component.
And so, and the components are typically ordered so that the first one explains
the most variation as possible, and the
second one explains the second most, et cetera.
And so you can plot the kind of proportion of
variance explained as we have in the in this plot below.
And so here on the left hand side I've just plotted the raw singular values
and you can see that they kind of decrease in value as you go across the columns.
But of course the raw singular
value doesn't really have much meaning cause it's not on an interpretable scale.
So if I divide by the total sum of
all the singular values, then on the right hand side
I've got the, I can interpret it as the
proportion of the variance explained, and you can see that's
exactly the same plot but the y axis has
changed, and so you can see that the first singular
value if you recall it captures kind of the shift
and the mean in between the rows and the columns.
That,
that captures about 40% of the variation in your data.
Alright, so almost almost half of the variation in the
data is explained by a single kind of singular value.
Or you can think of it as a single dimension.
And then, the remaining variation in the data is explained by the other components.
So, just to show that the relationship between the
singular value decomposition and the principle components is close, see,
I, I, here, I plotted, I've calculated the SVD
and the, and the PCA of the same data matrix.
And I've plotted the principle components on the, X axis, and the
for, sorry, the principle component, first principle component on the X axis.
And the first right singular val, vector on the y-axis.
And you can see that they fall exactly in a line that
they're exactly the same things.
So the svd and the principle components, and
principle components analysis is essentially the same thing.
And so if you hear, you know, you're in a cocktail party and you hear two people
talking about the svd and pca you can
rest assured that they're basically doing the same thing.
So if we look at, if we look, take a closer look at the variance explained.
Here we've got a very simple example which is just basically
a, kind of a matrix that has either zeros or ones.
And so the idea is that there's basically
there's only one pattern in this matrix, right?
And so the first five columns are zeros.
And the second five columns are wants that's it, nothing special and so if I
plot the data you can see on the left hand side that the, left part,
the first five are red with the further kind of second five are yellowish white.
So now if I take the SVD of this, kind of,
relatively boring matrix, you can see that I can plot the
singular values and you can see that there's one singular value
that's very high and the rest are kind of zero, basically.
And what you'll see is that the first singular
value explains 100% of the variation in the data set.
So how can that be?
Well, if you think about this data set, even though there is, kind of,
40 rows andten columns, there's really only one dimension to this data set, which
is that, if you're in the first five columns, your mean is, you're equal
to 0, and if you're in the second five columns, you're equal to 1.
So even though you have lots of so-called observations, there's really only
kind of a small fraction of information that's actually useful in this matrix.
And so that's captured by the SVD.
By the fact that you can explain 100%
of the variation with a singular, a single component.
So let's add a second pattern to the data set.
So we can add a pattern that's kind of that's
kind of goes across the rows and also goes across the columns.
And so one pat, the first pattern to be,
going to become a block pattern that we saw before.
So the first half of this is going to have one
mean and one half of this is going to have another mean.
And the second
column, the second pattern, basically, is going to alternate between the columns.
So you can see what that looks like over here.
So on the left hand side that's the data and you can see that there's one pattern.
That's kind of like, that's kind of a block pattern so the
left five columns are low and the right five columns are higher.
And then you can see that, within that, kind of nested within
that, is a kind of pattern that's kind of every other column.
And so, so the, every other column is, one is higher than the other.
And so if you plot, if we, if you plot the kind
of, what the truth is, right, we can see that on the,
in the middle plot here, the first five columns have a have
a lower mean, and the second five columns have a higher mean.
And that's the first pattern, and then the second pattern shows that the first column
has a mean of 0, the second column has a mean of 1.
The third column as
a mean of 0 and the fourth column et cetera.
So there's, there's two patterns here.
One is a shift pattern and the second is
this kind of alternating pattern, so that's the truth.
But of course, we rarely know the truth, so
we need to learn the truth from the data.
So if you, the idea is if you were presented
with the data set that we've shown here on the left.
Can you come up with an algorithm that can pick up
on the two separate patterns that are buried within the data?
And so that's
kind of what the SVD is for.
So if we run the SVD on this new matrix with the two patterns.
I got the data here on the left and the middle pan,
the middle panel here, I've got the first, the first right singular vector.
And you can see it roughly picks up the block pattern.
Right?
So the first, kind of, five are, are
somewhat lower, and the second five are somewhat higher.
Right? So it's not as pretty as the truth was.
But it's somewhat discernible, that there are two different kind of means here.
The second sorry, the second right singular
vector is on the right-hand panel here.
And you can see that, it's a little bit harder to see,
but it does try to pick up the alternating mean pattern, right?
It's not as obvious as it was when we were plotting the truth, of course.
But you can see that every other point is either higher or lower.
Now of course since we know what the truth is
it's a little bit easier to, kind of, talk about this.
But but in general you can see that the two patterns are roughly confounded
with each other because it's a little bit hard to set the two patterns apart.
So even though they're clearly in the, we
know that the truth is, is, is two separate
patterns, the, the first and the second right
singular vectors are also known as the principle components.
They kind of mix the two patterns together,
so each of them has a block pattern.
And each of them has a kind of alternating pattern.
And so unfortunately, just, just like with most real data, the truth is a
little, is a little bit hard to discern than if you'd known it in advance.
Now if you look at the variance explained in this problem
you can see that on the right-hand panel which is the
percent of the variance explained, that the first component explains over
50% of the variation, the total variation in the data set.
And that's basically because the shift pattern you know, with
the first five columns, the second five columns is so strong.
It represents a large amount of variation in the data set.
So you capture that shift
pattern, you kind of capture a lot of that variation.
You see the second component only captures about 18 or so
percent of the variation and it kind of trails off from there.
And so that's going to tell, that kind of tells you roughly,
you know, how many components are in this data set.
But that alternating pattern which kind of, which is kind of every other
column is a little bit harder to pick up, as you can see.
Explore nosso catálogo
Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.
O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.
© 2019 Coursera Inc. Todos os direitos reservados.
