In the last video of the last week,
we talked about Gaussian mixture models and how
they can be interpreted as models with a hidden state.
That gives the component that generates a given data point.
In this case, model estimation amounts to estimating both the hidden state s,
and means and variances of all Gaussian components.
We also said that this problem can be solved using
the EM algorithm that iterates between two steps.
The E step estimates the hidden variables given the observed variables.
The M step maximizes the low bound
on the likelihood of data by tuning all parameters in the model.
Now, I want to walk you through a few other latent variable models,
which will eventually pave the way to
our discussion of reinforcement learning in the second lesson of this week.
First, let's talk about factor analysis.
Let's assume we have data in the form of T observations over N-dimensional vector y.
Vector analysis seeks a decompositional signal y
into a weighted sum of some hidden or latent variables X,
that are assumed to be Gaussian with zero mean and unit variance, plus,
an N-dimensional white noise epsilon and which has a diagonal covariance matrix psi.
The equation reads y equals lambda x plus epsilon where lambda is emetic of size n by k,
and x is a K-dimensional vector.
Lambda is called the factor loading matrix as it gives weights
of different factors or components of X in the final observed value.
Now, because both x and epsilon Gaussian Y will also be Gaussian with
zero mean and its variance can be expressed as lambda times lambda transposed plus psi.
Now, what is the point of such the placing of
one Gaussian variable as a sum of two such variables?
Well, one reason for this is that we see that
our decompositional y provides a compact set of model parameters.
Indeed the number of tree parameters to describe a covariance matrix over
a general and dimensional vector y is n times n minus 1 divided by 2.
But the factor model leaves on the n times k parameters for matrix lambda
plus a k more parameters to describe variances of components of epsilon.
This makes N times K plus 1 parameters which is much less than the first series.
Also, we can compute the distribution of
hidden factors in factor analysis conditional on the observed data.
If we take a low number of factors,
this can also provide the low dimensional representation of your data.
There are also some subtle points about factor analysis.
The first point is that as it stands,
the factor model does not give a unique solution for lambda and x for given y.
To see this, let's assume that
the two given variables lambda and x are such that they provide a best fit to data y.
Now, assume that we have an arbitrary orthogonal matrix U,
so that U times U transposed equals a unit matrix.
Now, because U times transposed equals identity matrix,
we can ensure that just in between of w and x in this equation.
But then, we can rename the product lambda U as
lambda hat and also rename the product lambda transpose times x as x hat.
So, we get that the factor decomposition has
the same form as above but with different parameters around their necks.
It means that lambda and x are not unique for a given y.
To resolve this, some constraints need to be added to the definition of the factor model.
Usually, the factor loading matrix seeks constraint to be orthogonal.
One motivation for this is provided by a link
that exists between factor analysis and the PCA.
Namely, if we take the noise covariance matrix to be proportional to a unique matrix,
then it turns out that
the maximum likelihood estimation of the model produces the result that
the factor loading matrix should be a matrix of eigenvectors
of covariance matrix or y, stored column-wise.
Factor analysis model estimation can be done using the EM algorithm.
In a E-step, it estimates hidden factors
x while keeping the parameters from the previous step.
In the M-step, it adjusts the model parameters
by maximizing the low bound on the log likelihood.
Another latent variable model that I
wanted to briefly discuss here is called Probabilistic PCA.
Probabilistic PCA is a special case of factor analysis when we
take the noise covariance matrix to be proportional to a unit matrix.
Such noise is called isotropic.
Probabilistic PCA provides probabilistic generalization
of the conventional PCA exactly as its name suggests.
It can come very handy in many different, practically important situations.
For example, when your data has missing values or when
only certain components of the data vector are missing at some days.
Both these situations are very commonly encountered in finance.
Probabilistic PCA can work with such incomplete data
or it can be used to fill in missing values in data.
Also, because it's probabilistic,
it has much less issues with the outliers and noise in the data than the regular PCA.
Finally, the conventional PCA is produced from the results
of probabilistic PCA if you take the limit of sigma goes to zero.
Again, the probabilistic PCA can be estimated using
the MLE for the maximum likelihood estimation.
In the E-step, the algorithm makes inference of
the hidden state while keeping the model parameters fixed from the previous iteration.
In the M-step, it keeps the distribution over hidden nodes fixed and
optimizes over model parameters by maximizing the low bounds on the log likelihood.
Okay. This concludes our quick excursion into latent variable models.
Let's quickly check what we learned and then move on.
