By now you've seen a couple different learning algorithms, linear regression
and logistic regression. They work well for many problems, but
when you apply them to certain machine learning applications they can run into a
problem called over fitting that can cause them to perform very poorly.
What I'd like to do in this video is explain to you what is this over fitting
problem, and in the next few videos after this, we'll talk about a technique called
regularization, that will allow us to or to ameliorate, or to reduce this over
fitting problem and get these learning algorithms to work much better.
So, what is over fitting? Let's keep using our running example of
predicting housing prices with linear regression where we want to predict the
price as a function of the size of the house.
One thing we could do is fit a linear function to this data.
And if we do that, maybe we get that sort of straight line fit to the data.
But this isn't a very good model. Looking at the data, it seems pretty
clear that, as the size of the house increases.
The, housing prices plateau, or kind of flattens out as we move to the
right. And so, this, algorithm doesn't fit the
trading set very well, and we call this problem underfitting,
and another term for this is that this algorithm has high bias.
both of these roughly mean that it's just not even fitting the trading data very
well. The term bias is kind of a historical, or
technical one, but the idea is that if fitting a straight line to the data, then
it's as if the algorithm has a very strong preconception, or a very strong
bias that housing prices are going to vary linearly with their size.
And despite the data to the contrary, despite the evidence to the contrary, as
preconceptions still bias, still causes it to fit a straight line and this ends
up being a poor fit to the data. Now, in the middle,
we could fit a quadratic function to the data, and with this data set,
we fit a quadratic function, maybe we get that kind of curve, and that works pretty
well. And at the other extreme would be if we
were to fit say, a fourth of the polynomials of the data.
So here, we have five parameters, theta zero through theta four.
And with that, we can actually fill the curve that process through all five of
our training examples. We might get a curve that looks like
this. That, on the one hand, seems to do a very
good job fitting the trading set. It is processed through all of my data at
least, but this is sort of a very wiggly curve.
It's like going up and down going all over the place, and we don't actually
think that's such a good model for predicting housing prices.
So, this problem we call overfitting, and another term for this is that this
algorithm has high variance. The term high variance, is another, sort
of historical, or technical one, but the intuition is that, if we're
fitting such a high older polynomial, then the hypothesis can fit, you know, is
almost as if it can fit almost any function, and the space of the possible
hypothesis, is just to large, or is to variable.
And we don't have enough data to constrain it, to give us a good
hypothesis. So that's how overfitting, and in the
middle there isn't really a name, but so I'm just going to write, you know, just
write where a second degree polynomial, a quadratic function, seems to be just
right for fitting this data. To recap a bit, the problem of
overfitting comes when, if we have too many features,
then the learn hypothesis may fit the trading set very well.
So, your cost function may actually be very close to zero,
or maybe even zero exactly. But you may then end up with a curve like
this, that you know, tries too hard to fit the training set, so that it even
fails to generalize to new examples. And it fails to predict prices on new
examples well. And here the term generalize refers to
how well a hypothesis applies even to new examples.
That is to data, to houses that it hasn't seen in the training set.
On this slide, we looked at overfitting for the case of linear regression.
A similar thing can apply to logistic regression as well.
Here's the other logistic regression example, with two features, x1 and x2.
One thing we could do is fit logistic regression with just a simple hypothesis
like this, where, as usual, g is my sigmoid
function. And if you do that, you end up with a
hypothesis, trying to use maybe just a straight line to separate the positive
and the negative examples. And this doesn't look at a, like a very
good fit to the hypothesis. And so, once again, this is an example of
underfitting, or of a hypothesis having high bias.
In contrast if you were to add to your features these quadratic terms, then you
could get a decision boundary that might look more like this.
And, you know, that's a pretty good fit to the data,
probably what probably about as good as we could get on this training set.
And, finally at the other extreme, if you were to fit a very high order polynomial.
If you were to generate lots of high-order polynomial terms as features,
then the logistic progression may contort itself.
We tried really hard to find a decision boundary that
fits your training data, or go to great lengths to contort a cell to fit every
single training example well. And, you know, if the features X1 and X2
are for predicting maybe the cancer tumor, you know cancer is a malignant
benign breast tumors. This doesn't, this really doesn't look like a very good
hypothesis for making predictions. And so once again, this is an instance of
overfitting, and of hypothesis having high variance and not really, and being
unlikely to generalize well to new examples.
Later in this course, when we talk about debugging and diagnosing things that can
go wrong with learning algorithms. We'll give you specific tools to
recognize when over fitting and also when under fitting may be occurring.
But for now, let's talk about the problem of, if we think overfitting is occurring,
what can we do to address it? In the previous examples, we had one or
two dimensional data, so we could just plot the hypothesis and see what was
going on and select the appropriate degree polynomial.
So earlier, for the housing prices example, we could
just plot the hypothesis and you know maybe see that it was fitting this other
very wiggly function that goes all over the place predicting housing prices and
we could then use figure like these to select an appropriate degree polynomial.
So plotting hypothesis could be one way to try to decide what degree polynomial
to use, but that doesn't always work, and in fact more often,
we may have learning problems that, where we just have a lot of features.
And there is not just a matter of selecting
what degree polynomial, and in fact, it, when we have so many features, it also
becomes much harder to plot the data, and becomes much harder to visualize it, to
decide what features to complete or not. So concretely, if we're trying to predict
housing prices, sometimes we can just have a lot of different features, and all
of the features seem, you know, maybe they seem kind of useful, but if we have
a lot of features and very little training data, then overfitting can
become a problem. In order to address overfitting, there
are two main options for things that we can do.
The first option is to try to reduce the number of features.
One thing that we could do is manually look at the list of features and use that
to try to decide which are the more important features, and therefore which
are the features we should keep and which of the features we should throw out.
Later in this class we'll also talk about model selection algorithms which are
algorithms. But automatically deciding which features
to keep and which features to throw out. This idea of reducing the number of
features can work well and can reduce over fitting and when we talk about model
selection we'll go into this in much greater depth.
But a disadvantage is that by throwing away some of the features, it's also
throwing away some of the information you have about the problem.
For example, maybe all of those features are actually useful for predicting the
price of a house, so maybe we don't actually want to throw some of our
information or throw some of our features away.
The second option, which we'll talk in, which we'll talk
about in the next few videos, is regularization.
Here, we're going to keep all the features that we're going to reduce the
magnitude, or the values of the parameters, theta j.
And this method works well, we'll see, when we have a lot of features.
Each of which contributes a little bit to predicting the value of y.
Like we had, like we saw in the housing price prediction example,
where we could have a lot of features, each of which are, you know, somewhat
useful, so maybe we don't want to throw them
away. So this describes the idea of
regularization at a very high level, and, I realize that all of these details
probably don't make sense to you yet. But in the next video we'll start to
formulate exactly how, to apply regularization and exactly what
regularization means. And, then we'll start to figure out how
to use this to make our learning algorithms work well and avoid
overfitting.
