Suppose you'd like to decide what degree of polynomial to fit to a data set, so

the, what features to include that gives you a learning algorithm.

Or suppose you'd like to choose the regularization parameter, lambda, for

learning algorithm, how do you do that? This is called model selection problems

and in our discussion of how to do this, we'll talk about not just how to split

your data into a train and test sets but how to split your data into what we'll

discover is called the training validation and test sets.

We'll see in this video just what these things are and how to use them to do

model selection. We've already seen a lot of times the

problem of overfitting, in which just because a learning algorithm fits a

training set well, that doesn't mean it's a good hypothesis.

More generally, this is why the training set error is not a good predictor for how

well the hypothesis would do on new examples.

Concretely, if you fit some set of parameters, theta zero, theta one, theta

two, and so on, to your training set, then the fact that your hypothesis does

well on the training set, well, this doesn't mean much in terms of predicting

how well your hypothesis will generalize to new examples not seen in the training

set. And a more general principle is that once

your parameters were fit to some set of data, maybe the training set, maybe

something else, then the error of your hypothesis is measured on that same data

set, such as the training error. That's likely to be a good estimate of

your actual generalization error. That is of how well the hypothesis will

generalize to new examples. Now, let's consider the model selection

problem. Let's say, you're trying to choose what

degree polynomial to fit to data. Should you choose a linear function, a

quadratic function, a cubic function, all the way up to a tenth order polynomial?

So, it's as if that's one extra parameter in this algorithm, which I'm going to

denote d, which is what degree of polynomial do you want to pick?

So, it's as if in addition to the theta parameter, it's as if there's one more

parameter D that you're trying to determine using a data set.

So, the first option is d equals one. If you fit a linear function, we can

choose d equals two, d equals three, all the way up to d equals ten.

So, we like to fit this extra sort of parameter, which I'm denoting by d.

And concretely. let's say that you want to choose a

model, that is, choose a degree or polynomial,

choose one of these ten models, and fit that model and also get some estimate of

how well your fitted hypothesis would generalize to new examples.

Here's one thing you could do. What you could, first, take your first

model and minimize the training error and this would give you some parameter data

of theta. And you could then take your second

model, the quadratic function and fit that to your training set and this will

give you some other parameter vector theta.

In order to distinguish between these different parameter vectors I'm going to

use a superscript one, superscript two there, where theta superscript one, just

means the parameters I get by fitting this model to my training data and theta

superscript two, just means the parameters I get by fitting this

quadratic function to my training data, and so on.

And by fitting a cubic model, I get parameters theta three up to, you know,

say, theta ten. And one thing you could do is that take these parameters and look

at the test set errors. I can compute on my test set, J test of

one, J test of say theta two, and so on,

J test of theta three, and so on.

So, I'm going to take each of my hypothesis with the corresponding

parameters and just measure their performance on the test set.

Now, one thing I could do then, is, in order to select one of these models, I

could then see which model has the lowest test sets error.

And let's just say for this example, that I ended up choosing the fifth order

polynomial. So, this seems reasonable so far, but now

let's say, I want to take my theta hypothesis, this, this fifth order model,

and let's say, I want to ask, how well does this model generalize?

One thing I could do is look at how well my fifth order polynomial hypothesis had

done on my test set. But the problem is, this will not be a fair estimate, of how

well my hypothesis generalizes. And the reason is,

what we've done is we fit this extra parameter d, that is this degree of

polynomial, and we fit that parameter d using the test set.

Namely, we chose the value of d that gave us the best possible performance on the

test set. And so,

the performance of my parameter vector theta five on the test set,

there's likely to be an overly optimistic estimate or generalization error,

right? So that because I have fit this parameter

d to my test set, it is no longer fair to evaluate my hypothesis on this test set.

It's because I've fit my parameters to the test set, I've chosen the degree d D

of polynomial using the test set. And so, my hypothesis is likely to do

better on this test set than it would on new examples that it hasn't seen before.

And that's, which is, which is what we care about.

So, just to reiterate, on the previous slide, we saw that if we fit some set of

parameters, you know, say, theta zero, theta one, to some training set, then the

performance of the fitted model on the training set is not predictive of how

well the hypotheses we generalized in new examples, is because these parameters

were fit to the training set so they're likely to do well on the training set,

even if the parameters don't do well on other examples. And in the procedure I

just described on the slide, we just did the same thing.

And specifically, what we did was we fit this parameter d to the test set,

and by having fit the parameter to the test set, this means that the performance

of the hypothesis on that test set may not be a fair estimate of how well the

hypothesis is, is likely to do on examples we haven't seen before.

To address this problem in a model selection setting, if we want to evaluate

a hypothesis, this is what we usually do instead.

Given the data set, instead of just splitting it into a train and test set,

what we are going to do is instead split it into three pieces, and the first piece

is going to be called the training set, as usual.

So, let me call this first part, the training set. Anf the second piece of

this data, I'm going to call the cross validation set,