In our last class, we saw how to obtain the least squares estimates of the parameters Beta in the linear regression model. I'm going to talk for just a little bit about the properties of these estimators and then we're going to look at an example. The method of least squares produces an unbiased estimate of the parameters Beta in the original linear regression model. That can be easily shown by simply taking the expected value of Beta hat and the expected value of Beta-hat would simply be the expected value of X prime X inverse times X prime Y. You can substitute for Y in this expression to get X prime X inverse X prime X Beta plus Epsilon. Now, we multiply this last part out and we get X prime X inverse times X prime X. That's just going to be an identity matrix and X prime X inverse times X prime Epsilon, that's going to have expected value of zero because epsilon has an expected value of zero. So it's pretty straightforward to see that the expected value of Beta hat is just Beta, so Beta-hat is unbiased. We can also express the variance property of Beta hat in a rather simple way. We could express it in terms of the covariance matrix of Beta hat. The matrix algebra is a little more difficult to show this but it turns out that the covariance matrix of Beta hat has a very simple form. It's Sigma squared times X prime X inverse. So if Sigma square is replaced by an estimate Sigma square hat, say, from what we looked at previously, we can get an estimate of the covariance matrix of Beta hat. The square roots of the main diagonals of that matrix are the standard errors of the model regression coefficients that you see reported in computer output. Of course, the off-diagonal elements represent the covariances of the model regression coefficients. So let's take a look at an example. This is example 10.1 from the textbook and we have 16 observations on polymer viscosity along with two process variables for reaction temperature and catalyst feed rate. That data is shown in the book in Table 10.2 and we're going to fit a multiple linear regression model to the data, so it's going to have the form Y equal Beta 0 plus Beta 1 X_1 plus Beta 2 X_2. So the X matrix and Y vector are as shown in the left-hand panel of this table. So all we really did to create the X matrix was to take the X_1 and X_2 columns out of the data-table, enter them in the matrix, and then add a column of ones to the left to account for the intercept and then Y is, of course, just the vector of all of the 16 response observations. So your X prime X matrix would be found by taking this matrix X, transposing it and then multiplying the transpose by X. A simple way to do that is to find the main diagonal elements of X prime X by simply squaring all of the elements in each column of the matrix and adding. So for example, if we square all of these ones, we get 16. If we square all of these numbers, we get the second diagonal element 133,560. If we square all of these numbers and add them up, we get the third diagonal element, 1,726. Now, the off-diagonal elements, to get those, multiply the two columns together and add. For example, the 1, 2 element of X prime X is the product of these two columns. So we multiply those two columns together row by row and add them up and that gives us this element. To get the third element, the 1.3 element, you would take this column times this column and multiply the entries row by row and add them up and that gives you the entry that you see here. This entry, can you guess how we get that? That's column 2 times column 3. We multiply those two columns together row by row and add them up. You'll notice that this X prime X matrix is symmetric. It's symmetric. So what does that mean? That means that all the entries that are below the main diagonal mirror the corresponding entries above that main diagonal. The X prime Y vector is found simply by multiplying the transpose of X times Y or if you want to do this real simply, you take each column in X, multiply that column 1 row by row by the entries in the Y vector and add them up and that gives you the quantities that you see here in X prime Y. So the least squares estimate of Beta hat is X prime X inverse times X prime Y. What we need is the inverse of X prime X and at the bottom of the slide, those are the calculations being displayed there. The X prime X matrix is calculated to quite a few decimal places of accuracy. Generally, you need to get a pretty accurate inverse in order to get accurate estimates of your model regression coefficients and here they are in that last vector that you see at the end. By the way, sometimes we write the actual prediction model with fewer coefficients. Here's the data-table, this is Table 10.2 from the book and here is that least-squares fit, model regression coefficients here were only really reported to two decimal places. The first three columns of table 10.3, they show you the observed value of Y and the predicted value of Y and then the third column are the residuals from that model fit. So the way I got the residuals, I took the observed value in each row and subtracted the corresponding predicted value from it. Figure 10.1 is a normal probability plot of the residuals from this model fit and Figure 10.2 is a plot of the residuals versus the predicted viscosity. The normal plot looks pretty good, the residuals seem to fall pretty much along a straight line. But if you look at Figure 10.3, there is some indication that the variability in viscosity may be increasing as the predicted value increases. Figure 10.3 shows you something very similar with respect to temperature. You notice that there's a bit of an indication that the variability in those residuals is increasing with temperature. Not so much the same is true with the plot of residuals versus feed rate. But there does seem to be a bit of an indication of inequality of variance with respect to the predicted value of viscosity and with the temperature. Now, computers are widely used to do this, of course, and JMP does a very good job of this, so does Minitab. I'm going to show you the Minitab output in just a moment. All of the quantities in that output should be familiar because they're similar to the output display quantities that we've seen in looking at computer analysis of data from design experiments. We've also seen how some of those quantities are computed in this lecture. We're also going to see some other examples in this chapter that show you how regression modeling can be very useful in looking at data from design experiments. Here is the Minitab output, and here, we have the model coefficient estimates, not quite the same number of decimal places that I used and what I showed you and here are the standard errors of the coefficients. Remember those are the square roots of the main diagonal elements of Sigma square hat times X prime X inverse. Here is the residual or error mean square that we talked about earlier. So some of these are quantities that we've already seen how they are computed and of course, Minitab does show you the fitted equation with the coefficients rounded to two decimal places.
