Gradient Boosted Decision Trees

Loading...

Do curso por University of Michigan

Applied Machine Learning in Python

2093 classificações

University of Michigan

Applied Machine Learning in Python

2093 classificações

Curso 3 de 5 em Specialization Applied Data Science with Python

This course will introduce the learner to applied machine learning, focusing more on the techniques and methods than on the statistics behind these methods. The course will start with a discussion of how machine learning is different than descriptive statistics, and introduce the scikit learn toolkit through a tutorial. The issue of dimensionality of data will be discussed, and the task of clustering data, as well as evaluating those clusters, will be tackled. Supervised approaches for creating predictive models will be described, and learners will be able to apply the scikit learn predictive modelling methods while understanding process issues related to data generalizability (e.g. cross validation, overfitting). The course will end with a look at more advanced techniques, such as building ensembles, and practical limitations of predictive models. By the end of this course, students will be able to identify the difference between a supervised (classification) and unsupervised (clustering) technique, identify which technique they need to apply for a particular dataset and need, engineer features to meet that need, and write python code to carry out an analysis. This course should be taken after Introduction to Data Science in Python and Applied Plotting, Charting & Data Representation in Python and before Applied Text Mining in Python and Applied Social Analysis in Python.

Na lição

Module 4: Supervised Machine Learning - Part 2

This module covers more advanced supervised learning methods that include ensembles of trees (random forests, gradient boosted trees), and neural networks (with an optional summary on deep learning). You will also learn about the critical problem of data leakage in machine learning and how to detect and avoid it.

Naive Bayes Classifiers8:00

Random Forests11:42

Gradient Boosted Decision Trees5:53

Neural Networks19:02

Deep Learning (Optional)7:00

Data Leakage11:25

Conheça os instrutores

Kevyn Collins-Thompson
Associate Professor
School of Information

Another tree based ensemble method that's gain wide use in real world application is

gradient boosted decision trees.

Like random forest, gradient boosted trees used an ensemble of multiple tress to

create more powerful prediction models for classification and regression.

In this lecture,

we'll provide a brief overview of gradient boosted decision trees,

along with the discussion of their key parameters, the control model complexity.

Unlike the random forest method that builds and

combines a forest of randomly different trees in parallel, the key idea

of gradient boosted decision trees is that they build a series of trees.

Where each tree is trained, so

that it attempts to correct the mistakes of the previous tree in the series.

Typically, gradient boosted tree ensembles use lots of shallow trees known in

machine learning as weak learners.

Built in a nonrandom way, to create a model that makes fewer and

fewer mistakes as more trees are added.

Once the model is built, making predictions with a gradient boosted

tree models is fast and doesn't use a lot of memory.

Like random forests, the number of estimators in the gradient boosted tree

ensemble is an important parameter in controlling model complexity.

A new parameter that does not occur with random forest is something called

the learning rate.

The learning rate controls how the gradient boost the tree algorithms,

builds a series of collective trees.

When the learning rate is high, each successive tree

put strong emphases on correcting the mistakes of its predecessor.

And thus may result in a more complex individual tree, and

those overall are more complex model.

With smaller settings of the learning rate, there's less emphasis on thoroughly

correcting the errors of the previous step,

which tends to lead to simpler trees at each step.

Here's an example showing how to use gradient boosted trees in scikit-learn on

our sample fruit classification test, plotting the decision regions that result.

The code is more or less the same as what we used for random forests.

But from the sklearn.ensemble module,

we import the GradientBoostingClassifier class.

We then create the GradientBoostingClassifier object, and

fit it to the training data in the usual way.

By default, the learning rate parameter is set to 0.1, the n_estimators parameter

giving the number of trees to use is set to 100, and the max depth is set to 3.

As with random forests, you can see the decision boundaries have

that box-like shape that's characteristic of decision trees or ensembles of trees.

Now let's apply gradient boosted decision trees to the breast cancer dataset.

This code trains two different gradient boosted classifiers.

The first one uses the default settings.

We can see that the first result has perfect accuracy on the training set,

which indicates the model is likely overfitting.

Two ways to learn a less complex gradient boosted tree model are,

to reduce the learning rate, so that each tree doesn't try as hard to learn a more

complex model, that fixes the mistakes of its predecessor.

And to reduce the max_depth parameter for the individual trees in the ensemble.

The second classifier example makes these changes in the parameters.

And you can see, that the training set accuracy does decrease,

while the test set accuracy increases slightly.

Gradient boosted decision trees are among the best off-the-shelf supervised learning

methods available.

Achieving excellent accuracy with only modest memory and runtime requirements to

perform prediction, once the model has been trained.

Some major commercial applications of machine learning have been based on

gradient boosted decision trees.

Like other decision tree based learning methods,

you don't need to apply feature scaling for the algorithm to do well.

And the futures can be a mix of binary, categorical and continuous types.

Boosted decision trees do have several downsides.

So like random forests, ensembles of trees are very difficult for

people to interpret, compared to individual decision trees.

However, this often may not matter for

many applications where prediction accuracy is the most important goal.

Gradient boosted methods can require careful tuning of the learning rate and

other parameters.

And the training process can require a lot of computation.

And like the other tree based methods we saw, using gradient boosted methods for

text classification or other scenarios.

Where the featured space has thousands of features with sparse values,

is usually not a good choice for accuracy and computational cost reasons.

The key parameters controlling model complexity for gradient boosted tree

models are, n_estimators which sets the number of small decisions trees the week

learns to use in the ensemble, and the learning rate.

Typically, these two parameters are tuned together.

Since making the learning rates smaller,

will require more trees to maintain model complexity.

Unlike random forest, increasing an n_estimators can lead to overfeeding.

So typically, the n_estimators setting is chosen to best exploit the speed and

memory capabilities of the system during the training.

And other parameters like the learning rate are then adjusted,

given that fixed an n_estimators setting.

The max_depth parameter can also have an effect of model complexity, but

controlling the depth, and has a complexity of the individual trees.

The gradient boosting method assumes, that each trees is a weak learner, and so

the max_depth parameter is usually quite small, on the order of three to five, for

most applications.

Explore nosso catálogo

Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.

O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.

© 2018 Coursera Inc. Todos os direitos reservados.

Baixar na App Store

Baixar no Google Play