Machine learning is transforming the world around us. To become successful, you’d better know what kinds of problems can be solved with machine learning, and how they can be solved. Don’t know where to start? The answer is one button away. During this course you will: - Identify practical problems which can be solved with machine learning - Build, tune and apply linear models with Spark MLLib - Understand methods of text processing - Fit decision trees and boost them with ensemble learning - Construct your own recommender system. As a practical assignment, you will - build and apply linear models for classification and regression tasks; - learn how to work with texts; - automatically construct decision trees and improve their performance with ensemble learning; - finally, you will build your own recommender system! With these skills, you will be able to tackle many practical machine learning tasks. We provide the tools, you choose the place of application to make this world of machines more intelligent. Special thanks to: - Prof. Mikhail Roytberg, APT dept., MIPT, who was the initial reviewer of the project, the supervisor and mentor of half of the BigData team. He was the one, who helped to get this show on the road. - Oleg Sukhoroslov (PhD, Senior Researcher at IITP RAS), who has been teaching MapReduce, Hadoop and friends since 2008. Now he is leading the infrastructure team. - Oleg Ivchenko (PhD student APT dept., MIPT), Pavel Akhtyamov (MSc. student at APT dept., MIPT) and Vladimir Kuznetsov (Assistant at P.G. Demidov Yaroslavl State University), superbrains who have developed and now maintain the infrastructure used for practical assignments in this course. - Asya Roitberg, Eugene Baulin, Marina Sudarikova. These people never sleep to babysit this course day and night, to make your learning experience productive, smooth and exciting.
Introduction to large scale machine learning
Loading...
Do curso por Yandex
Big Data Applications: Machine Learning at Scale
38 classificações
Na lição
Spark MLLib and Linear Models
Conheça os instrutores
Vladimir Lesnichenko
Pavel MezentsevSenior Data Scientist
PulsePoint inc
Emeli Dral
Alexey A. DralFounder and Chief Executive Officer
BigData Team
Ilya TrofimovPrincipal Data Scientist
Yandex
Evgeny FrolovData Scientist, PhD Student @Skoltech
Computational and Data Intensive Science and Engineering
Hello, my name is Pavel,
and during this week,
you will learn how to use machine learning on big data.
Sounds interesting to you?
Let's go. This week,
you will learn why machine learning should be used on big data.
You will also know Spark MLlib,
and learn how to use linear models on large scale to predict events,
and learn some techniques for improving quality of prediction.
In this video, I will tell you how to solve the problem
of big data sampling in the right and the wrong way.
You will get the answer to the question,
why you need a large scale machine learning?
And why you should study Spark MLlib?
So, imagine that you have a huge data sets with billion of events,
the first idea that comes to my mind,
what if you take a small piece of the events from the data set,
load it into a local machine and work with it and somehow,
create models on it, make some predictions.
All that you did last week with the help of scikit-learn.
This idea is not so bad, to do this,
You need to pull out a random split,
random subset of the examples from your data set, and use them.
However, there is one pitfall I want to tell you about.
So, you have a giant log of user's activity in the Internet.
You don't load 10% of it to your computer,
and use to predict the gender of a person.
Suppose that a visitor chosen by us,
once read an article about fishing.
This can tell us that this is a man. "Stereotypes".
What is the problem here?
If a person made 100 clicks on your site,
then in the sample he will have about 10 of his clicks.
In the 10 clicks selected randomly,
the site about fishing will fall with probability of 10%,
and with a 90% probability,
we will miss this important event.
Sample correctly, in this case,
you need to sample by selecting not the events but the users who created these events.
Why do we need a complete data set?
Let's figure it out.
If you train your model on a very small data set,
your model can easily become overfitted.
Consider this example, where we distinguish boys and girls.
You train a complex model,
it determines correctly the classes for all the boys and girls,
but if you try to classify,
another data set using this model,
you will see that this quality is far from 100%.
Such an event is called overfitting,
and training on big data is one of the ways to solve this problem.
The second reason, why you need large data,
is a very complex model for solving your problem,
in which there are a lot of parameters to be automatically configured,
and a large data set will allow you to train it to correctly predict an event,
distinguish between kittens and dogs, and avoid mistakes.
The quality of the prediction as usual,
increase from a number of examples.
Therefore, the more data you have,
the more curious you get.
If you have a giant data set,
then trying to train the model on the entire data set,
and see what quality will be in this case, make sense.
Another case, where large data can really help you,
when you need to predict a rare event.
For example, if someone throw a coin,
and it stood upright, of course,
this can be predicted, because a coin is a random number generator.
But, this is a good example of rare event.
The facts are such rare events are very important,
and sometimes critical for business,
for example, the fall of service,
if you can write a program that predicts a drop half an hour before it happens,
you will save the company a lot of money.
99% of all operations in the bank are performed by honest, law abiding citizens.
However, amongst them, there a small
percentage of frauds who try to steal money from someone.
If you write a program that will catch such fraud,
banks will get their profit.
You need a lot of data here,
because a number of fraudulent operations,
compared to the number of legal ones, is vanishly small.
We have figured out why we need large data.
Now, why do we need a separate library of machine learning for large data?
What it will give for us?
First, it can build a predictive model using all data that it can get.
Terabytes, ten terabytes, petabytes of data,
only you need to deliver a sufficient number of machines.
Second, it can be distributed to apply
this model to the same terabytes and petabytes of data.
If your algorithm can classify your data within 1,000 hours,
then if you paralyze it on 1,000 processors,
the whole classification will take one hour,
compare the classifications that lasted 40 days and one hour,
the choice is up here.
So, complex machine learning models have a bunch of settings that you set by yourself.
The so-called hyper-parameters, machine learning on
large data allows you to run the selection of hyper-parameters,
distributing them across the cluster.
Some machines will train and check the quality of classification,
with one hyper-parameter,
another machine will train model with another hyper-parameter, thus,
you can advance of the fact that you have
a giant part of machines which you have access to.
So, what library do I talking about?
What kind of library do I mean?
Well, of course I'm talking about Spark MLlib.
In Spark 2.0, Spark MLlib works on top of the Spark SQL library.
You already know it.
You started it during the last course.
What are the advantages of Spark SQL?
The first advantage is simplicity.
Is there anybody here not to know how to use SQL?
It is integrated as a bunch of sources.
It can read data from files,
write data into files,
read from Hive, write to Hive,
use any external databases.
The third advantage is its performance.
The Spark SQL library runs 10 times faster than the MapReduce,
because it doesn't store the intermediate data on the disk,
and if you can cache the data,
save it into the memory,
then the rate of speeds will be treated as one to 100.
Just think about it.
What as advantages of Spark MLlib?
Actually, Spark MLlib was inspired by the library scikit-learn,
with which you have already got acquainted I hope.
What are its advantages?
First, unified way of processing data and apply in models.
Any model, any transformation,
they all are applied in the same way.
The second advantage is an easy deployment for large data.
You just need to run your program,
it will pump out your data set,
distribute it and process it,
and the third, a very important quality,
is a large number of sub-party libraries that are written by other people.
Often professional scientists, who are interested in developing something for Spark.
The fact is that Spark is now a hot topic,
and everyone interested and trying to write something for it.
That are the advantages.
And so, what have you learned today?
How to incorrectly and correctly circumvent the problem of large data?
Why you need a large scale machine learning,
and why you should study Spark MLlib?
Stay with us.
In the next lecture,
I will tell you how to train the simplest models on the Spark MLlib.
It will be interesting.
Explore nosso catálogo
Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.
O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.
© 2018 Coursera Inc. Todos os direitos reservados.
