Have you ever heard about such technologies as HDFS, MapReduce, Spark? Always wanted to learn these new tools but missed concise starting material? Don’t miss this course either! In this 6-week course you will: - learn some basic technologies of the modern Big Data landscape, namely: HDFS, MapReduce and Spark; - be guided both through systems internals and their applications; - learn about distributed file systems, why they exist and what function they serve; - grasp the MapReduce framework, a workhorse for many modern Big Data applications; - apply the framework to process texts and solve sample business cases; - learn about Spark, the next-generation computational framework; - build a strong understanding of Spark basic concepts; - develop skills to apply these tools to creating solutions in finance, social networks, telecommunications and many other fields. Your learning experience will be as close to real life as possible with the chance to evaluate your practical assignments on a real cluster. No mocking, a friendly considerate atmosphere to make the process of your learning smooth and enjoyable. Get ready to work with real datasets alongside with real masters! Special thanks to: - Prof. Mikhail Roytberg, APT dept., MIPT, who was the initial reviewer of the project, the supervisor and mentor of half of the BigData team. He was the one, who helped to get this show on the road. - Oleg Sukhoroslov (PhD, Senior Researcher at IITP RAS), who has been teaching MapReduce, Hadoop and friends since 2008. Now he is leading the infrastructure team. - Oleg Ivchenko (PhD student APT dept., MIPT), Pavel Akhtyamov (MSc. student at APT dept., MIPT) and Vladimir Kuznetsov (Assistant at P.G. Demidov Yaroslavl State University), superbrains who have developed and now maintain the infrastructure used for practical assignments in this course. - Asya Roitberg, Eugene Baulin, Marina Sudarikova. These people never sleep to babysit this course day and night, to make your learning experience productive, smooth and exciting.
Estimating proportions
Loading...
Do curso por Yandex
Big Data Essentials: HDFS, MapReduce and Spark RDD
151 ratings
Conheça os instrutores
Ivan PuzyrevskiyTechnical Team Lead
Emeli Dral
Evgeniy Riabenko
Alexey A. DralHead of Big Data and Machine Learning
Algorithms and Programming Technologies dept. MIPT
Let's introduce some notation and
mathematical facts about the process of sampling and its error.
There's a profound theory behind this, but don't worry,
we won't go deep now,
you'll get just enough of statistics to use it than sampling big data.
Let's start with the simplest case.
Say you have a binary feature,
the one that could only take two values.
For example, in a database of criminal records,
it might be an indicator of whether their committed crime is violent or non-violent.
In a housing data set,
a binary feature might described if the apartment has a canal view,
or whether it has a roof terrace.
All the features could possibly take only two values.
Canal is either in front of the house or not.
For convenience, we'll encode those values with zeros and ones.
Now, then you have a feature like that,
you might be interested in the proportion of its values.
What is the percentage of the apartments with roof terraces?
What proportion of crimes are violent?
If you have the whole data set,
you could just calculate that proportion,
but could the same quantity be estimated from just a small piece of data?
Sure. Let's take N items from our data set.
Denote X with a superscript N,
a sample of size N. Elements of the sample denote as X with
subscripts from one to N. What is the proportion of ones in this sample?
It is actually the same as the average of all Xs.
It seems logical to use this average p-hat as a sample estimate of
the unknown proportion of ones in the whole data set P. Indeed,
if we had a whole data set we would have used the same average to calculate P itself.
It seems reasonable to apply the same operation to the sample to calculate the estimate.
This logic does not always work,
but in this case it does.
By the way, an estimate like that,
the one that estimates the value of the unknown parameter
with just one number is called a point estimate.
Could an estimate possibly be something other than one number?
You'll see very soon.
Let's get back to our sample of 100 taxi trips,
and the question of what percentage of passengers leaves tips.
There is a column called tip amount in the data.
Let's create a binary vector is tipped with
ones indicating whether these columns values are above zero.
The mean of this vector and our estimate
of the proportion of the tipping customers in the whole data set will be 0.66,
about 13 times more than I expected.
Now, is that estimate P Hat good?
Meaning is it close to the unknown value of P,
the proportion of ones over the whole data set?
Well, that actually depends on the sample itself.
First of all it must be random.
That's why we need to shuffle the rows.
Second, the bigger the sample of course the better the estimate,
the closer it is to the true value of the parameter
P. If you have just one object in the sample,
your estimate could only possibly be zero or one.
Given two objects you could also get 0.5 and so on.
The set of all possible values of P Hat grows with N. This is just common sense.
Bigger sample, better estimates.
But can we make this statement more precise?
Can we actually quantify the accuracy of the estimate?
There is a way to do that.
First of all, for the estimator you could
calculate the quantity called standard deviation.
It is a measure of the spread of the values your estimate could
take on all possible samples across its mean value.
For the proportion estimate P Hat.
The standard deviation is approximately equal to the square root of
the P Hat times one minus P Hat divided by N. In our taxi tipping sample,
the standard deviation is equal to 0.05 approximately.
But is it a lot,
or is it a little?
To eliminate that vagueness we need one more concept, confidence interval.
For the parameter P. It is a pair of functions of the sample CL and CU,
such that an interval from CL to CU covers
P with probability not less than one minus alpha.
Usually Alpha is set to 0.05,
and then the confidence interval is called 95% confidence interval.
It means that if we calculate such intervals on a hundred different random samples,
about 95% of them will cover the true unknown value of the parameter.
Confidence interval estimates the value of the parameter not with just one number,
but with a whole interval.
So it's not a point estimate,
it's an interval estimate.
For the proportion, confidence interval is given by the formula P Hat plus minus Z,
times the standard deviation of the estimator.
Z actually depends on the confidence level Alpha,
and for standard Alpha equal to 0.05.
It is equal to approximately 1.96 or even more approximately two.
This formula is a more precise version of the two sigma rule.
You might have heard of it. For our sample of 100 taxi trips,
that 95% confidence interval for the proportion of tipper's could be
calculated with the function proportional confident from the module stats models.
It gives us the interval from 0.567 to 0.753,
and it's pretty wide.
It turns out that we are not so sure in our estimate of 66% of tipper's,
with 95% confidence, the percentage of tipper's might be as low as 57% or as high as 75%.
Can we get a more precise estimate?
Sure we can. We just need a bigger sample.
Just like point estimates get more precise with growing sample sizes,
confidence intervals get narrower,
but how big do we need N to be exactly?
There is a special function, sample size underscore,
confidence underscore proportion that's giving your guess of
the true proportion and the desired precision
which is a half width of the confidence interval,
returns to the required sample size.
If we want our confidence interval to be 2% wide, as you can see,
we might need a sample of at least 9,108 taxi trips.
Let's take a sample of 10,000.
In that sample we have 61% of tipper's with 95% confidence from 60.3 to 62.2.
Indeed, the width of this interval is about 2% just like we wanted.
From this video, you learned how to estimate a proportion of ones in the binary feature.
A point estimate is just a proportion of ones in a sample.
We have also talked about the concept of confidence intervals which
allow us to explicitly quantify the degree of your uncertainty in the estimate.
And learn to apply functions proportional confluent
which calculate the interval for the proportion and
sample size confluent proportion which helps you to choose
a sample size big enough to obtain a confidence interval as narrow as you like.
Explore nosso catálogo
Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.
O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.
© 2018 Coursera Inc. Todos os direitos reservados.
