Dynamic Allocation

Loading...

Do curso por Yandex

Big Data Analysis: Hive, Spark SQL, DataFrames and GraphFrames

60 classificações

Yandex

Big Data Analysis: Hive, Spark SQL, DataFrames and GraphFrames

60 classificações

Curso 2 de 5 em Specialization Big Data for Data Engineers

No doubt working with huge data volumes is hard, but to move a mountain, you have to deal with a lot of small stones. But why strain yourself? Using Mapreduce and Spark you tackle the issue partially, thus leaving some space for high-level tools. Stop struggling to make your big data workflow productive and efficient, make use of the tools we are offering you. This course will teach you how to: - Warehouse your data efficiently using Hive, Spark SQL and Spark DataFframes. - Work with large graphs, such as social graphs or networks. - Optimize your Spark applications for maximum performance. Precisely, you will master your knowledge in: - Writing and executing Hive & Spark SQL queries; - Reasoning how the queries are translated into actual execution primitives (be it MapReduce jobs or Spark transformations); - Organizing your data in Hive to optimize disk space usage and execution times; - Constructing Spark DataFrames and using them to write ad-hoc analytical jobs easily; - Processing large graphs with Spark GraphFrames; - Debugging, profiling and optimizing Spark application performance. Still in doubt? Check this out. Become a data ninja by taking this course! Special thanks to: - Prof. Mikhail Roytberg, APT dept., MIPT, who was the initial reviewer of the project, the supervisor and mentor of half of the BigData team. He was the one, who helped to get this show on the road. - Oleg Sukhoroslov (PhD, Senior Researcher at IITP RAS), who has been teaching MapReduce, Hadoop and friends since 2008. Now he is leading the infrastructure team. - Oleg Ivchenko (PhD student APT dept., MIPT), Pavel Akhtyamov (MSc. student at APT dept., MIPT) and Vladimir Kuznetsov (Assistant at P.G. Demidov Yaroslavl State University), superbrains who have developed and now maintain the infrastructure used for practical assignments in this course. - Asya Roitberg, Eugene Baulin, Marina Sudarikova. These people never sleep to babysit this course day and night, to make your learning experience productive, smooth and exciting.

Na lição

Spark Internals and Optimization

Persistance and Checkpointing7:00

Memory Management3:35

Resource Allocation6:24

Dynamic Allocation5:16

Speculative Execution4:39

Conheça os instrutores

Natalia Pritykovskaya
Pavel Klemenkov
Chief Data Scientist
NVIDIA
Pavel Mezentsev
Senior Data Scientist
PulsePoint inc
Alexey A. Dral
Founder and Chief Executive Officer
BigData Team

In this video, we will talk about dynamic allocation.

In the previous video,

you're trying to answer the following question.

How much resources should you allocate for your app?

And it turned out that you actually had to answer three questions.

How many executors should you start?

How many cores and memory should each executor have.

That's quite a few decisions to make to start a job.

It happens that Spark allows you to skip one of

these decisions with the mechanism called, dynamic allocation.

So how does this work?

For simplicity, suppose you have a cluster which can run 12 executors of a fixed sized.

At the beginning of the day,

your colleague starts a Jupyter Notebook which

allocates nine executors to compute an expansive analytical query.

But then, he remembers that he has forgotten to grab

his morning coffee and goes to the kitchen to pour some.

His Jupyter session is still there on a cluster holding nine executors which do nothing.

Then you come to the workplace and want to start a simple job with just four executors,

but you fail because only three executors are available,

although, the cluster does nothing.

Dynamic allocation handles this situation by deallocating

idle executors and starting new executors if your job needs them.

So, if dynamic allocation is turned on,

Spark will kill several idle containers of

your colleague and your job will successfully start.

So, why the Spark needs dynamic allocation of resources?

Classical MapReduce frameworks use short-lived containers for their tasks,

but Spark architecture has used long running containers to speed up computations.

And this may not be the optimal approach in shared and noisy clusters.

So, how do you configure a dynamic allocation in Spark?

The main option is Spark.dynamicAllocation.enabled,

which is false by default.

Then you should also specify,

what does the idle time out of your executor?

Meaning, how much time should pass for executor being idle before Spark removes it?

It is 60 seconds by default.

There is also another important option which is forgotten all the time.

You now know that you can speed up your job by caching data.

And if this executor has cached data,

Spark can't remove it,

because cached executor idle timeout is set to infinity, by default.

So you really have to tune this option for dynamic allocation to work it out.

Two other options are, minExecutors and maxExecutors.

Minimum number of executors which are allocated to your job are zero by default.

This means, that you may come up with a situation when your app

is running only the driver and no executors at all.

This behavior may not be desirable.

Maximum number of executors is not bounded.

So you may or may not want to tune these option.

Another important thing to keep in mind with the dynamic allocation,

is that executors have a piece of memory for execution.

We talk about this in one of the previous videos.

This means, that the results of the shuffles,

aggregations, and joints are stored in this memory.

And if Spark removes an executor,

your job will definitely lose this data.

To prevent this, when your cluster runs on yarn,

you have to enable an external shuffles service in Spark,

and also, point yarn to this external shuffle service.

So, what are the use cases for dynamic allocation?

The first one, are long-running ETL jobs.

These jobs start on a regular basis and process batches of new raw data.

As soon as data generation is some probabilistic process,

each start of the job will deal with different volumes.

The second one, and the most useful for our course,

are interactive jobs like Jupyter Notebooks.

Due to the nature of analytical work with data,

it is best to use dynamic allocation here.

And the last one are jobs with large shuffles.

So let's sum up.

Spark Architecture assumes using long-running containers for speed.

These may costs under utilization of the cluster resources.

And that's why dynamic allocation can help.

Dynamic allocation starts new executors when they are

needed and remove executors when are idle.

But dynamic allocation requires fine tuning of your job,

and you need to pay attention for different configuration options.

Finally, due to the dynamic nature of this strategy of resource allocation,

temporary shuffle results should be stored in an external service.

More information on dynamic allocation is available in these talks.

Explore nosso catálogo

Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.

O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.

© 2018 Coursera Inc. Todos os direitos reservados.

Baixar na App Store

Baixar no Google Play