Aggregation Operations in Big Data Pipelines

Loading...

Do curso por University of California San Diego

Big Data Integration and Processing

1082 classificações

University of California San Diego

Big Data Integration and Processing

1082 classificações

Curso 3 de 6 em Specialization Big Data

At the end of the course, you will be able to: *Retrieve data from example database and big data management systems *Describe the connections between data management operations and the big data processing patterns needed to utilize them in large-scale analytical applications *Identify when a big data problem needs data integration *Execute simple big data integration and processing on Hadoop and Spark platforms This course is for those new to data science. Completion of Intro to Big Data is recommended. No prior programming experience is needed, although the ability to install applications and utilize a virtual machine is necessary to complete the hands-on assignments. Refer to the specialization technical requirements for complete hardware and software specifications. Hardware Requirements: (A) Quad Core Processor (VT-x or AMD-V support recommended), 64-bit; (B) 8 GB RAM; (C) 20 GB disk free. How to find your hardware information: (Windows): Open System by clicking the Start button, right-clicking Computer, and then clicking Properties; (Mac): Open Overview by clicking on the Apple menu and clicking “About This Mac.” Most computers with 8 GB RAM purchased in the last 3 years will meet the minimum requirements.You will need a high speed internet connection because you will be downloading files up to 4 Gb in size. Software Requirements: This course relies on several open-source software tools, including Apache Hadoop. All required software can be downloaded and installed free of charge (except for data charges from your internet provider). Software requirements include: Windows 7+, Mac OS X 10.10+, Ubuntu 14.04+ or CentOS 6+ VirtualBox 5+.

Na lição

Processing Big Data

This module introduces Learners to big data pipelines and workflows as well as processing and analysis of big data using Apache Spark.

Big Data Processing Pipelines7:06

Some High-Level Processing Operations in Big Data Pipelines8:24

Aggregation Operations in Big Data Pipelines5:15

Typical Analytical Operations in Big Data Pipelines10:43

Conheça os instrutores

Ilkay Altintas
Chief Data Science Officer
San Diego Supercomputer Center
Amarnath Gupta
Director, Advanced Query Processing Lab
San Diego Supercomputer Center (SDSC)

Aggregations in Big Data Pipelines.

After this video you will be able to compare and

select the aggregation operation that you require to solve your problem.

Explain how you can use aggregations to compact your dataset and reduce volume.

That is in many cases.

And design complex operations in your pipeline, using a series of aggregations.

Aggregation is any operation on a data set that performs a specific transformation,

taking all the related data elements into consideration.

Let's say we have a bunch of stars which are of different colors.

Different colors denote diversity or variety in the data.

To keep things simple, we will use letter 'f' to denote a transformation.

In the following slides,

we will see examples of how 'f' can take the shape of different transformations.

If we apply a transformation that does something using the information of

all the stars here, we are performing an aggregation.

Loosely speaking, we can say that applying a transformation 'f'

that takes all the elements of data as input is called 'aggregation'.

One of the simplest aggregations is summation over all the data elements.

In this case, let's say every star counted as 1.

Summing over all the stars gives 14,

which is the summation of 3 stars for yellow,

5 stars for green, and 6 stars for color pink.

Another aggregation that you could perform is summation of individual star colors,

that is, grouping the sums by color.

So, If each star is a 1, adding each group will result in 3 for

yellow stars, 5 for green stars, and 6 for pink stars.

In this case,

the aggregation function 'f' will output 3 tuples of star colors and counts.

In a sales scenario, each color could denote a different product type.

And the number 1 could be replaced by revenue generated by a product

in each city the product is sold.

In fact, we will keep coming back to this analogy.

You can also perform average over items of similar kind,

such as sums grouped by color.

Continuing the example earlier,

you can calculate average revenue per product type using this aggregation.

Other simple yet useful aggregational operations to help you extract meaning

from large data sets are maximum, minimum, and standard deviation.

Remember, you can always perform aggregation as a series of operations,

such as maximum of the sums per product.

That is, summation followed by maximum.

If you first sum sales for each city, that is for

each product, then you can take the maximum of it

by applying maximum function to the result of the summation function.

In this case, you get the product, which has maximum sales in the country.

Aggregation over Boolean data sets that can have true-false or

one-zero values could be a complex mixture of AND, OR, and NOT logical operations.

A lot of problems become easy to manipulate using sets.

Because sets don't allow duplicate values.

Depending on your application, this could be very useful.

For example, to count the number of products from a sales table,

you can simply take all the sales tables and create sets

of these products in those tables, and take a union of these sets.

To summarize, by choosing the right aggregation, you can generate compact and

meaningful insights that enable faster and effective decision making in business.

You will find that in most cases,

aggregation results in smaller output data sets.

Hence, aggregation is an important tool set to keep in pocket when dealing with

large data sets, and big data pipelines.

Explore nosso catálogo

Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.

O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.

© 2018 Coursera Inc. Todos os direitos reservados.

Baixar na App Store

Baixar no Google Play