Reading Large Tables

Loading...

Do curso por Johns Hopkins University

R Programming

13315 classificações

Explore Related Videos

At Coursera, you will find the best lectures in the world. Here are some of our personalized recommendations for you

Johns Hopkins University

R Programming

13315 classificações

Curso 2 de 10 em Specialization Data Science

In this course you will learn how to program in R and how to use R for effective data analysis. You will learn how to install and configure software necessary for a statistical programming environment and describe generic programming language concepts as they are implemented in a high-level statistical language. The course covers practical issues in statistical computing which includes programming in R, reading data into R, accessing R packages, writing R functions, debugging, profiling R code, and organizing and commenting R code. Topics in statistical data analysis will provide working examples.

Na lição

Week 1: Background, Getting Started, and Nuts & Bolts

This week covers the basics to get you started up with R. The Background Materials lesson contains information about course mechanics and some videos on installing R. The Week 1 videos cover the history of R and S, go over the basic data types in R, and describe the functions for reading and writing data. I recommend that you watch the videos in the listed order, but watching the videos out of order isn't going to ruin the story.

Conheça os instrutores

Roger D. Peng, PhD
Associate Professor, Biostatistics
Bloomberg School of Public Health
Jeff Leek, PhD
Associate Professor, Biostatistics
Bloomberg School of Public Health
Brian Caffo, PhD
Professor, Biostatistics
Bloomberg School of Public Health

So with larger data sets of beyond the small to moderate,

then there are a couple of things you can do when reading in tabular data.

That will make your life a lot easier, and

more importantly it will prevent R from totally choking.

So first you should read the help page for read.table.

In fact, you should probably have it memorized.

There is a lot of key hints in that help page.

Lot of useful information.

And in my opinion not enough people read this help page carefully enough.

So that they can kind of recite in their sleep.

And if I, so there's a lot of so once you've read that you'll see

there's a lot of important information for kind of how to optimize read.table.

In particular for large data sets.

And so one of the things you're going to want to do

is to make a very rough calculation.

Of how much memory you need to store the data set you're about to read.

And so that way you can get a sense of well,

is there enough memory on my computer to store this data set?

Because if you recall correctly,.

R will have to,

R is going to store your entire dataset in memory unless you do otherwise.

So when you call read.table or

read.csv, it's reading your entire dataset into the RAM of the computer.

And so you need to know,

roughly speaking, how much RAM this datasets going to require.

And we'll talk about how to calculate that in a second.

So another opti,

easy optimization you can say is if there's no comment lines in your file.

Then just set the comment char to be the comment.char is meant to be blank.

So just an empty quote there.

The call classes argument is actually very important.

Because it whe, if you don't specify it then what R does by default is it

goes through every column of your dataset.

And tries to figure out what type of data it is.

Now, that's all fine, well I'm fine when the dataset is small to moderate.

But reading each of these columns and trying to figure out what type of data it

is takes time, it takes memory, and it can generally slow things down.

If you can tell R, what type of data, is in each column,

then R doesn't have to spend the time to figure it out on its own.

And so, it'll, it'll generally make read.table run a lot faster.

So you can save yourself a lot of time.

So if you, if, if you have a few columns in your dataset,

then then you can usually just say what, what the classes are.

But if you have, or if they are all the same, so for

example if all the columns are numeric.

You can just say, you can just set call classes equal to numeric.

And if you only sent, you give it a single value,

it will just assume that every column has that same value.

So if you just say numeric it will assume that every column is numeric.

Otherwise what you can do if you have a huge data set,

you can read in maybe the first 100 or first 1,000 rows.

By specifying the nrows argument.

And then going through each of the looping over each of the columns using sapply and

calling the class function.

So the class function will give you,

will tell you what class of data is in each column.

And then you can use this, and then you can save, store this information.

And then read the entire data set after by specifying the call classes argument.

So the n rows argument is actually very useful too.

It doesn't necessarily make R run any faster, but

it does help with memory usage.

And so, if you can tell R how many rows are going to be read in to,

to the, in to, in to R.

Then it can calculate the memory that's going to be required.

And not have to kind of figure it out on the go.

So even if you mildly overestimate how many rows there are in the data set,

that's okay.

Because it won't make a difference, it'll still read the correct number of rows.

So in general, when you're using R with large data sets, and

there's lots of large data sets out there nowadays.

It's useful to have a few things, a few bits of information on hand.

So, for example, how much memory does your computer have?

How much physical RAM is there?

These days in most computers will have on the order of a few gigabytes up

to many gigabytes of physical RAM.

What other applications are in use?

So are there other applications that are running on your computer that are eating

up some processor time or memory?

If you're on a multi-use system, are there other users logged into the system.

Are they using up some of the resources on the computer?

What is the operating system for your computer?

So, is it a Mac?

Is it Windows?

Is it Unix?

Is it Linux?

Is it something like that?

And then, also it's useful to know whether the O,

the operating system that you're running is 32-bit or 64-bit.

On a 64-bit system there, there, you'll generally be

able to access more memory if the computer has a lot more memory.

So if you want to do a rough calculation before you read in a table into R,

using the read.table or the read.csv function.

You can just do a very quick calculation.

So here is,

suppose I have a data frame here, with 1.5 million rows and 120 columns.

So this is not a particularly big data set but it's reasonable.

so, suppose that all of the nu, all the columns are numeric.

So, I don't have to worry about different types of data.

They're all, all the columns are numeric.

The question is how much memory is required to store this data frame in

memory, okay?

So, I can do a simple calculation.

So, the num, the number of elements in this data, in this data

frame is going to be 1.5 million times 120, right, because it's a square object.

And so that's, so that's the number of elements in the data frame.

Now, if it's a numeric all the data are numeric then each

number requires eight bytes of memory to store.

Because the, because the numbers are stored using 64-bit numbers and

there's eight bits per byte.

So that's eight bytes of memory per numeric object.

So that's going to,

so here's the number of bytes, now there's two to the 20 bytes per megabyte.

So I can divide that, the number of bytes by 2 to the 20, and

that's how many megabytes I got.

So it's got, I've got 1,373.29 megabytes.

And I can divide that again by 2 to the 10 to get the number of gigabytes,

that's going to be roughly 1.34 gigabytes.

So the, the raw storage for this data frame, is roughly 1.34 gigabytes.

now, you're actually going to need a little bit more memory than that to

read the data in.

Because there's a little bit of, overhead required for reading the data in.

And so, and so the rule of thumb, is to, is that you're going to need

almost twice as much memory to read this dataset into R using read.table.

Then the, then the object itself requires.

So if your computer only has, let's say two gigabytes of RAM eh, and

you're trying to read in this 1.34 gigabyte table.

You might want to think twice about trying to do it.

Because it, you're going to be pushing the boundaries of of memory that,

that is required to read this dataset n.

Of course, if your computer has like four or eight or 16 gigabytes of RAM,

then you should have no problem in terms of the memory requirements.

It will still take some time just to read it in just because it takes time to

read in all the data, but you won't be running out of memory.

So doing this kind of calculation is enormously useful when you're reading in

large data sets.

Because it can give you a sense of you know do I have enough memory.

Is the reason, if you grunt any errors,

you'll know whether the error is because of memory, running out of memory or not.

So I encourage you to do

this kind of calculation when you're going to be reading in large data sets.

And you, and you, and you know in advance kind of how big it's going to be

Explore nosso catálogo

Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.

O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.

© 2019 Coursera Inc. Todos os direitos reservados.

Baixar na App Store

Baixar no Google Play