"How do they know that?" Modern astronomy has made some astonishing discoveries - how stars burn and how black holes form; galaxies from the edge of the universe and killer rocks right next door; where the elements come from and how the expanding universe is accelerating. But how do we know all that? The truth is that astronomy would be impossible without technology, and every advance in astronomy is really an advance in technology. But the technology by itself is not enough. We have to apply it critically with a knowledge of physics to unlock the secrets of the Universe. Each week we will cover a different aspect of Astronomical technology, matching each piece of technology to a highlight science result. We will explain how the technology works, how it has allowed us to collect astronomical data, and, with some basic physics, how we interpret the data to make scientific discoveries. The class will consist of video lectures, weekly quizzes, and discussion forums. Each week there will be five videos, totalling approximately 40 minutes. They will be in a regular pattern - a short introduction, an example science story, an explanation of the key technology area, a look at how the technology is used in practice, and finally a look at what the future may hold.
W5 Part 3 - Computers, Databases, and the Internet
Loading...
Do curso por The University of Edinburgh
AstroTech: The Science and Technology behind Astronomical Discovery
215 ratings
The University of Edinburgh
215 ratings
Na lição
Computers and Killer Rocks
Why computers are crucial in astronomy, and how they are used to find rare objects.
Conheça os instrutores
Andy Lawrence
Physics and Astronomy
Catherine Heymans
Physics and Astronomy
Let's look at some of the technical challenges involved in
astronomical data and calculations and how we tackle those challenges.
We're going to want to look at how much data we want to store, how
long it takes to search through it, and how long it takes to do calculations.
And in each of those cases, we'll see
there's a brute force method, and a smart method.
And in fact we usually want to do both.
And then right at the end we'll wrap up with looking at how
we try to make data access over the internet as easy as possible.
[BLANK_AUDIO]
So let's talk about how much data we need.
First of all, let's think about a single CCD image.
So maybe one CCD image is about 4,000
pixels across, and each one of those pixels,
there's a number stored with 16 bits of
information which is then 2 bytes of information.
And if you do the sums that adds up to 32 megabytes.
So that's what a single CCD image might require for storage.
Not too much by modern standards.
However as you've seen we tend to use large mosaic
cameras with many CCD's put together in a big array.
Those gigapixel cameras could be much bigger.
You could be talking here about a a mosaic camera...
... a single image might be 1 to a few gigabytes.
So that's a big image. Now what about if we want to survey the whole sky?
Well you could ask how many pixels are there over the whole sky?
If we try to pave the sky with CCD images, what do we need?
Well it depends how fine the pixels are,
but let's suppose the pixels are about one third - 0.3 -
of an arc second, so that makes
the pixels small enough to get reasonably decent images.
That makes about 5 trillion pixels over the whole sky.
Then again, if you assume we have 16 bit numbers.
Oh, and by the way, that 16 bit number
is enough to store numbers up to about 65,000.
And, that's typically what CCDs do.
So covering the whole sky, 5 trillion pixels.
That makes about 10 terabytes.
Now actually of course we want to do the same thing at
several different wavelengths, different colors and also we want to repeat the sky.
So in practice a typical modern sky survey, the
whole sky will be something like petabyte scales.
Now a reminder about all of these petas and gigas and so on... so a factor of 10 to
the power of 3,000 in scientific notation, that's kilo, as in kilobyte, etcetera.
10 to the 6, that's big M for mega.
10 to the power 9, a billion, is a big G for giga.
10 to the power 12, big T for tera.
And then we'd get up to 10 to the power 15, a thousand trillion.
That's a big P for peta.
So, a petabyte is a thousand trillion bytes and
remember each byte is 8 bits in computer speak.
So that's how big a sky survey needs to be.
Now today if you've got a reasonably good laptop, you know, you could
get yourself a 1 terabyte disk, it's not that unusual to do so.
So maybe to store, for yourself, a sky survey, you need about 1000 laptops.
So it's doable, but really a bit daft.
If every astronomer has to have 1,000 laptops to store
their own copies of all their favourite databases, that's just silly.
So here's the smart solution. What it drives
us into is what's known as a service economy.
Around the world, there's a handful of big data centers, one of
which is here in Edinburgh, and there're a number of others,
where we store on big computers with lots of disks the major sky surveys.
Astronomers around the world through the internet go and get only the data
they want, or do calculations or searches on our servers here.
So, that's the smart way to do it.
[BLANK_AUDIO]
It's not just the pixels, the images, that
astronomers want, but it's catalogues of objects.
If you imagine here's an image of the sky,
and there are lots of stars and galaxies on it.
We have software that goes over here, spots each of these objects.
Each of those becomes a line in a table and that makes a catalogue of objects.
And for each one of these objects then is a row, and there are lots
of columns, and, each one of these columns is
a different piece of information about this object here.
So, that, then, is our catalogue.
So, how big are these catalogues?
Well, it's not nearly as big as the pixel data.
So for instance - it's a lot of objects,
out of a sky survey, we might have a billion objects, or a few billion -
and maybe there might be 50 of these columns.
And if each one of those is a couple of bytes, then
we end up with something like 100 gigabytes for a big sky survey.
So that's no problem to store, however searching through it is something else.
And that's what we'll look at next.
[BLANK_AUDIO]
So how do we search through a table of a billion objects and
find just the one we want, that redshift seven quasar or the killer rock or whatever?
So imagine here we've got lots of rows
in our table and this is sitting on our hard drive on
our computer, and then let's imagine over here is our
CPU in the computer, the bit that does the calculation, calculating.
Now in order to, do our search, essentially what
we have to do, is take one row of
data, bring it into the CPU, do a calculation
and decide whether we want that one or not.
And then we take the next row and do it again,
and the next row and do it again and so on.
Now, if imagine all this data streaming from the hard drive
to the CPU. A good PC will run at gigahertz rates,
So in principle, you can stream a billion rows
of data like this through in a split second.
It's not a problem.
However, it doesn't really work like that.
Any search process like this, any transfer from a hard drive to
the CPU - because you do it in lots of chunks, each one of those has
some kind of overhead, and that overhead may be only a few milliseconds, but
then when you multiply a billion times
a few milliseconds, you're into a much longer time.
It could take days to search through your big database.
Now modern solid state disks as opposed to
spinning hard drives have much smaller overheads - they're
faster, but still they are very expensive. Data centers,
at least scientific ones, are not using those because they're more expensive.
So, we need a smarter way to do the searching.
So the key point is that you don't
actually necessarily have to search through everything every time.
The first thing - and this is just the same
as say Google or Amazon do - you save the most
popular searches so they can be brought back quickly
the next time somebody asks pretty much the same thing.
The next thing is that the various columns here,
you figure out which of those are the most
common and put your database in the right
order so that you can search through them quickly.
And then you can pick examples of these columns to
make an index on and search through those particularly quickly.
[BLANK_AUDIO]
So let's talk about astronomical calculations, and how long they take.
And I'll take as an example, so
called N-body calculations that cosmological theorists use.
And the idea here is that we take lots of
fake matter particles and if you take any two of
those particles, we can calculate the gravitational force between them
and that tells us how they're going to move overtime.
But we need to take every possible pair
of particles and calculate the forces between all
those particles, to understand how the whole ensemble
of particles is going to evolve with time.
Now, a big calculation might have a million fake matter
particles and a really big one might have a 100
million fake matter particles. That step from a million to a
100 million makes a big difference as we'll see.
But first of all, let's make the basic point.
Let's imagine we've got one of
these big simulations with 100 million particles.
So, that's 10 to the power 8 particles.
Now, let's just assume, to begin with,
we're doing a single calculation on each particle.
[BLANK_AUDIO]
Now, on a fast computer, that's going to take less than a second.
It's not a problem.
Let's just say, that takes, 1 second.
However, as I just described, we need to do not one calculation per
particle, but one calculation for every pair of particles. So we need
to do 10 to the power 8 times 10 to the power 8 calculations.
Okay, every particle in that 10 to the 8 has to do all the others.
So then, that's an enormous amount of time.
That's going to take years.
And that essentially would make one frame, one time step
in that simulation movie that we saw earlier.
And you need lots of those to see how the universe evolves.
So this is just very difficult.
So the brute-force solution is to say okay, if it, this
is what one PC can do, we need a super computer.
So a supercomputer is really just like
thousands of computers chained together working in parallel.
So a big supercomputer might have several thousand nodes, and every
one of those nodes might have 10 or 20 cores in.
So there could many thousands effectively of CPUs working in parallel.
But even then there're two snags. It still
wouldn't be fast enough with this sort of calculation,
to get things done really quickly - and also the
other snag is that those machines cost millions of pounds.
We'd like to do something a bit smarter.
So the smart solution is to do with being approximate.
So with what I've described here, it's what you
have to do if you're going to do this calculation exactly.
Every particle and its effect on every other particle.
But, there's shortcuts you can do which have kind of to do with
the fact that particles further away, as individual pairs, are less important.
(They add up to a lot.)
Now, we haven't got time to explain exactly how this works.
For the mathematically minded, instead of a problem that is
n times n, we can get a speed change so that this is n times the logarithm of n.
Now this makes a very serious
difference to the speed of our calculation.
So for instance let's start... if we imagine we have
our 10 to the 6 particles and let's say...
that if we're doing N calculations ... let's say that that takes us 1 second.
Okay?
If we then do the the N times N,
On that it's going to take 12 days.
It's a million seconds.
If we do the N log N version, this is the approximate but
smart calculation, that takes 14 seconds.
It's a huge difference.
Then if we move up to 10 to the 8, right
then the the simple N calculations would then be 3 minutes.
If we do, the naive N times N calculation, that comes out to be about 317 years.
And that's obviously impossible, and infeasible even with a super computer.
If we do the n times the logarithm of
n method, that comes out to about 30 minutes.
Now notice that's still quite a long time - to calculate one timestep
in this simulation, but at least it's something plausible that that we can do.
So big calculations are very difficult.
[BLANK_AUDIO]
So we've been talking about technical challenges, how
much data there is to store, how hard that is, how long it takes to search, how long
it takes to do calculations, and all those problems are about machine time,
how long it takes a computer or a super computer to perform these tasks.
But the real world problem is not just
about machine time, it's about human time as well.
So for example what we don't want is that every time we go and get some data we have
to spend a whole afternoon working out how this particular website works, what
we have to do, and when we get the data we have to write a special
piece of software to deal with that data and plot it on top of something else.
All of that sort of then just uses up
astronomer time even if the machines are very fast.
Now, the modern internet that we're used to - when we're looking
for information, doing our shopping, etc, is very point and click.
It's very easy.
It took a lot of effort to get it that way, but it's automated.
And we want astronomy to work the same way, grabbing data, mixing and matching it.
That ideal is known as the virtual observatory.
And it's the secret - to either the virtual observatory, or
to the internet as a whole - is the same thing.
It's standardization.
What we need essentially is for everything to have the
same screw threads so that, so the bits fit together.
The different web pages, the data sets, and so on.
So forthe internet that's about things like TCP/IP, the
basic internet protocols, or HTTP the way, how you define how
you speak to a website or HTML, how you write the content of a website.
So all those things are standards which were agreed
internationally, and that's what makes it all magic and easy.
In the same way in astronomy we want to standardize
the format of data, the data access protocols and and so on.
And we're in the middle of that process as we speak.
Explore nosso catálogo
Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.
O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.
© 2018 Coursera Inc. Todos os direitos reservados.
