Capstone 1: Sonification of Movement

Loading...

Do curso por University of California San Diego

A internet das coisas: como chegamos aqui?

614 classificações

University of California San Diego

A internet das coisas: como chegamos aqui?

614 classificações

Curso 1 de 6 em Specialization Internet of Things

It is hard to imagine life without your Smartphone – you have come to rely on it so much – for your work; to stay in touch with family and friends; to capture and share those special moments; to find your way around in a new neighborhood. Did you ever wonder how and when all this happened? Or how and when GPS sensors came to be in your cell phone? In this course, we will explore the convergence of multiple disciplines leading to todays’ Smartphones. You will learn about the birth and evolution of Telephony Networks, Broadcast Networks (TV and Radio) and Consumer Electronics. We will discuss the impact of Internet, (multimedia) content, smartphones and apps on everyday lives. We will then look at how this emerging platform called the Internet of Things – wherein billions and trillions of devices communicating with each other and “the cloud” – could enable unprecedented, innovative products and services. Take this course if you want to understand what great new advances in mobile-enabled products will be coming our way! Learning Goals: This course provides a core grounding in how science and technology have developed to enable the Internet of Things – in a way appropriate for any learner. For those interested in developing further hands-on expertise in designing and developing for the Internet of Things, this course will provide a context to the discoveries and converging technologies that will springboard the next round of innovations. After completing this course, you will be able to: 1. Compare how the telephone system works (that is, peer-to-peer networks) with how media delivery works (that is, broadcast/multicast networks). 2. Explain the tradeoffs between circuit switched networks (that is, dedicated resources) and packet switched networks (that is, shared resources). 3. Tell interesting stories about key innovations that transformed the communications, entertainment and consumer electronics industries. 4. Explain how email, YouTube, SMS, etc. work. 5. Find resources for those wishing to do more of a “deep-dive” into the above topics.

Na lição

Welcome

Welcome to the first course in the Internet of Things specialization! We hope you’re just as excited as we are to begin this fascinating journey! Before getting started with the material, we would like to introduce you to the team that made this all possible, as well as give you a preview of what to expect throughout the entire specialization.

Welcome to the Internet of Things Specialization!11:46

Capstone 1: Sonification of Movement9:44

Capstone 2: Healthcare Remote Monitoring9:36

Qualcomm talks about the course3:04

Conheça os instrutores

Harinath Garudadri
Associate Research Scientist
Qualcomm Institute of Calit2, UC, San Diego
Ganz Chockalingam
Principal Engineer
Qualcomm Institute of Calit2, UC, San Diego

So, this is our video for Capstone 1,

which is centered around the sonification of movement,

which is taking a persons movement in front of the camera and

using this to change some audio representation.

And let us see how this works in more detail in the future slides.

So, the first thing we want to go over is just what you would need for this project.

So first, you need the DragonBoard to process everything,

you will need the power supply to power that DragonBoard,

and you will need a keyboard, mouse,

and monitor to interact with the DragonBoard.

There is an important note for the monitor.

It should either have its own speakers or you should have audio out on

the monitor and have speakers to run the audio portion and actually hear audio back,

since this is an audio changing project.

And then, you also need a USB webcam to take in video and that will be

the video that we would be changing the audio against or comparing it against to.

Since we are taking video and changing audio depending on that.

A video is just a bunch of frames of pictures and in that picture is a bunch of pixels.

So, each sections of the picture control certain aspects of the audio.

So, motion on the left side of the picture highlight in green controls

the tempo and then motion on the right side of

the video highlighted in blue controls the melody.

Now, the little red box at the bottom,

any motion detected there will control the beat and

motion in the overall picture itself will control the volume.

So, I will go over the video portions of this project which is covered in more detail

in course five and it uses a little bit of stuff from the previous courses as well.

So, the first thing it does is it takes RGB values from

a frame and it converts every frame into grayscale values,

which is basically converting everything to some form of gray colors instead of just RGB.

You can think of this kind of like brightness, but it is not.

There is a little nuance to it as well,

but you guys get the point.

So, the next part is that it takes

the previous frame and the current frame and it subtracts them.

So, if there was a pixel that did not have movement the subtraction would be zero,

which means that if you look at the difference,

the grayscale value would be zero and it would be completely black.

But, if there was some kind of difference between them then you would see gray and

the frame that is the difference between the previous and current frame.

And this is how we detect whether there was motion

and to what extent the person moved as well.

So, the next part is MIDI,

which will be covered in more detail in course four,

which is going over some kind of communication aspects of the DragonBoard.

MIDI is one way we can communicate audio.

It is packaged into a binary message and it is labeled by dot mid.

And you put this into a synthesizer and the synthesizer knows this binary message,

it represents audio, but I can edit some parts of it as well.

So, I can turn off the piano parts of

the music or I can adjust the tempo or just the volume.

And you do not have this much freedom with file formats such as MP3 because

MP3 is just one music file and that is all it is.

But, this like binary message.

So you can just change individual parts of the message itself.

And then the next part is how we took the video parts and the audio parts

and put them together to complete this program.

The first thing we did was we made it so that

the difference between the two frames so as how to grayscale

value up to a certain threshold because if you barely

moved into frame or like some dust moved into frame as well,

this should not cost the music to change because the amount of movement was insufficient.

So, we had the person reach a certain threshold of movement before it affected anything.

Afterwards, we saw how much changed in the overall frame and how many pixels had reached

that much threshold and if there were a lot of pixels that changed from

the previous image then we would make it so that it would change

volume but if there is like a normal amount of movement,

we just said this value is considered

normal movement then we would just maintain volume at 50 percent.

Afterwards, we adjusted tempo discretely.

So, in the picture where we showed the diagram by a grid,

you saw that the tempo was shown in like a shade of green.

So, each shade of green means that is how much the tempo was at that point.

So, if you moved into

upper left hand corner

of your frame then that means that the tempo would increase to the maximum level.

If you just placed your hand in the bottom left hand corner then it would slow down.

We will demonstrate this later.

We just made it so that each tempo box had

a discrete amount of tempo difference instead of changing continuously.

Afterwards, we made it so that the movement in the right portion and

the bottom portion just turned on or off the melody and beat respectively.

So, there was not like a degree to which they showed in the audio,

its is just it is on or it is off.

And afterwards, we will show you a quick demo of how this works in person.

Simon will be showing us how it is done.

So, let us get on to our demo of Capstone

1 and we are going to have Simon demonstrate

how it works and I will just narrate over it.

So let us start off the program.

It will take a moment to load.

So as you start off,

the sound starts playing but you cannot hear the melody anymore.

Just ignore the blue circle in the middle,

that was specific for me,

just to see where I should be positioned but if Simon starts moving his right hand,

you will see that the melody starts playing again.

Let us have Simon stop moving his right hand and the music stops playing.

So, let us have Simon put on his left hand at the very top left hand corner.

Let us have him stop doing that and then let us

have him start moving his right hand again.

You can see that the music is significantly faster now.

And let us have Simon move his hand to the bottom left hand corner

now and then start moving his right hand.

You can see that the music is significantly slower now.

Let us have Simon start moving his legs.

You can see that the beats start playing now.

Let us have Simon just move a lot on the camera so that the volume increases.

You can see that the volume gradually increases but if Simon moves a lot more,

we can see that the volume will increase significantly.

So, Simon has basically reached the maximum volume levels.

But, if Simon stops moving like all of a sudden,

you will see that the volume would decrease significantly as well.

You can see that volume starts to decrease.

So, this is the basic idea of Capstone 1.

So, in conclusion, what this

demonstrates is that there is a potential to help out people,

who are having trouble with movement.

So, it is just a fun way to have them practice

some muscle memory and to practice how they can psychologically move again.

So, this is just an entertaining way for them to help themselves

and it also an entertaining way to just play around with music.

We are very excited to what you guys can create using this as

a platform and just show us cool stuff that you can do.

We will see you guys next time.

Explore nosso catálogo

Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.

O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.

© 2018 Coursera Inc. Todos os direitos reservados.

Baixar na App Store

Baixar no Google Play