In this course you will learn how to evaluate recommender systems. You will gain familiarity with several families of metrics, including ones to measure prediction accuracy, rank accuracy, decision-support, and other factors such as diversity, product coverage, and serendipity. You will learn how different metrics relate to different user goals and business goals. You will also learn how to rigorously conduct offline evaluations (i.e., how to prepare and sample data, and how to aggregate results). And you will learn about online (experimental) evaluation. At the completion of this course you will have the tools you need to compare different recommender system alternatives for a wide variety of uses.
Usage Logs and Analysis
Loading...
Do curso por University of Minnesota
Recommender Systems: Evaluation and Metrics
109 classificações
Na lição
Online Evaluation
Conheça os instrutores
Michael D. EkstrandAssistant Professor
Dept. of Computer Science, Boise State University
Joseph A KonstanDistinguished McKnight Professor and Distinguished University Teaching Professor
Computer Science and Engineering
Welcome back.
In this video, we're going to talk through some examples of log analysis
as a form of evaluation and exploration in recommender systems.
I'm going to use the example of MovieLens.
Now, you're obviously already familiar with MovieLens,
if you've taken this whole specialization.
But here's a quick screenshot that can show you what MovieLens looks like today.
It's changed over the years.
What I want to add to that is the fact that we gather all sorts of data.
This happens to have been a report that I asked for that looked by user
ID at the number of log ins, tags, and ratings over the past month.
I'll come back to that in spreadsheet form, but actually,
I just got it graphed to show, look.
Most of our users who logged in, in the past month, logged in once or twice.
In fact, if I look here I can see that the median was two.
But the maximum was 182 times, that's a lot, 6 times a day.
I can look at other data this way too.
Tags 90% of users don't apply tags.
Our top users, apply a lot of tags.
Ratings are a little more spread out, but it's still true that a quarter of our
users apply zero or one rating in the past month.
Half of them, 6 or fewer,
three quarters, 25 or fewer, that's not a huge number.
That's a lot of movies to rate, if you're watching movies as you rate them,
for somebody whose knew it's not a huge number.
At the high end, 2,700, and
we can see this in the spreadsheet and do analysis base on them.
And we can actually look at the correlations between ratings and
other things.
Not going to do that because that's not actually the biggest question that I have.
We logged every thing, we can log about this site.
When I logged in and got this set of displays,
it said what all of these movies were that were shown to me.
If I say see more, in addition to generating a set of recommendations.
It's logging away that here is a set of 24 movies that I've just displayed to you.
With this, there's a bunch of types of interesting analysis that we can do
entirely retrospectively.
And I want to take you through just a couple examples of that.
One of the things that we can look at is the relationship
between what people rate, and whether and
when those things were recommended to them in the past.
So one of our former students, now a graduated PhD did
a study looking specifically at the question of whether
recommender systems narrow people's behavior.
Whether there's a filter bubble, whereas people use the recommender
system they consume narrower and narrower choices.
Or whether maybe recommender systems broaden people's behavior.
Well to do this, one of the things he could do is say,
let's look at the diversity of ratings people have made over time.
We have all of people's ratings, and
it turns out that if you look at users of movielens over time.
And you measure diversity by things like genre and keyword.
And in this case, they used a metric that is based
on tags that can give you a distance between movies.
People's tastes narrow over time, but he went a step further,
because we had good logs.
He went back and said, well let's see what the effect is of
people who see the movie as we give them recommendations for,
versus people who choose movies on their own.
How do you do this?
Well he said, we're going to assume for
the moment that if we displayed a recommendation for a movie,
like this one here, I should see the Shawshank Redemption.
I probably should, I haven't.
But that if this is displayed to me and sometime in the next three hours,
after three hours, up to I think he used three months from now.
I come back and rate it.
We're going to assume I was at least influenced by the display of it.
Now if I rate it right now, I didn't see it because of this.
I was reminded that I'd seen it before.
But if I go off, finish taping this video, go watch the movie,
come back and rate it or if I watch it this weekend or the end of the month.
We're going to count that as a recommendation that was made,
taken and used.
By contrast if I come back, and
I rate a movie that it was never recommended to me during that interval.
We're going to say that wasn't a recommended movie.
What this student Tian Wynn did is he created this measure and looked at
people and discovered that we have a lot of people who don't take recommendations.
They use MovieLens, they put stuff in.
But they rarely have ever see any movie we display to them.
What that means to a large extent is they're not looking at these lists.
Once in a while they might stumble upon something that's on the home screen.
But they're rushing in to search for things and rate them.
They're using the system in a different way.
We also have a bunch of users who take recommendations often,
often defined as a fraction of how many movies they come back later to rate.
And he found something interesting.
Yes, MovieLens, movie watching tastes narrow as you use MovieLens.
But they narrow more if you never take recommendations.
In other words, people's tastes narrow naturally.
But the recommender might help diversify those tastes.
Let me give you an example of a very different kind of study,
we were able to do using log data.
We wanted to understand the effect of where somebody saw movies in this list.
On the probability that they would rate it later.
We just had the data, we went and we looked at it.
That led us to do experiments that followed up.
We've done things like varying this list and
then tracking what happened, but we started with log data.
We actually built models about how people consume lists of
recommendations by log data and a little bit of lab experimentation.
A third thing we've been doing for a long time is trying to track how well
the actual performance of the system, prediction to eventual rating is
compared against the offline performance of the system.
So the offline performance says, let's calculate that prediction with all
the data that we have at the time we're running our study or running our metric.
The online version says, well what did we really display?
Did we really display something that related properly to what the user
came up with?
All of these things are possible with logged data.
Just three quick observations about this, then we're going to draw it to a close.
One is, these days it's a pretty good idea to log just about everything.
The data is awfully useful.
Two, when you do that if you're not careful,
you log everything in a form where nobody can figure out what to do with it.
So it's not enough to log things.
You've got to develop a structured logging form where you have
information on user, session, time stamp,
anything it is that you might need to reconstruct what happened.
And three, you've got to go out and actually use that data.
And thinking about how you can use your log data on a regular basis to deliver
reports about what's going on will help you get the most out of that data.
We do some very simple reporting where every week I get an email
with some statistics about what's going on inside movie lens.
Certainly, they could be more sophisticated.
But they at least give us a picture of is everything going as we expected.
If I were redesigning that today, I might come back and
say, gee, what is the error distribution of ratings
against the last prediction of those movies?
That might be interesting to know.
I might have some other things I would want to look at related.
I'd get some statistics on user retention for instance, which are really useful.
But with good log data behind the scenes,
you can do an amazing variety of interesting analysis.
So this was our example of log data analysis using MovieLens.
Look forward to seeing you next time.
Explore nosso catálogo
Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.
O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.
© 2018 Coursera Inc. Todos os direitos reservados.
