Hey folks, welcome back. In this video we're going to talk about heuristic evaluation, which is going to be the focus of a series of videos. And it's one of my favorite topics in user interface design, so I hope you enjoy this. All right, the first thing we're going to do in this video is to just go over a basic definition of heuristic evaluation. Next, we'll talk about the context in which heuristic evaluation is typically applied. Third, we'll give you a sense of how heuristic evaluation was developed. I'll just do this very briefly. And then finally, in the next sequence of videos, we'll talk about each individual heuristic, you'll understand that more in a second, in a great deal of detail. So what is this thing called Heuristic Evaluation? Well the basic idea here is that a small group of people, trained in heuristic evaluation, so that will soon include you. They examine a existing user interface. Each evaluator lists the problems of the interface with respect to established usability principles, otherwise known as heuristics. And then each evaluator generally ranks each problem by severity. So, for example, here, this is a fun example from a recent research project. I have entered in a route to Google maps you can see, from Amsterdam to Amman, Jordan, two generally very safe places. But this route takes us through Western Syria, which as you might have heard, at least currently, is in a tremendous state of turmoil associated with Isis and their civil war there. But Google would never let you know this, right? Or Google just doesn't let you know this, it just says, you happen to go through some tolls and cross country borders. So, if I were a heuristic evaluator, I would list this with a severity of three. We'll talk about that more in a second. Categorize this in the error prevention heuristic, that'll be covered in the next couple videos. And I'll write a description while it directs users through very dangerous areas. This is the basic idea, right? We're just going to list problems with interface, give them the severity score, associate each problem with a heuristic, and then write a description of that problem, pretty basic idea. Typically you do this in teams, so you'll have around, if folks are following best practices, three to five people. You can this on this graph here we have a number evaluators on the x axis, and the proportion of usability problems found. This is from a study that was done back in the 90s, and you can see there are diminishing returns as you might expect. But when you're around three to five people, each person is contributing new problems that are going to be important to find. One thing that is critical to point out is each evaluator should work independently, right? You shouldn't be teaming out working on this together, that ruins your independent judgement. And secondly, this generally happens in two phases. So you'll have a problem identification phase and you'll have a severity assessment phase. And we'll talk about that in a second. Okay, let's go through a few additional specifics here. When you're doing heuristic evaluation, you generally want to go through or explore each interface at least twice. So in the first pass, you want to explore the interface. Just generally, just see what's there, get an idea of the scope of the system. And then, in your second pass, you're grinding through the details, you're listing out those problems, in each part of the interface. So in the first pass, you're typically kind of mapping where things are, in the second pass you're doing more or less the real work. When you're doing a heuristic evaluation in general, you're not performing a specific task, you're exploring. But there are some exceptions to that rule. Many times user interface design experts are asked to evaluate a system in a domain they don't understand. Say for instance if I were asked to evaluate a system Professor Yuroshi is here watching me record this today, a system on some of her research topics like a system for children, or a system that maybe is for caretakers of the elderly. Well I don't know too much about that, I'm not a domain expert. So what Professor Yuroshi might do is say hey, here's a scenario. Can you do a heuristic evaluation according to this scenario. Okay, let's talk some specifics in terms of output, because I think this can be very helpful, just in terms of having taught this material in the past, helping folks really understand what's going on. It's not all that complicated, basically your output is just a list of severity ratings, heuristics, and details of the problem. The sets of these three items, just a huge list of them right? And it's very critical that you don't skip the second step, right? That you have to site which usability principle is violated, which heuristic is violated, for each problem. Otherwise this exercise devolves into a different, type of experience, maybe it can be more critique associated. And if you're trained to do critiques, well that's useful in a different context. But if you're not trained to do that, it can really devolve into basically how two people talk about how they didn't like a movie ,or a song, or these types of things. In the United States, we have this expression armchair quarterbacking. [LAUGH] It could sort of devolve into that, and this is a typical problem. I see students, particularly students who don't study enough, are they fall into. So they're not as familiar with the heuristics, so they just find problems, they don't say why those problems are problems, and the heuristic evaluation is less valuable as a result. So this is why I made this cute little icon here for the intro slide. You've got your severity ranking. You've got that first black box which is the heuristic you're citing, and the second black box which is the more detailed description of the problem. Okay, there is one more thing I want to point out about the output, this is a point of confusion for a lot of folks. If the same interface element or interface area has multiple problems you want to list those separately. So, for example here, lets go back to our Google Maps interface. In this case I have routed myself from Amsterdam to Raqqa, or Ar-Raqqah, which is a very heavily fought over area of the Syrian Civil War. And Google will again allow me to route myself there, kindly telling me that the route has tolls and may cross country borders, but not informing me of the danger that will result. To me there are two problems here, we have our old problem, which is it's routing us through Syria through some dangerous areas. But a second problem is that it's allowing me to route myself to Ar-Raqqah, to this very dangerous place. One might argue that Google, in fact I would actually argue, that Google should put severe warnings in place. Or just not provide that route for the benefit of their users, right? So for me there's two separate problems here, both error prevention associated, so the heuristic is error prevention. And those two problems are it directs users through very dangerous areas, and allows people to route themselves to very dangerous areas. Okay, so that is what I wanted to say about basically this small group of people trained in heuristic evaluation, examining a user interface for now. But there are two more points to this basic idea behind heuristic evaluation that we want to go through in a bit more detail. I mentioned that we referenced heuristics, let's take about what these heuristics are with a bit more specifics. So again we're going to go through each individual heuristic in more detail in later videos, but this just gives you an idea of what this whole idea of heuristics is. It's important to point out that heuristic evaluation is independent of the set of heuristics that you're using. So you could use a heuristic set from these folks or these folks, but you're still doing heuristic evaluation. These heuristics could be in theory any set of usability principles that have been established over the years. In particular you're going to find yourself using custom heuristics when you're dealing with, again, domain specific areas. So for instance again with the billing systems for older adults, there might be heuristics especially associated with that population, right. For instance font size associated with these types of things. You might add to your standard list of heuristics those domain-specific ones in certain contexts. However, in most cases you're just going to be using this list of 10 common heuristics that were developed by Jakob Nielsen, who along with a colleague developed this whole idea of heuristic evaluation. And these are the heuristics that we're going to iterate through in the next couple videos. So things like error prevention like we saw, of visibility of system status, a user control and freedom, and on and on. So that's the basics of what I want to talk about for heuristics for this video. The last point i want to make just to get across the basic idea, or the last subject I want to cover to get across the basic idea associated with heuristic evaluation, is this notion of severity of ranking or severity assessment. This has a bit of detail to it, so bear with me and we'll walk through it. So, basically what you do in phase one, the problem identification phase, is you generate a list of problems. So routes people through dangerous areas, directs people to dangerous areas, these types of things, right? But you won't have that severity score. What you'll do is you'll have each independent evaluator, typically three to five, generate this list of problems. Then someone will aggregate this list of problems, send around the list of problems to all the evaluators. And each evaluator will provide a severity assessment on all problems, even ones that they didn't find. Severity assessment are typically done on a scale from something like zero, so not problem to four, my goodness stop the presses. We can't launch this system, our entire company will go under, maybe a little less than that. But problems that essentially will severely sink the system, if the system is launched with those problems. And the general factors you consider when making an assessment are these four. You think about the impact of the problem. You think about the persistence of the problem. You think about the frequency of the problem. And let's talk about frequency and impact here. The impact of routing yourself through Syria right now is very, very high, potentially life threatening, but the frequency is probably pretty low. Google probably doesn't have too many people who generate these directions to places in Syria or through places in Syria, and actually take those directions. Drive that car all over through Europe, down through Turkey into Syria and, for instance, into Jordan. Although, we're finding in our research that some people do this type of thing, so it is something to be aware of. And so the last thing is, actually I should back up for a second, that is why in normal cases of life threatening, a system that has it's users do life threatening thing, that's going to be a four right. That's a problem that's a four, but because of frequency, in my view, right. If I were doing this for real I probably would have a good sense of the actual frequency. But in my view, the frequency of that routing problem is going to probably pretty low. Okay, last thing here is marketing effects, obviously this is sort of the financial version of impact, right. If some problem is going to cost the company millions of dollars, that's obviously a very important thing, or maybe give the company a bad reputation, these types of things. I have a bit more specifics for you, which I'm sure you'll find quite useful, at least my students in the past have. Nielsen, a guy who popularized and had a large role in heuristic evaluation. He provides a standard severity scale. Zero is again, this are quotes from one of his documents. Zero is i don't agree that this is a usability problem at all. So maybe you got a problem from one of your fellow evaluators. And you said, no i don't think this is a problem at all. One is a cosmetic problem only, it doesn't really needs to be fixed. If there's extra time, we might go ahead and fix. Minor usability problem, fixing this problem, should be given low priority. Three is a major usability problem, so it's important to fix, given high priority. And then we have our four which is a catastrophic problem, right? Some of them you may recall from the video all the way back in course one, when I talked about some cases where usability issues had led to deaths of the system's users, those would be definite fours. Okay, and then just to clear up a common point of confusion, you use the average of all your evaluator scores. And then you have a list of problems to work on and it can be very convenient. Okay, so that's the basic idea behind heuristic evaluation. Let's talk a bit about the context in which heuristic evaluation is applied. I had a lot of text in the last couple of slides so I wanted to liven things up here with a bit of graphics, so bear with me. Two points, two very important points, about the context that heuristic evaluation typically is applied in. One is that it's applied throughout the design cycle, right? So heuristic evaluation can be done on paper prototypes. It can be done on nearly complete systems. It can be done on live systems, just as I showed you. So heuristic evaluation will identify problems, then you'll go back into other phases in the design process, you'll fix those problems. Heuristic Evaluation can identify more problems, and so on and so forth. This is a very important, the second point on the right here, is a very, very important point. And that is that heuristic evaluation is a compliment to and not a replacement of user testing. So heuristic evaluation is extremely low cost relative to user testing. In fact, it's called a discount usability technique, so it's very efficient in terms of dollar spent and time spent to problems found. But it won't find other problems that you'll find with user testing techniques like we'll talk about in this course. You definitely need to do both. Now heuristic evaluation might be something you do early in the design phase. Maybe if you can't find any users yet, if you feel like your prototype isn't ready for users, heuristic evaluation might be great. There might be other contexts where you can't find users. But if you really want to make sure your system is up to snuff you've got to do both of these. And in fact, they have been found to find quite different problems. Okay, a bit more about the context in which heuristic evaluation is applied. It's very important to point out that the focus of heuristic evaluation is not on problem solving, but on identifying problems. Now that said, heuristic evaluation does have this nice property of linking problems to well known usability principles. Which often, in my view at least, allow you to bridge problems that you find in the interface to techniques, and ideas, and solutions that you've learned in the rest of this course. So for instance, with the error prevention problem, with Google Maps, you might have a technique from another part of this course you can use to fix it. You'll learn that there are these mappings between certain heuristics and certain solutions. In certain problem contexts and certain solutions, problems that arise through heuristic evaluation. So in practice I've found that while heuristic evaluation is not intended to solve problems, it certainly can help us solve problems as well as, of course, its main its man purpose identifying problems. And, it's important to point out too, that heuristic evaluation, one of it's strengths, relative to user testing, right. Remember that there compliments. One of its strengths is that heuristic evaluation can find minor problems, generally, more effectively than user testing. So, for instance, minor problems with consistency. We'll talk about in the next couple of videos. Okay, so that's all I wanted to say about context. Let's wrap things up quickly with a brief description of how heuristic evaluation came to be. Well, we have largely this fellow, Jakob Nielsen, along with his colleague Rolf Molich, for giving this to us. It's a relatively old technique, it comes from the 90s, but it's still very prominent today. It was developed through a series of academic papers published at the ACM SIG CHI Conference which you've heard referenced to, probably at this point, a number of times in the course. And one paper I want to talk about in a bit of detail, one of these three papers is this third one that was published in 1994. And this is a screen grab from the PDF of the paper, it looks roughly like this. What Nielson did in this paper is quite clever especially for the time. He took a series of lists, of usability principles that were in practice here and there and these other places, and he used a clustering technique called factor analysis to essentially find the usability principles that were similar to one another. So a cluster of these all have to do with consistency, these all have to do with error prevention, these types of things. This clustering was done mathematically. And from this list we have our current list of heuristics which we'll we talking about in the next couple of videos. Okay, so that's it for me, or for this video, and in terms of the introduction to heuristic evaluation. I'll see you in the next video to talk about specific heuristics.
