This lecture is going to focus on statistics and usability measures, and hopefully give you some background if you find yourself in a position where you need to gather some specific measure data and then present it in a statistical fashion. We'll talk about the different measures that are often used in usability assessment. We'll talk about some analysis methods. We'll take a quick walk through some best practices for visualizing data. I'm also going to give you some good resources to look at that can support statistical analysis if you find yourself in that kind of work. Statistics and measures, occasionally, you will want to know whether or not the time to do an operation on one version of an interface is better than doing it on another version, or you may want to find out how many errors people are making as they work through an interface. Gathering, presenting, and analyzing that data can be a challenge. It's normally found more in formal testing than in informal testing. But it is possible to use discount methods to gather some of these measures. These are some typical measures, again, success at a task, time on a task, error counts, learnability. If you've used the interface once and then maybe took a tutorial and used it again, is it a better experience? Satisfaction, which you can often get from those pre-validated surveys we discussed, etc. You can decide when it's appropriate to gather these measures for your own projects. Now we have to revisit sample size a bit. Because again, while most surveys probably use around 11 users to gather data from their testing, if you're doing statistical levels of testing and you need to get some certain confidence in that testing, then you're probably going to have to look at 30, 40, 50 users to get that data buildup. Again, qualitative work, we can use small numbers of users. When we certainly look at quantitative measures, depending on how important the strength of this statistics are, we may have to gather more data. There's a couple of really good resources that you can look at if you have to do statistical analysis related to usability. One of them is Tullis and Albert's Measuring the User Experience. It's an excellent book on metrics for usability. There's a companion website that has papers, and spreadsheets, and tools for putting together statistical studies of different measures. I highly recommend you take a look at it. We'll reference it again at the end of this lecture. These tips here actually come from that book. They talk about things that you should consider as you're trying to measure things for usability or for user experience. One thing that you'll notice here is the use of the system usability scale. Tullis and Albert feel that the SUS survey is a really good way to get quantitative satisfaction measures. But regardless of what you do, you're going to want to be careful about how you use confidence limits, how you compare data, how you use frequency distributions when it's appropriate, how your combine metrics can be tricky. All these things have to be considered. But on the other hand, it's certainly reasonable to gather usability metrics, and you can do it even if you're using a small sample size. Don't shy away from it because of cost or because you think the data might be noisy or what have you. It's perfectly reasonable to gather these metrics if you have something in your study that you need them for. It is important when you're considering the data to think about how you're going to analyze data. If you're getting data that's ordinal or nominal that some categories, say I'd like a yes-no answer or a poor, fair, good, excellent measure, those things have to be analyzed as frequencies. There's a set of methods like chi-square analysis to do the analysis of that type of data. Differently from mean data that you might get from intervals, on a scale, on a survey, or ratio data from times, or averages. Those, you can use more standard descriptive statistics: t-tests, ANOVAs, regression, etc. Again, hopefully your engineering statistics classes are kicking it a bit. But these reference books that I'm going to mention can guide you through this type of analysis. One of the things that's really important and even Tufte brings this up, if you're going to present statistical data, it has to be credible and there has to be some integrity behind the graphics that you use, including documenting where the data comes from and how you did the analysis. It's also important when you're showing the data that you do it appropriately. I'm going to walk through some of the most common errors that people make when they're showing data visualization. Here's an example of comparing bar graphs. This is the same data in both graphs. But if you look at that first one, that data looks different, and part of the reason it looks different is they're not starting at zero with the graph, and they're also not comparing the level of error in the data that was gathered. If you look at the confidence intervals graphed onto the data and you start at zero, it's pretty obvious that these aren't that far apart. In fact, their confidence intervals overlap all these from the same base dataset. It's also common to make errors and use the wrong types of graphs when we're looking at data. You don't want to use a line graph when you should be using a bar, because a line implies that there's a relationship between the points. Again, the bar graphs, if you can do it, it should start at zero and should include some measure of error. I'm not a big fan of pie charts in general, but one of the dangers of a pie chart is depending on color to differentiate what it's telling you. It's much better to use labeling that makes it clearer what each of the wedges actually means. Don't depend on color as the only way to get that data. I prefer a bar chart to a pie chart. In fact, stacked bar charts that represent 100 percent can be easily compared to each other, and it makes it easier to tell what's changed from one situation to another. I highly recommend taking a look at that. Again, your resources, the Tullis and Albert Measuring the User Experience is a great book, and a lot of these examples are taken from there. There's another one by Sauro and Lewis. You might recognize their names going back to some of the pre-validated surveys. This book is interesting because it has a companion book that includes some Excel and R-based routines for doing standard analysis. If you're not a statistics maven, one of these two textbooks and some of the supporting information from their websites or from their companion books might prove really valuable to you. Take a look at them when you can. Again, in summary, we don't recommend that you get into statistical analysis for usability unless you have to. If you're doing this generally with qualitative data, this may not ever come up. But if you're trying to make a comparison, trying to convince somebody of something, then statistics starts to come up, and gathering this data might become important. Certainly, using the pre-validated surveys is easy to do to get satisfaction measures, and look at before and after results. Use what's appropriate to your project, putting the right level of rigor that you need to get to a good UX design. Thanks.
