I'm going to talk about assessment types. I'm going to talk particularly about reading and I'm going to talk particularly about writing. And in a way, historically there have been two major types of assessment, select response assessment and supply response assessment. In fact, towards the end of this little sequence of videos, now, I'm going to talk about another new emerging form of assessment around big data, but let me get to that later. In the past or until now, there have really been two main assessment types. What are select response assessments? Well essentially it's item-based assessment where a question is asked and there's A, B, C, D answers. Let's say C is correct. So what we have to do IS we have to give plausible answers to A, B, and D, okay? One of those plausible answers has to be pretty close to the C, and these ones that are close to the C, the correct answer, are called distractor items. They're actually designed to trick you. They're actually designed to lead you into the wrong answer because it seems right. Right, so the funny thing is about select response assessment is, it's a game. You know you might say yes is the correct answer. You learn or you can be taught methods of approximation and methods of process of elimination. Working out what the trick is in the distractor item. They were actually, it's a game. It's a technical thing to learn how to do a select response assessment. And one of the problems with select response assessments is in the world it isn't always the case that sees right from one perspective. One of the other things might be right, and your incorrect answers might have a certain logic to them. Or your correct answer may be an accident or a mistake, or based on faulty reasoning. What do we do? We have to have quite a few questions before we can eliminate those kinds of vagaries in a test, and hopefully by the time we've done that scoring across all the questions, we've got a reasonable kind of score. And then there's a whole lot of mathematical voodoo, statistical voodoo that we do to compare items across different groups. And hopefully we try and make the test half reasonable and half fair. One of the things about it, though, is that it involves a kind of indirect set of influences about my thinking. So I've got these questions, and it doesn't, the questions don't really say that I understand the logic of science or the logic of math or the logic of literacy. They don't really tell us that we, what it's about, it's about creating statistical inferences from those, from that reality. And in fact there's a whole science who has a lot response theory, which tries to calibrate this process and ensure the inferences of, are more or less half correct. Now, let's think about what we do with reading reading is essentially accessible via select response assessment. It's the main form of assessment for reading. And it's essentially about reading comprehension. And with select response assessment come a whole lot of assumptions. Firstly, if I'm reading a novel, the really relevant question is, you know, which characters do you relate to and why. But in fact, the kind of questions I can ask about that novel in a select response assessment, are not interpretive because interpretations is variable in terms of who the person is. There are often things which are factual, there are often things which are empirical details, which often not central. In fact, so not central that often to answer some of these questions in a comprehension test you've gotta go back to the text and look it up. Because there may have been details that you didn't notice, which do in fact have a correct or an incorrect answer. And one of the tragedies of the assessment fetish that we've had over the last couple of decades of assessment obsession, is that reading comprehension has become a proxy for literacy. Writing assessments are expensive to run, reading assessments with the little check boxes or if they're computer based they're inexpensive. They can be standardized with using all these statistical techniques. So in the realm of literacy reading comprehension has become a proxy for literacy, a measurement of literacy skills. Now again one of the problems is a lot of reading is interpretation, that's left out. There's a lot of the aspects of reading which are profoundly important which get left out. There's a whole theory From Roland Bart or Humberto Echo of people who, reader response theory. Who talk about the role of the reader as they mix their subjectivity, and their identity with the text they see meaning and purpose in the text. Which of course is going to be variable from one reader to the next. That all has to be cut out. All we do is we do comprehension. And comprehension is based on the assumption that it's possible for me to understand what is actually said in a text. What is actually said in a text? I mean, if I read a political text or an op-ed piece. I'm looking for things that I want to find and it's not. What's actually said, what the author actually means is a very funny, very limited concept. So the worry about select response assessment in relation to comprehension is that it's pretty narrow. Now we can also do select response assessment around vocabulary, around grammar. If we believe there's a single absolutely rigid formal grammar and there are a set of rules. There are a number of other areas in literacy where we can do select response, but in fact it's mostly used in the area of comprehension. Now the second major assessment type that I want to talk about here is supply response, which is, you're right, a sentence for the answer, you fill in the gap with a word or you write and essay. So you're given a prompt, it might be now interestingly an interpretation of a character in Shakespeare, or a novel that you've read. Or it might be a supplier response., where we're interested in the quality of your writing, and you write a narrative of some sort. Or it might also be writing a science essay. It might actually be using writing as a means of representation of a scientific argument, which again you can't really do by select response assessment. You need to write a piece of continuous text for that to be accessible. Now, writing assessment has a whole lot of challenges to it. And it gets neglected a lot in these big, high-stakes, standardized assessments. It gets neglected just because it's really expensive. Someone's gotta read that text. You have a machine look at all the little check boxes in a select response Assessment. but to read the text, and also there are all the vagaries of human interpretation. So one reader might think the text is very good, another might think it's not so good. So, what you've got to do is have moderation processes, where you check out whether this particular reader is always tough of their, on the people who are reading it and this person's more generous. You have to build very very specific criteria which describe writing levels and use rubrics. It's expensive and it's time consuming and therefor it often gets neglected. Now in the supply response area, there are some more recent technologies around natural language processing, which use machines to assist those texts. So I'll explain to you briefly how they work, and then I'll explain to you what some of the problems are. So how they work is this. If I've got 300 already humanly graded, text which have been properly moderated. And we're sure that the ratings are pretty good as we've looked at the rubrics very carefully, and multiple people have checked them. And we feed those into the machine. And usually there are three or four rating levels. There's never much more than that. We feed those things into the machine, and the machine says these hundred are ones. These hundreds are twos. These hundreds are threes. These hundreds are fours, on the writing levels. And then what the machine does is the next essay that comes in. It does a statistical analysis of that text, the range of vocabulary, the number of words, the syntax used. And what it does it compares those to the humanly rated ones statistically and says okay, well this one looks like it's a one and this one looks like it's a two because statistically it's similar. So these are the texts. So that's called natural language processing. It's also fraught because what it's measuring is not what's being said. Not the ideas, not the quality of the argument, but in fact a number of very formal statistical things about language. How many syllables, how many syllables in the word, and are they multisyllabic words, and a whole lot of stuff like that. But in fact it doesn't measure the quality of ideas. And if you ever wanted to do a fun theme, if you regardless this as fun, we did this as fun theme. You can write an absolute nonsense text remembering about four or five rules about what's being measured and you can get a great score. So it's sort of it's fraud but the thing about it, which is kind of interesting. And the reason why these natural language processing systems that are being used even for high stakes supply assessment tests now, is that in fact they end up, being about as reliable as human writers. And human writers are, in their nature, unreliable. These are hard things to judge, written text and, the machine assessment, on these different kinds of measures, happens to be about right. So one thing, for example, is, there's a correlation between the score you get, and the length of the text. So that kind of makes sense in a funny kind of way. So if you ever told kids what was at the bottom of the NLP assessment you give them five rules on how to get great scores. Write a lot is one, use multi slavic words there’s another one, write relatively long sentences there’s another one so you know we could actually give them a set of rules of what. Statistically correlates with more successful writing in school. And the text wouldn't necessarily be more profound in terms of the intellectual content. So it's kind of fraught. So anyhow they're at the moment, the main types of tests that we have. And both of them now have been very heavily mechanized. Select response assessments are now via computers, and in fact we have these very sophisticated computer adaptive tests, computer diagnostic tests these days. And supply response assessments increasingly use natural language processing technologies, but there are problems with each of them. And in a way, we need to be aware of the nature of these technologies and the problems that there are with them.
