This course introduces innovative approaches to learning and teaching, with a focus on the use of e-learning and social web technologies.
16.5 Select and Supply Response Assessments
Loading...
Do curso por University of Illinois at Urbana-Champaign
Multimodal Literacies: Communication and Learning in the Era of Digital Media
13 classificações
University of Illinois at Urbana-Champaign
13 classificações
Na lição
Module 4: Literacies and Learner Differences + Literacies Standards and Assessment
The final module of the course examines the question of learner differences – including literacies learning at different age levels and second language learning. We also explore strategies for differentiated instruction. Finally, we investigate the range of assessment strategies that can be used to diagnose learner needs, offer feedback during the learning process, and evaluate learning outcomes.
Conheça os instrutores
Dr William CopeProfessor
Department of Education Policy, Organization & Leadership, College of Education
Dr Mary KalantzisProfessor
College of Education
I'm going to talk about assessment types.
I'm going to talk particularly about reading and
I'm going to talk particularly about writing.
And in a way, historically there have been two major types of assessment,
select response assessment and supply response assessment.
In fact, towards the end of this little sequence of videos,
now, I'm going to talk about another new emerging
form of assessment around big data, but let me get to that later.
In the past or until now, there have really been two main assessment types.
What are select response assessments?
Well essentially it's item-based assessment where a question is asked and
there's A, B, C, D answers.
Let's say C is correct.
So what we have to do IS we have to give plausible answers to A, B, and D, okay?
One of those plausible answers has to be pretty close to the C, and these
ones that are close to the C, the correct answer, are called distractor items.
They're actually designed to trick you.
They're actually designed to lead you into the wrong answer because it seems right.
Right, so the funny thing is about select response assessment is, it's a game.
You know you might say yes is the correct answer.
You learn or you can be taught methods of approximation and
methods of process of elimination.
Working out what the trick is in the distractor item.
They were actually, it's a game.
It's a technical thing to learn how to do a select response assessment.
And one of the problems with select response assessments is
in the world it isn't always the case that sees right from one perspective.
One of the other things might be right, and
your incorrect answers might have a certain logic to them.
Or your correct answer may be an accident or a mistake, or
based on faulty reasoning.
What do we do?
We have to have quite a few questions before we can eliminate
those kinds of vagaries in a test, and hopefully by the time we've done that
scoring across all the questions, we've got a reasonable kind of score.
And then there's a whole lot of mathematical voodoo,
statistical voodoo that we do to compare items across different groups.
And hopefully we try and make the test half reasonable and half fair.
One of the things about it, though,
is that it involves a kind of indirect set of influences about my thinking.
So I've got these questions, and it doesn't,
the questions don't really say that I understand the logic of science or
the logic of math or the logic of literacy.
They don't really tell us that we, what it's about,
it's about creating statistical inferences from those, from that reality.
And in fact there's a whole science who has a lot
response theory, which tries to calibrate this process and
ensure the inferences of, are more or less half correct.
Now, let's think about what we do with reading
reading is essentially accessible via select response assessment.
It's the main form of assessment for reading.
And it's essentially about reading comprehension.
And with select response assessment come a whole lot of assumptions.
Firstly, if I'm reading a novel,
the really relevant question is, you know, which characters do you relate to and why.
But in fact, the kind of questions I can ask about that novel in a select response
assessment, are not interpretive because interpretations is variable
in terms of who the person is.
There are often things which are factual,
there are often things which are empirical details, which often not central.
In fact, so not central that often to answer some of these questions in
a comprehension test you've gotta go back to the text and look it up.
Because there may have been details that you didn't notice,
which do in fact have a correct or an incorrect answer.
And one of the tragedies of the assessment fetish that we've had over the last
couple of decades of assessment obsession,
is that reading comprehension has become a proxy for literacy.
Writing assessments are expensive to run, reading assessments
with the little check boxes or if they're computer based they're inexpensive.
They can be standardized with using all these statistical techniques.
So in the realm of literacy reading comprehension has become a proxy for
literacy, a measurement of literacy skills.
Now again one of the problems is a lot of reading is interpretation,
that's left out.
There's a lot of the aspects of reading
which are profoundly important which get left out.
There's a whole theory From Roland Bart or
Humberto Echo of people who, reader response theory.
Who talk about the role of the reader as they mix their subjectivity, and
their identity with the text they see meaning and purpose in the text.
Which of course is going to be variable from one reader to the next.
That all has to be cut out.
All we do is we do comprehension.
And comprehension is based on the assumption that it's possible for
me to understand what is actually said in a text.
What is actually said in a text?
I mean, if I read a political text or an op-ed piece.
I'm looking for things that I want to find and it's not.
What's actually said,
what the author actually means is a very funny, very limited concept.
So the worry about select response assessment in relation
to comprehension is that it's pretty narrow.
Now we can also do select response assessment around vocabulary,
around grammar.
If we believe there's a single absolutely rigid formal grammar and
there are a set of rules.
There are a number of other areas in literacy where we can do select response,
but in fact it's mostly used in the area of comprehension.
Now the second major assessment type that I want to talk about here is
supply response, which is, you're right, a sentence for
the answer, you fill in the gap with a word or you write and essay.
So you're given a prompt, it might be now interestingly an interpretation
of a character in Shakespeare, or a novel that you've read.
Or it might be a supplier response., where we're interested in the quality
of your writing, and you write a narrative of some sort.
Or it might also be writing a science essay.
It might actually be using writing as a means of representation of a scientific
argument, which again you can't really do by select response assessment.
You need to write a piece of continuous text for that to be accessible.
Now, writing assessment has a whole lot of challenges to it.
And it gets neglected a lot in these big, high-stakes, standardized assessments.
It gets neglected just because it's really expensive.
Someone's gotta read that text.
You have a machine look at all the little check boxes in
a select response Assessment.
but to read the text, and also there are all the vagaries of human interpretation.
So one reader might think the text is very good,
another might think it's not so good.
So, what you've got to do is have moderation processes,
where you check out whether this particular reader is always tough
of their, on the people who are reading it and this person's more generous.
You have to build very very specific criteria which describe writing levels and
use rubrics.
It's expensive and it's time consuming and therefor it often gets neglected.
Now in the supply response area,
there are some more recent technologies around natural language processing,
which use machines to assist those texts.
So I'll explain to you briefly how they work, and
then I'll explain to you what some of the problems are.
So how they work is this.
If I've got 300 already humanly graded, text which have been properly moderated.
And we're sure that the ratings are pretty good as we've looked at
the rubrics very carefully, and multiple people have checked them.
And we feed those into the machine.
And usually there are three or four rating levels.
There's never much more than that.
We feed those things into the machine, and
the machine says these hundred are ones.
These hundreds are twos.
These hundreds are threes.
These hundreds are fours, on the writing levels.
And then what the machine does is the next essay that comes in.
It does a statistical analysis of that text, the range of vocabulary,
the number of words, the syntax used.
And what it does it compares those to the humanly rated ones statistically and
says okay, well this one looks like it's a one and
this one looks like it's a two because statistically it's similar.
So these are the texts.
So that's called natural language processing.
It's also fraught because what it's measuring is not what's being said.
Not the ideas, not the quality of the argument, but
in fact a number of very formal statistical things about language.
How many syllables, how many syllables in the word, and
are they multisyllabic words, and a whole lot of stuff like that.
But in fact it doesn't measure the quality of ideas.
And if you ever wanted to do a fun theme, if you regardless this as fun,
we did this as fun theme.
You can write an absolute nonsense text remembering about four or
five rules about what's being measured and you can get a great score.
So it's sort of it's fraud but the thing about it, which is kind of interesting.
And the reason why these natural language processing
systems that are being used even for high stakes supply assessment tests now,
is that in fact they end up, being about as reliable as human writers.
And human writers are, in their nature, unreliable.
These are hard things to judge, written text and, the machine assessment,
on these different kinds of measures, happens to be about right.
So one thing, for example, is, there's a correlation between the score you get,
and the length of the text.
So that kind of makes sense in a funny kind of way.
So if you ever told kids what was at the bottom of the NLP assessment
you give them five rules on how to get great scores.
Write a lot is one, use multi slavic words there’s another one,
write relatively long sentences there’s another one so
you know we could actually give them a set of rules of what.
Statistically correlates with more successful writing in school.
And the text wouldn't necessarily be more profound in terms of the intellectual
content.
So it's kind of fraught.
So anyhow they're at the moment, the main types of tests that we have.
And both of them now have been very heavily mechanized.
Select response assessments are now via computers, and in fact we have these very
sophisticated computer adaptive tests, computer diagnostic tests these days.
And supply response assessments increasingly use
natural language processing technologies, but there are problems with each of them.
And in a way, we need to be aware of the nature of these technologies and
the problems that there are with them.
Explore nosso catálogo
Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.
O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.
© 2018 Coursera Inc. Todos os direitos reservados.
