This course presents research conducted to increase our understanding of how data collection decisions affect survey errors. This is not a “how–to-do-it” course on data collection, but instead reviews the literature on survey design decisions and data quality in order to sensitize learners to how alternative survey designs might impact the data obtained from those surveys. The course reviews a range of survey data collection methods that are both interview-based (face-to-face and telephone) and self-administered (paper questionnaires that are mailed and those that are implemented online, i.e. as web surveys). Mixed mode designs are also covered as well as several hybrid modes for collecting sensitive information e.g., self-administering the sensitive questions in what is otherwise a face-to-face interview. The course also covers newer methods such as mobile web and SMS (text message) interviews, and examines alternative data sources such as social media. It concentrates on the impact these techniques have on the quality of survey data, including error from measurement, nonresponse, and coverage, and assesses the tradeoffs between these error sources when researchers choose a mode or survey design.
3.1.1 Interviewer Roles, Obtaining Interviews
Loading...
Do curso por University of Michigan
Data Collection: Online, Telephone and Face-to-face
131 classificação
University of Michigan
131 classificação
Curso 2 de 7 em Specialization Survey Data Collection and Analytics
Na lição
Module 3: Interviewers and Interviewing
In this lesson, we explore the various roles interviewers take on beside asking questions and collecting answers, as well as some of the different approaches to interviewing that have been proposed and how they affect the accuracy of responses. By the end of Lesson 3, you will appreciate the benefits and costs of collecting data in interviews and will be able to contrast them with the costs and benefits of self-administration.
Conheça os instrutores
Frederick Conrad, Ph.D.Research Professor, Survey Methodology
Institute for Social Research
Welcome back, we are beginning our third lesson on interviews and interviewers.
We'll first talk about a number of tasks that interviewers engage in that
really are not about conducting interviews per se, but
they're kind of preliminary activities.
So recruiting sample members, that is placing phone calls to,
or knocking on the doors of households.
And essentially persuading sample members to become respondents.
And also, within household selection or within household sampling,
which is an additional layer of sampling that interviewers
are sometimes required to engage in to select one respondent from
the household once the interviewer has obtained agreement in general
this kind of extends the sampling design into the household.
Then we'll talk about interviews themselves, and
talk about a number of different interviewing techniques which are often,
sort of presented in counterpoint to one another, or in opposition to one another.
So, for example, we'll talk about the standardization debate,
in which on the one hand, there's a strong argument for standardizing the wording,
requiring interviewers to say essentially use the same words
in every interview they conduct versus allowing interviewers to choose
their words in a way that will essentially standardize the Interpretation.
So if it takes different words to establish the same meaning across
respondents, interviewers are able to do that under this second view.
So this is known as the standardization debate.
Another distinction is made between what's called personal style of interviewing
in which interviewers attempt basically to establish rapport with respondents, and
the formal style of interviewing, which is more business like.
And the question is as with the previous distinction
which of these produces higher quality data?
Because the personal style is, essentially, about establishing rapport,
we'll spend a little time of talking about rapport between interviewers and
respondents, and how this may help, or hurt, data quality.
And then, finally, we'll cover what are known as interviewer effects.
So, while interviewers can add value to the Enterprise in many ways, they can
also introduce error that isn't present in self-administered questionnaires.
And these interviewer effects come within two varieties.
One is biased in which interviewers,
different interviewers elicit responses in different directions, so for
example one set of interviewers might elicit more.
Pro-feminist responses then another set of interviewers question concerning feminism.
And it turns out that it's the gender of the interviewers that seems
to be responsible for this.
So female interviewers may elicit more pro-feminist responses than
male interviewers.
So this is called bias because the error is in one direction.
Or interviewers may increase the response variance by
behaving differently from one another and we'll focus on the behavior of probing,
which is the kind of one unscripted part of standardized interviews.
If interviewers do this differently the idea is that
response variance will increase as a result.
Let's spend a little bit of time talking about the different roles that
interviewers play.
As I said, they do a number of tasks besides asking questions and
recording answers, and each of these can
affect the different sources of error that we've been talking about.
So when it comes to sampling and coverage, they have an impact on coverage by
locating and contacting sample households and determining which units are eligible.
Some structures are not residential for example, and would be ineligible in
a survey that requires residential structures to be included.
And they sample within the household as I mentioned a moment ago.
And to the extent that this is successful or
not can have an impact on sampling error.
When it comes to non-response error or response rate at least interviewers do
have the task of contacting the household and gaining cooperation,
potentially persuading reluctant household members to become respondents and this can
vary between interviewers again to the extent that this is done effectively
the response rate should be high but it does vary between interviewers.
Interviewers are probably most closely associated with measurement error,
because they are asking the questions which are the measurements in
conventional surveys, where self report is the predominant measure.
So, they ask the questions, they probe for complete answers, they clarify questions,
and they record answers, so what can go wrong here?
Well interviewers can ask questions in different ways.
They can probe differently, as I mentioned a moment ago.
The questions may be misunderstood, and interviewers will be unaware of this or
unable to address misunderstanding because, for example,
the interviewing technique that they've been trained in and are required to use
requires them to stick to the script, and if departing from the script is necessary
to clarify questions that are really not able to do that.
So interviewers can contribute to these different sources of error, more or less.
So they can actually reduce these sources of error.
So for example, response rates are generally higher with interviewers than in
self-administered questionnaires.
So interviewers both add value and introduce error.
For better or worse, they can have an enormous impact on costs and timeliness.
They are usually the single largest expense in
a study that involves interviewers.
That is more money spent on interviewers than anything else when
there are interviewers in the study.
So, it's a very Important decision, whether or
not the mode involves interviews, and one that researchers should make carefully.
So, turning to recruitment, one of these tasks
that interviewers engage in that really don't involve conducting interviews.
Some interviewers are more successful than others in obtaining interviews.
If you look across studies and a system called lifetime response rate.
So what might be responsible for this?
Well it could have to do with some interviewers just being better
at recruiting perhaps because they're more extroverted or their personalities just
lend themselves to persuading strangers to take part in the study.
Another possibility particularly for telephone interviews
is that it may have to do with how they speak and interact with respondents.
So that's actually what we'll turn to next.
On the telephone, really the only information that
a household member has about the interviewer who's trying to persuade them
to take part in the study Is what they sound like and how they speak.
Really, telephone interviewing may sort of concentrate sample members
on interviewers' vocal properties and the way they interact.
In one study, Conrad and his colleagues Coded the behavioral attributes
of interviewers in 1300 telephone survey introductions.
These are audio recorded telephone introductions from
five studies that were conducted at the University of Michigan.
And the behaviours they coded included fluencies like um's and ah's.
What are known as back channels these were by the household member the sample member
not by the interviewer, back channels are for
example saying uh-huh and nodding although of course
on the telephone the interviewer would never see the household member nodding but
saying uh-huh is generally considered an acknowledgement,
a way to say to the speaker, yeah please continue I'm still listening,
and over speech and interruptions, and that's called simultaneous speech.
The idea being that these generally signal
some sort of troubled interaction if there's a lot of overspeech.
And they modeled the impact of these different behaviors
on one of three outcomes.
Agreement to take part in the study, refusal scheduled callback which
basically refers to deferring the decision until later so
the interviewer and the household member agree that someone,
maybe not the interviewer there currently talking, but somebody will call them back
in the future to talk about a possibility of taking part in the study.
So a scheduled callback is kind of In between it agreement and refusal and
so they were looking at these three outcomes on the basis of behaviors like
disfluency a household member back channels and over speech or interruptions.
So the first finding was that fillers or disfluencies like ums and
uhs, are both good and bad when it comes to participation.
So this figure shows us the filler rate, or the disfluency rate, per hundred words.
And the rates are broken into these quintiles.
The y-axis is the proportion of degrees as a function of the filler rate.
So you can see when interviewers are making zero disfluencies,
that is when they're highly fluent.
They're actually quite unsuccessful in recruiting household members.
This is actually the lowest agreement rate of all the quintiles.
As soon as they make even a small number of disfluencies,
this is more than 0 and less that 1.28 per 100 words, so not a lot of disfluency.
Agreement rate goes up substantially to 0.35.
It tends to drop off as they get more until,
when they're making the maximum number of disfluencies, fillers, greater than three
point four nine per hundred words, it's at it's lowest when there are any fillers.
However, even when interviewers are highly disfluent,
they're more successful than when they're perfectly fluent.
And the thinking was that, when they're perfectly fluent, they sound robotic.
And they don't seem to be tailoring the invitation,
the recruitment speech for the particular household member.
That seems to be the big disincentive to participate.
Another speech behavior that's involved in recruitment is, as I mentioned,
backchannels.
The household members produce more back channels uh-huh and okay or I see.
Where they ultimately agree.
This figure shows that the proportion of lose or
conversational terms with a back channel by the household member.
And as you can see, scheduled callbacks are in between agrees and refusals, but
really when they're signaling that they are paying attention and
sort of authorizing the speaker, or
in this case the interviewer to continue, they're most likely to agree.
And then, the third behavior that the authors looked at was overspeech or
simultaneous speech.
As I said before, when there is a lot of overspeech,
it generally indicates that this is an interaction that's not going well.
And as you can see, overspeech is highest when the invitation, the case,
ultimately ends in refusal.
A scheduled callback is lower, but
not as low as when the invitation led to agreement.
So the amount of over speech in an invitation
is pretty clearly related to the outcome.
However, not all overspeech works the same way.
What the authors looked at was what they called interruptions,
which are actually just what they sound like.
This is overspeech in which the household member was speaking and
then the interviewer began to speak before the household member had finished.
This type of overspeech seemed to have been used quite strategically
by the interviewers to essentially block a refusal.
So as you can see, when the invitation ended in a scheduled call back,
which again is a deferral to another time to make the decision
interviewer interruptions of household members was at its highest.
And again that the idea that the authors proposed is that they were preventing or
interrupting an utterance that would have led to a refusal
had they not intervened at that point.
So that's one set of examples of how the speech and
vocal attributes of interviewers and how they interact with household members and
how household members speak in response can effect outcomes.
Another study published at about the same time by Nora Cate Schaeffer and
her colleagues looked at a pairs of respondents
with the same propensity to participate.
So what they did was, they modeled the likelihood to participate on the basis
of a number of variables in their dataset, and this is a longitudinal dataset so
they had information about these household members going back a number of waves,
meaning previous interviews.
They found pairs who had the same likelihood,
same propensity to participate, only one member of the pair
accepted an invitation in the wave they were looking at.
And the other member of the pair declined.
So they have the same likelihood of participating but different outcomes.
What can be going on?
So they looked at the speech of both interviewers and I'm calling them
respondents here, but again they're household members because they haven't yet
agreed to do this interview.
Using conversation analytic techniques similar to, a little different than what
was used in the Conrad et al study when they coded the interactions.
They uncovered a number of Interesting provocative explanations for
this difference.
One thing that I look at it was when did at these
sample members decline the invitation and when did they accept it?
That's at what point in the invitation.
So as you can see if you look at declinations, they're occurring at various
points during the interview not exactly even numbers but with non-zero frequency.
So during the opening, during the study description, and after the request for
participation.
When you look at acceptance,
that's only happening after an explicit request for participation.
So you can see that household members,
sample members are feel no qualms about declining at any point.
It's much harder to elicit an acceptance, and
that really requires an explicit request.
Will you take part in our study?
Essentially.
They also looked at particular interviewer actions and
how they affected the odds of participating.
The odds ratio in a logistic regression.
So, if the interviewer said hello versus hi, after the respondent
answered the phone, the odds ratio was 1.44, which is marginally significant.
When the interviewer requested to speak to a sample member in a very polite way,
versus a not so polite way.
The odds also were greater that the invitation would lead to an interview,
marginally significant but, you can see, in the right direction.
And, when the interviewer introduced herself, for identified herself or
himself first versus second,
the odds were again higher that the invitation would lead to an interview.
In this case significantly higher.
Finally, they looked at a number of sample member behaviors
that seemed to indicate a willingness and interest in participating.
Or an engagement in the process that would make them more likely to participate.
So, if the sample member asked at least one question.
They were reliably more likely to participate than if they ask none.
If a sample member asked about the or
said something about the length of the interview, how long is this going to take.
Much more likely to participate than if they had no.
Nothing to say about the length as if to suggest that it would only be thinking
about how long will this take if they were considering participating.
If the sample member asks at least one W-H type question like what is this about,
the odds are less likely they'll participate than if they don't ask
such a question.
So this seems to indicate a kind of skepticism or reluctance to participate.
The effect is even stronger if they ask this question
before the interviewer requests that they participate.
But if they ask a WH type question after the interviewer asks them to participate
there's no difference, essentially, between asking that question and
not asking that question.
Because by that time the household member, sample member, has made up his or
her mind about whether they would participate or not and
so these questions have a kind of different significance at that point.
So these are some examples of how interviewers' speech and
how they interact with household members.
Can affect the outcome of their recruitment efforts,
at least on the telephone.
This is among the kind of non-interview tasks that interviewers engage in.
Next we'll turn to within household sampling, or respondent selection,
which is an interviewer activity that follows acceptance of the invitation or
agreement to participate by the household member who first spoke to the interviewer.
That person doesn't necessarily become the respondent but is considered the gate
keeper, and if that person agrees to participate another
household member may still be selected and may have to be convinced to participate.
This process of within household sampling or respondent selection
assures that any eligible member of the household can potentially be selected to
be the respondent and that's important in implementing the sample design.
Explore nosso catálogo
Registre-se gratuitamente e obtenha recomendações, atualizações e ofertas personalizadas.
O Coursera proporciona acesso universal à melhor educação do mundo fazendo parcerias com as melhores universidades e organizações para oferecer cursos on-line.
© 2018 Coursera Inc. Todos os direitos reservados.
