What Can Response Latencies Do For Pretesting?

(together with Maike Reimer, Max-Planck-Institut für Bildungsforschung)

We examine whether response latencies measured in online surveys can be used to identify problematic features of questions and questionnaires by pointing to the increased cognitive effort such problems pose to the respondent. We compare the response latencies for well−formulated and problematic questions and control for individual differences in response accessibility.

We find that response latencies can identify question problems, which bring about increased cognitive effort, such as awkward question wording. In other cases, when problematic questions lead to less cognitive effort, response latencies can be inconclusive or even misleading.

Figure 1: The stages of question comprehension and answer generation.
Figure 1: The stages of question comprehension and answer generation.

Theory

Question comprehension is the first of four cognitive steps involved in the response generation process, which influences response latencies (figure 1). Questions that are hard to understand increase cognitive effort and response latencies. Cognitive effort at the other stages also influences response latencies, especially the ease of judgment computation and information retrieval (stage 2): strong attitudes can be retrieved more quickly than weak ones because they are more accessible. Extreme frequencies are reported more easily and more quickly because less effortful strategies are used to develop a response. We compare response latencies between well− and badly−formulated questions and show that difficulties at the question comprehension stage influence response latencies independently of other stages. It can, therefore, be used to identify problematic questions and questionnaire features.

Design & Method

In a one−factor between−subjects comparison, respondents are first confronted with eight statements about two topics, which are randomly presented in a well− or badly−formulated version and rate their agreement on a 7−point Likert−type scale. Secondly, a filter question asks which Internet services are used by the respondents. Afterwards, they are asked for information on their frequency of use of the services, either in accordance with the filter or regardless of it. Self reports about attitude strength and strategies in reporting are asked after each item.

Examples:

Well−formulated question: One should only become politically involved if one believes in the possibility that one can gain influence.

Badly−formulated question (superflous negation): One should not become politically involved if one does not believe in the possibility that one can gain influence.

Sample: 653 respondents from an online−panel of people who volunteered to participate in online−surveys provided answers to our questions. Measure of response latency: The online response latency measure takes the time as soon as a page is loaded and stops when the respondent clicks on the response scale.

Figure 2: Attitude Questions − Mean response latencies of well−formulated questions and badly formulated questions
Figure 2: Attitude Questions − Mean response latencies of well−formulated questions and badly formulated questions
Figure 3: Attitude Questions − Mean response latencies of respondents with strong attitudes and moderate or weak attitudes
Figure 3: Attitude Questions − Mean response latencies of respondents with strong attitudes and moderate or weak attitudes
Table 1: MANOVA −Response Latencies, Question Quality and Attitude Strength
Table 1: MANOVA −Response Latencies, Question Quality and Attitude Strength

Results: Attitude Questions

Badly−formulated questions and weak/moderate attitudes (measured by the extremity of the attitude reported for an item) both lead to longer response latencies. In seven out of eight comparisons, the main effects of question quality reflect the influence of cognitive effort at the question comprehension stage (figure 2). Main effects of attitude extremity present in half of the cases reflect cognitive effort at the response retrieval stage (figure 3). In half of the cases, interaction effects between question quality and attitude extremity are present and attitude extremity indicate possible influences of bad question wording on the responses they generate (table 1). 

Figure 4: Frequency Reports − Response Latencies and Filter Consistency
Figure 4: Frequency Reports − Response Latencies and Filter Consistency
Figure 5: Frequency Reports - Response Latencies and Ratings
Figure 5: Frequency Reports - Response Latencies and Ratings
Figure 6: Frequency Reports −Rating Extremity and Filter Consistency
Figure 6: Frequency Reports −Rating Extremity and Filter Consistency

Results: Frequency Reports

Inconsistent filtering does not lead to increased response latencies (figure 4). In the inconsistent filter condition, respondents report more extreme (lower) frequencies, since these come closest to the (correct) answer never (figure 5). More extreme frequency ratings are associated with estimation strategies that are cognitively less demanding than recall and count strategies and can, therefore, lead to equal or lower response latencies (figure 6). Main effects of strategy reflect the influence of the cognitive effort at the retrieval stage.

Tabelle 2: MANOVA −Response Latencies, Filter Condition and Response Strategy
Tabelle 2: MANOVA −Response Latencies, Filter Condition and Response Strategy

An interaction effect between the filter condition and the retrieval strategy for newsgroup use indicates possible differential effects of question quality on response selection (table 2).

 

Conclusions

In online surveys, response latencies are valid measures of the cognitive effort in response generation, both at the comprehension and at the response retrieval stage. They relate to question quality, to simple indices of ease of response generation and to self reports about response strategies. In order to make response latency measurement useful for pretesting, a comparison between different versions of a question or between a question and a standard is needed. The interpretation of response latencies as indicators of question problems is not straightforward. When question and questionnaire problems lead to a decreased cognitive effort at any of the response generation stages, the response given to a problematic question can be as quick as or even quicker than a response to a flawless one, A deep understanding of the cognitive mechanisms of response generation and its conditions of occurrence is, therefore, indispensable.

Müller, J. & Reimer, M. (2002). Time measurement and pre-testing in Online Questionnaires. Poster presented at the 5th German Online ResearchConference, Hohenheim.

 

 

Deutsch|English