Date:
Eric M. Mindich Encounters with Authors Symposium
"The Handbook of Questionnaire Design: Insights from Social and Cognitive Psychology" featuring Dr. Jon Krosnick.
Discussants:
- Gary Langer, ABC News - Open versus Closed Questions
- Norbert Schwarz, University of Michigan - Response Choice Order Effects
- Adam Berinsky, MIT - "Don't KNow" Response Options
- Stanley Presser, University of Maryland - The Problem of Acquiescence
- Jack F. Fowler, University of Massachusetts, Boston - Designing Rating Scales
- Herb Weisberg, Ohio State University - Designing Rating Scales
- Bob Groves, University of Michigan - Rating versus Ranking
- Barry Burden, Harvard University - Question Wording
The program offers scholars an extended, intensive seminar with the authors of new, path breaking scholarly works. Professor Jon Krosnick of Stanford University will conduct meetings that feature a mix of lecture and discussion components. Unless otherwise noted, sessions will take place at 1737 Cambridge Street, Room N354 (map and directions).
Since the beginning of quantitative social science, a great deal of research has been done using questionnaires, asking people to provide reports describing their mental states, their personality dispositions, their behavioral tendencies, their attitudes and beliefs, and much more. Thus, asking questions and interpreting answers are core activities for social science. It is therefore no surprise that most research methods textbooks in psychology, sociology, political science, and many other disciplines include a discussion of questionnaire design.
Remarkably, the structuring, wording, and ordering of questions has traditionally been viewed as "an art, not a science", in the words of Princeton University psychologist Hadley Cantril (1951, p. vii) over five decades ago. And in his book, The Art of Asking Questions, Stanley Payne (1951) cautioned that "the reader will be disappointed if he expects to find here a set of definite rules or explicit directions. The art of asking questions is not likely ever to be reduced to some easy formulas (p. xi)." Thirty years later, Sudman and Bradburn (1982) agreed, saying that "no codified rules for question asking exist (p. 2)." Sampling and data analysis are indeed guided by such rules that are backed by elaborate theoretical rationales. But questionnaire design has been thought of as best guided by intuition about how to script a naturally flowing conversation between a researcher and a respondent, even if that conversation is sometimes mediated by an interviewer. Experienced questionnaire designers have followed some conventions over the years, but those conventions varied enough from individual to individual and from discipline to discipline to suggest there are few universally-accepted principles. If a questioning approach seems to work smoothly when respondents answer a questionnaire, then many researchers presumed it would probably yield sufficiently useful data.
In recent years, it has become clear, though, that this is an antiquated view that does not reflect the accumulation of knowledge throughout the social sciences about effective question-asking. To be sure, intuition is a useful guide for designing questions, and a good questionnaire yields conversations that feel natural and comfortable to respondents. However, intuition can sometimes lead us astray, so it is useful to refine our intuitions via scientific evaluation. Fortunately, a large body of relevant scientific studies has now accumulated, and when taken together, their findings clearly suggest formal rules about how best to design questions. However, this work has been scattered across the publication outlets of numerous disciplines (e.g., psychology, sociology, and political science, although some work has appeared in marketing, statistics, communication, education, and the health professions), and this literature has not yet been comprehensively and integratively reviewed in a central place. Doing so has been a principal project for me during the past ten years.
Because of the complexity of this literature, it does not yield a short and efficient list of rules, each supported by a few documentary references, and each obviously justified by all relevant studies. The issues addressed are multifaceted, and many are still in the process of being resolved by innovative and creative new research. But there is a great deal of richness in the existing literature that provides useful guidance for scholars interested in maximizing the reliability, validity, and efficiency of the measurement instruments they employ in their research.
The Handbook of Questionnaire Design reviews and integrates the large literature dating back to the turn of the century illuminating the cognitive processes involved in answering questionnaires measuring attitudes. Primarily by experimentally manipulating question format, wording, or ordering, these studies have illuminated how people go about answering attitude questions and how different question formulations can produce quite different answers.
This literature has two important sets of implications. First, it does support numerous clear and practical recommendations regarding how best to construct questionnaires for surveys, laboratory experiments, depth interviewing, or other forms of empirical social research. It also makes clear recommendations about how to interpret questionnaire data and when to be wary about its implications. The primary goal of this book is to highlight a list of practical recommendations to questionnaire writers and analysts of questionnaire data, and to make clear the nature of the evidence justifying each recommendation.
The second primary goal of the book is to reorient the entire conceptual approach questionnaire writers and analysts bring to their tasks. Experimental studies of questionnaire design have highlighted many insights regarding human cognition and communication generally. Taken together, these insights constitute a way of viewing social information processing and social behavior that is currently gaining favor as a result of the cognitive revolution in psychology. At its core, this view contrasts sharply with the conventional wisdom that if you want someone's opinion, all you need to do is ask, and he or she will tell you. The new view asserts that numerous subtle and not-so-subtle aspects of questions can have very potent influences on responses, even among people whose views on an issue are quite strong.
Therefore, writing effective questions and effectively interpreting the meaning of questionnaire data requires careful attention to numerous small details. By explaining this new view, this book will help consumers of questionnaire data to become more sophisticated about the meaning of those data and the degree to which their apparent implications may be contingent upon the particular questions asked. In addition, this new view will help questionnaire writers to become cognitive theorists who can develop and test hypotheses about aspects of question construction that have yet to be investigated systematically.
The third primary goal of the book is to bring together a large and diverse set of empirical evidence scattered throughout many academic disciplines. Questionnaire designers and analysts in any given discipline may well be familiar with the studies in their own literature, but they are unlikely to be informed about those in other fields. By bringing together this wide array of studies, we hope to paint a more vivid and compelling portrait of the cognitive processes involved in questionnaire responding than can be provided by any single discipline's literature.