Preface
By Leib Litman and Jonathan Robinson
Science is shaped by the tools at hand. When Galileo Galilei sought to measure the weight of air, for example, he used a glass bottle, a leather stopper, a syringe, and a bellows. But in the biological and social sciences the right tools can be much more difficult to obtain. In antiquity, Egyptian physicians were highly sought-after by people from all corners of the ancient world because they had acquired a knowledge of anatomy through their practice of mummification. But in the Roman world and during much of the Middle Ages, progress in medicine had all but halted because the use of cadavers was illegal. In the 19th century, before teaching hospitals and before people could donate their bodies to science, students interested in human anatomy used what they could find: amputated limbs, deceased relatives, and often a corpse taken from a fresh grave. More recently, before public opinion permitted the use of cadavers in crash test research, scientists who wanted to understand how the body reacts in collisions so that they could design safer cars sometimes jumped into the crash simulators themselves, risking life and limb to quell their curiosity.
Whereas progress in the medical sciences has been shaped by access to human bodies (or the lack thereof), progress in the social sciences is dependent on researchers being able to interact with human minds. The way in which social scientists sought to do this can be characterized as having proceeded through five stages. In the first stage, roughly spanning the 19th century, pioneering researchers like Gustav Fechner and Hermann Ebbinghaus used themselves as their own research subjects. The second stage began once psychology laboratories were established in German universities. From 1879 through the first decades of the 20th century, Wilhelm Wundt, Edward Titchener, and their students used highly trained professionals, who were usually psychologists themselves, as participants for their studies. By the middle of the 20th century, psychology was an established science. This marked the third stage, during which psychologists mostly used community recruitment to gain access to research participants. For example, in the early 1960s, Stanley Milgram performed what are perhaps some of the most famous psychological experiments ever conducted. To recruit subjects for his studies, Milgram followed what was a standard procedure at the time: he posted fliers around town and asked people interested in the study to mail in slips of paper with their contact information, basic demographic data, and an indication of when they might be able to come to the lab. Once Milgram received people’s responses, he called prospective participants, scheduled a time for the experiment, and ensured that research staff were in the laboratory to conduct each experimental session. Clearly, finding people to participate in the study and actually running them through the protocol was a demanding process, requiring significant time and resources.
Because of the labor-intensive nature of community recruitment, beginning in the 1960s (marking the fourth stage), the vast majority of academic institutions created subject pools through which students were required to participate in research studies. Other sources of information about human beings, such as national datasets, probability surveys, and field research, were also commonly used and still are. By the 1990s close to 80% of all studies in psychology and consumer behavior consisted of participants who were recruited through the university subject pools (Peterson, 2001; Sears, 1986).
The 21st century brings us to the fifth stage of data collection with human participants, which makes up the subject matter of this book. Over the past several decades, technology has rapidly refashioned the tools of social scientists. The personal computer, the Internet, and various forms of browser-based software have made it easier than ever before to find research participants, create engaging study materials, and observe people’s natural behavior in direct and indirect ways. In a relatively short period of time, technology has supplanted the university subject pool and largely replaced paper-and-pencil measures as the primary means of gathering data about people’s thoughts, feelings, and behavior. This change has had a profound effect on all of social science research.
The catalyst for this change was Amazon’s Mechanical Turk, an online platform that began to receive significant attention from research scientists starting in 2010. Since that time, thousands of scientific papers have been published using Mechanical Turk participants. But Mechanical Turk is not the only online platform from which participants can be recruited. Although Mechanical Turk provides many unique advantages, other platforms have much to offer as well. Overall, a variety of platforms are currently available to research scientists, giving them access to tens of millions of participants from around the world.
The proliferation of these online resources brings with it many questions and concerns. How good is the data quality of online studies? Are the data representative? What is the best way to recruit hard-to-reach samples? Can participants be trusted to provide accurate information about themselves? When should one platform be preferred over another? What ethical issues are unique to the online environment? On top of all of that, sometimes just getting started with a new technological tool can seem daunting and raises additional questions.
This book aims to answer these questions. It aims to be a resource for both the beginner and the seasoned online researcher. For beginners, this book contains multiple illustrated step-by-step guides for how to get started with online research on Mechanical Turk. This basic material is presented in the first four chapters. The first chapter provides a historical overview of online platforms, focusing on Mechanical Turk and market research panels. Chapter 2 provides an overview of concepts that are unique to Mechanical Turk and describes the Mechanical Turk culture and ecosystem. Chapter 3 provides an introductory discussion of stimulus development platforms, shows how to set up a study on Mechanical Turk, and discusses best methodological practices for study setup. Chapter 4 provides a conceptual introduction to the Mechanical Turk application programming interface (API), discusses several third-party apps that make Mechanical Turk research more effective, and describes several features of the CloudResearch (formerly TurkPrime) app. These chapters also contain much advice about best practices that even experienced researchers will find helpful.
Later chapters address many topics that are critical for getting the most from online research. Chapter 5 deals with issues of data quality on online platforms. Chapter 6 describes the demographic composition of Mechanical Turk, including the size of the MTurk population. Chapter 7 discusses issues of sampling, focusing on standard practices, the bias that such practices engender, and ways to avoid such bias. Chapter 8 discusses the representativeness of data collected on Mechanical Turk. Chapter 9 describes best practices for conducting longitudinal research, and Chapter 10 provides an overview of market research platforms and discusses the advantages and disadvantages of using Mechanical Turk over other platforms. Finally, Chapter 11 discusses the ethics of conducting research on Mechanical Turk and other online platforms.
Many topics in this book—including demographics, sampling, and ethics—are informed by the CloudResearch database, which consists of hundreds of thousands of participants and tens of millions of completed assignments. This database provides many insights about Mechanical Turk participants, who they are, how they have been changing over time, and how they work.
Finally, this book is associated with a companion website, where many of the how-to guides will be continually updated to keep up with the changes inevitable on all online platforms. You can access supplementary resources for Amazon Mechanical Turk, CloudResearch, and Qualtrics at https://www.cloudresearch.com/resources/book.
