What Is Prime Panels? Breaking Down CloudResearch’s Panel Management Tool
Moving to the Virtual Lab
The way researchers find subjects has changed dramatically over the last century. Techniques and standard practices have shifted from an original emphasis on self-study and introspection to exploring archival data, recruiting participants on the street, inviting people to the laboratory, soliciting individuals in shopping malls, and requiring students in introductory courses to participate in studies as part of their learning about a discipline. Most recently, researchers have turned to the Internet. Since the mid 2000’s, online data collection has led to a sampling revolution. The Internet has given researchers fast and easy access to a more diverse group of research subjects than ever before.
The Rise of Mechanical Turk
For many academic researchers, the technological revolution has primarily meant sampling from Amazon’s Mechanical Turk (MTurk). Originally created as a microtasking platform, MTurk has become a primary source of data for academic researchers in less than ten years. Hundreds of peer-reviewed papers are now published each year with data collected from MTurk.
One benefit of MTurk’s rapid adoption by academic researchers is that the research community has learned a lot about the platform in a short time. Numerous studies have investigated MTurk’s demographic composition, data quality, and factors related to MTurk’s functionality as well as the consequences of how researchers use the platform. Generally speaking, this research has shown that MTurk is a source of quality data, yet the size of the platform constrains researchers’ ability to sample specific demographic groups and leads to the existence of some participants who are extremely experienced with social science studies. Together, these limitations suggest that some studies may be better suited for a larger platform with more naive participants.
Advantages of Prime Panels
Prime Panels is a participant recruitment platform that was developed by CloudResearch, formerly TurkPrime, in 2017. Prime Panels has an easy-to-use interface that allows highly targeted recruitment from over 50 million participants.
In a recent publication, members of the CloudResearch team collaborated with outside experts in the field of online sampling to examine the value of Prime Panels as a research platform. Data collected were compared to MTurk samples, and to nationally representative samples when available. The research revealed that it is possible to collect high quality data on Prime Panels from diverse participants who are unfamiliar with many academic measures and manipulations. We review the highlights below.
Data Quality
The first question that has to be answered about all potential sources of research participants is: Do participants from this source provide quality data? In the research our team conducted, we compared data quality on Prime Panels to MTurk. Whereas MTurk employs a reputation mechanism to ensure data quality, on Prime Panels we developed and tested a quality control mechanism as a substitute for MTurk’s reputation system. The quality control mechanism includes a series of validated attention and English comprehension screener questions. We compared responses from those participants who passed and failed the screener on Prime Panels with participants from MTurk and we examined a host of data quality measures.
The results from our study revealed that participants who failed the screener on Prime Panels provided low quality data, however, participants who passed the screener provided quality data comparable to that from MTurk participants across multiple measures of data quality. For example, almost all participants who passed the screener on Prime Panels also passed subsequent attention checks and provided responses to a long personality questionnaire (the Big Five Inventory) that were highly consistent. In addition, as another testament to the quality of responses on Prime Panels, we replicated several classic experiments. Altogether, these results demonstrate the power of the quality control mechanism that we employed to improve data quality on Prime Panels. This quality control mechanism is used by default for all CloudResearch studies run on Prime Panels.
Demographic Representativeness
In addition to examining data quality, our study compared the demographics of Prime Panels and MTurk to the American National Election Survey (ANES), a nationally representative, probability-based sample of Americans. Prime Panels participants were more representative of the country across many variables—age, marital status, number of children, political affiliation, and religious devotion—than participants from MTurk. This suggests that some groups of participants like older adults or those with more conservative values may be easier to recruit in online panels than on MTurk. Here is a snippet of some of that data with striking differences highlighted:
Demographics by Age
| Age | Prime Panels | MTurk | ANES |
|---|---|---|---|
| 18-29 | 21.4% | 31.2% | 16.8% |
| 30-39 | 21.2% | 38.5% | 17.1% |
| 40-49 | 15.2% | 16.1% | 16.4% |
| 50-59 | 16.3% | 10.0% | 17.0% |
| 60-69 | 17.8% | 3.8% | 20.4% |
| 70+ | 8.1% | 0.5% | 12.2% |
Demographics by Religion
| Religion | Prime Panels | MTurk | ANES |
|---|---|---|---|
| Buddhist | 1.5% | 1.3% | 0.8% |
| Christian | 59.4% | 46.2% | 58.9% |
| Muslim | 0.6% | 0.2% | 0.7% |
| Jewish | 2.6% | 1.3% | 2.5% |
| Hindu | 0.4% | 0.7% | 0.2% |
| Agnostic | 8.8% | 18.8% | 6.6% |
| Atheist | 7.9% | 20.4% | 6.8% |
| Other | 12.0% | 7.0% | 11.5% |
The Pros and Cons of Prime Panels vs. MTurk Toolkit
At CloudResearch, we see Prime Panels and MTurk as complementary sources of research participants. On MTurk, participants are willing to engage in difficult tasks that last several hours, to return for multiple waves of longitudinal data collection at very high rates, and, generally speaking, they do so while providing quality data. On Prime Panels, participants appear less willing to participate in very long studies and longitudinal attrition is higher. However, given the size of the platform, researchers are able to sample very narrow segments of the population. On Prime Panels it is possible to target participants in ways not possible on MTurk, like sampling people within specific US zip codes, by a range of occupations, or by matching to the US census. While there is still much to learn about the strengths of the platforms the current finding indicates that both platforms can be usefully employed for a variety of online research projects.
Published Research Powered by Prime Panels
Hundreds of academic and market researchers have already run successful studies through Prime Panels. Examples include: researchers who have sampled people who had a hip or knee replacement or spinal surgery in the last 10 years, coffee drinkers in cities across the U.S. between ages of 18-44, and 400 gay men and 400 lesbian women in exclusive committed romantic relationships. Prime Panels has also been used to recruit participants for live in-person clinical sessions in hospitals. Prime Panels shows great promise as a platform that can be further used by researchers to collect high quality data with targeted precision at incredible scale.
Selected publications with samples recruited via Prime Panels:
- Income-stratified sample of 1111 U.S. residents (aged 18–88) with 200 participants in each of 5 income brackets
- 889 Individuals currently in therapy who have attended at least 3 sessions
- 375 current dog and cat owners
- 251 Black and 249 White U.S. residents between the ages 40-65
- 230 women who had an abortion (n = 230), and 126 women who considered abortion but decided against it
- 1,872 U.S. adult e-cigarette-only users, cigarette-only smokers, and dual users
- 4,581 U.S. participants with equal numbers of Democrats and Republicans
Citation
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019). Online panels in social science research: Expanding sampling methods beyond Mechanical Turk. Behavior Research Methods, 51(5), 2022-2038. https://link.springer.com/article/10.3758/s13428-019-01273-7
