Five Years of Mechanical Turk Data in Five Figures
Back in 2015, CloudResearch—at that point called TurkPrime—began as a website researchers could use to make setting up and managing MTurk studies easier. Since then, CloudResearch has facilitated tens of thousands of studies with hundreds of thousands of participants and tens of millions of individual assignments. Here, we present five graphs that tell a story about MTurk over the last five years.
Figure 1: The Number of MTurk Workers per Year
When a human intelligence task (HIT) is launched on MTurk through CloudResearch, we capture meta-data about that HIT. Using this metadata we see that the number of U.S. MTurk workers has increased each year since 2016. From 2016 to 2020, the total number of U.S. workers on MTurk grew by over 30%.
Figure 2: The Average Number of New Workers per Month
Fueling the growth of MTurk is the number of new people that join each month. Over the last five years an average of 5,906 new U.S. workers have joined per month.
Across years, the average number of new workers per month ranged from a low of 4,926 in 2017 to a high of 6,923 in 2019.
Figure 3: The Number of Active People per Month 2016–2020
While it is always good to have new people joining the platform, a question that is crucial for researchers is: how many people are available at any one time to participate in a study?
Over the last five years, there have been an average of 27,824 active workers per month. During each of the last three years, that number has averaged 30,887. Both of these numbers are notably higher than previous estimates (e.g., Stewart et al., 2015). However, the key to reaching these participants may be the reputation qualifications researchers use when sampling.
Figure 4: Aggregate Demographics of MTurk Workers
A longtime lure of platforms like MTurk has been the potential for academic researchers to access a more diverse group of people than the traditional subject pool or, in most cases, even the local community can offer.
Across the last five years, there has been remarkable stability in the aggregate demographics of MTurk workers, including it appears during the COVID-19 pandemic. At the same time, however, there has been a slow movement toward becoming more representative of the general population.
Perhaps the most noteworthy changes are that: 1) the share of White workers has decreased, 2) the share of workers 18-24 has increased, and 3) the share of workers who identify as Republican has slightly increased. With the exception of age, these changes move the platform demographics closer to each groups’ share of the U.S. population. Of course, however, there are several other characteristics that make MTurk workers unlike the general US population like educational attainment.
For a full look at MTurk demographics, see Chapter 6 in Conducting Online Research on Amazon Mechanical Turk and Beyond.
Figure 5: Average Payment per HIT 2016–2020
Finally, the average payment offered to workers has increased in recent years, as reported elsewhere (here and here).
In 2016 and 2017, the average payment across all HITs on CloudResearch—from short demographic screeners for a few cents up to the most complicated and engaging longitudinal designs—was $1.05 and $1.02, respectively. In the last three years, this rate jumped to $1.50 in 2018, $1.47 in 2019, and $1.44 in 2020.
Although this data shows that researchers have been paying participants more, this data also lacks context. For instance, this data cannot say why compensation has gone up. What we do know is that this data is in line with other metrics that indicate compensation on MTurk has increased in the last several years.
Conclusion
The five figures presented here tell a story about Mechanical Turk using CloudResearch data. If you are interested in tracking changes in the demographics of people on MTurk, check out CloudResearch’s Metrics tool. Using the Metrics tool you can see real time changes in worker demographics and create crosstabs and figures to explore changes in demographics over time.
