How Many Amazon Mechanical Turk Workers Are There in 2019?

By Aaron Moss, PhD & Leib Litman, PhD

In a recent research article, we reported that there are 250,810 MTurk workers worldwide who have completed at least one Human Intelligence Task (HIT) posted through the TurkPrime platform. More than 226,500 of these workers are based in the US.

Across three years, from 2016 to 2018, we saw approximately eighty to eighty-five thousand US workers active on MTurk each year (Figure 1). During the same period of time, we saw that more than half of US workers were new to the platform each year (Figure 1).

These numbers are based on observed participation rates from the TurkPrime database. Using this database allows us to give a concrete number on the lower bound of the MTurk population. In other words, because we cannot be sure that all MTurk workers take HITs posted through TurkPrime, our numbers indicate there are at least as many workers as we report.

Research context

Many academic researchers use Amazon’s Mechanical Turk as a source of quick, affordable, high-quality research data. Yet MTurk’s popularity among academics has led to several questions about whether the participant pool is overused and whether MTurk workers are too familiar with measures and manipulations common in the social sciences. One key to answering this question is finding out how many workers are active on MTurk.

Previous studies have estimated, on the low end, that the average lab has access to less than 10,000 workers in any three-month span (Stewart et al., 2015). Other estimates place the total number of workers at around 100,000 (Difallah, Filatova, & Ipeirotis, 2018). While these estimates are informative, the methods used to obtain them result in a wide margin of error.

The numbers from our database reveal that MTurk has more workers than previously estimated and that there is a significant number of new workers joining the pool on a regular basis. Over each of the last three years, more than half of US workers taking HITs through TurkPrime have been new to the platform. We believe these numbers offer a reason for researchers to be optimistic about the possibility of sampling participants from Mechanical Turk.

In addition to discussing the size of the worker pool, our article addresses other issues relevant for sampling on Mechanical Turk. The full preprint of the article is available here.

References

Difallah, D., Filatova, E., & Ipeirotis, P. (2018, February). Demographics and dynamics of Mechanical Turk workers. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (pp. 135-143).

Stewart, N., Ungemach, C., Harris, A. J., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10, 479-491.