Mechanical Turk Demographic Trends During the Year of COVID-19

Last month, people in the U.S. marked the one year anniversary of the COVID-19 pandemic. For most of 2020, circumstances on the ground were changing faster than many forms of data could catch up with, and there was fear this may be true with the demographic composition of people on Amazon Mechanical Turk. So, after one year, we decided to look at whether there were any major changes to the demographics of people on MTurk as a result of the pandemic.

In a recent blog, we summarized data that CloudResearch published in a letter in the June issue of Trends in Cognitive Sciences. In that letter, we reported that the demographics of people on MTurk had not shifted much during the first three months of the pandemic. Here, we take a broader view and report on the demographic trends we examined across all of 2020.

129,428 Workers Completed Studies through CloudResearch

In 2020, just under 130,000 people completed a CloudResearch-hosted study on MTurk. This number represents nearly a 7% increase from 2019 and almost a 20% increase from 2018. For the last five years, we have seen consistent growth in the number of people completing CloudResearch-hosted studies.

Despite significant upheaval to people’s lives during 2020, there was a similar pattern to the number of people completing studies each month. As the red line in the figure below shows, at no point in 2020 was there a sudden change in the number of participants completing studies.

Overall Participant Demographics Didn’t Change Much

CloudResearch regularly gathers demographic data from workers on MTurk. Over the course of 2020, we did not see large deviations from 2019 data in terms of participant gender, race, ethnicity (Hispanic or not-Hispanic), age, or household income. 

The stability in household income is noteworthy because some people suspected that a portion of the tens of millions of people who lost jobs during the year may turn to platforms like MTurk for money. The data we have do not suggest this happened.  

Participant Join-Rates Showed Seasonality

The rate of new people joining MTurk throughout 2020 largely mirrored the month-by-month progression of 2019. In the second half of the year, and especially in the fourth quarter of the year, new workers joined MTurk at a slower pace than they did in 2019. Overall, however, the rate of new people joining shows a seasonal effect that was largely similar in both years.  

Researchers Ran More Online Studies

Finally, at the risk of sounding obvious, researchers ran more online studies in 2020 than 2019. A month-by-month comparison shows a sharp rise in April 2020—the month after all in person data collection stopped—before a leveling off in the summer and fall months.

Although COVID-19 dramatically changed how researchers gather data, it also led to some remarkable opportunities like this study that used CloudResearch to study how lockdowns affected people’s level of paranoid thinking.

Learn how CloudResearch can make it easy to run studies on Mechanical Turk.