How Has COVID-19 Affected MTurk Worker Demographics?

By Aaron Moss, PhD, Cheskie Rosenzweig, MS & Leib Litman, PhD

The COVID-19 pandemic has changed nearly everything. An important change for social scientists is that most, if not all, human subjects research has moved online. Given the massive disruptions to society, researchers have asked: who is available and willing to take online studies these days?

As it turns out, our data from June of last year suggest about the same people who were available before the pandemic. In a letter published in Trends in Cognitive Sciences, the research team at CloudResearch reported little change in the overall demographics of people completing studies on Amazon Mechanical Turk (MTurk). Here, we outline some of the reasons why researchers might have expected the demographics of people taking studies to change and report data from last June that show the overall pool of people was largely unchanged.

Why COVID-19 May Change the MTurk Population

It isn’t hard to conjure theories about how the pandemic may change who is able to complete studies on platforms like MTurk. Schools and daycares across the country have largely been closed, meaning most parents have had less free time. In addition, tens of millions of people have lost jobs and remain out of work, raising questions about how people will continue to pay their bills. 

As people cut back on non-essential expenses, the diversity of users on online platforms may decrease as low income and minority households are likely to feel this pinch the most. Furthermore, if parents have less free time and other people are unable to access the internet from work or public spaces, researchers may be left with a narrower group from which to sample. 

But data show this is not the case—at least not as of last June. Using data from CloudResearch’s publically available Metrics tool, we found that the demographics of people completing studies on MTurk were largely consistent going back to January 2019. As Table 1 shows, there was relatively little variation in the race, income, and gender of people on MTurk from January 2019 through mid-May 2020, which includes the first few months of the pandemic.

Table 1. Selected participant demographics on Mechanical Turk across 18 months.

Given that there are many reasons to suspect demographic changes in online platforms during the pandemic, why do the numbers appear stable? It’s hard to say for sure, but one reason may be because MTurk is, itself, a way to earn extra money. People who may have lost their source of full-time or primary employment may increase their activity on MTurk to help make ends meet.  

Trends to Keep an Eye On

From a research perspective, social scientists may be concerned about the diversity and representativeness of the populations available for research during the COVID pandemic. Even though data from CloudResearch show stability in the demographics we examined, there are some trends among new people joining the platform that are worth keeping an eye on. Interestingly, these trends may be toward increased diversity, not decreased diversity.

In a working paper, Arechar and Rand (2020) compared information about new participants who participated in their studies since the pandemic began, to a group of “legacy” participants who were on MTurk before the pandemic. New participants showed a different demographic profile than legacy participants. New participants were more likely to identify as male, non-White, and Republican when compared to legacy participants. However, their new participants also showed more evidence of careless responding.

When we examine the overall CloudResearch data we see mixed support for these trends among new participants. In addition, we see that the number of new participants is small enough that these trends do not impact the overall demographics of participants completing studies. Nevertheless, monitoring the demographics of participants throughout the pandemic will be important. And, it is something you can do using the CloudResearch Metrics tool.

Update

As we approach the one year mark of the pandemic, CloudResearch is going to take another look at our data to see if there have been any changes in the composition of people taking studies on MTurk. Keep an eye out for this blog in the next few weeks!

Learn how CloudResearch can make it easy to run studies on Mechanical Turk.