What Is It Like to Participate in Online Research on Amazon Mechanical Turk?

By Leib Litman, PhD & Aaron Moss, PhD

MTurk Surveys Compared to Everyday Life

In the world of human subjects research, Institutional Review Boards (IRB) often conduct a cost-benefit analysis to assess whether a study is ethical. A universal standard applied in these assessments is asking how much risk participants will be exposed to when compared to the things people encounter in everyday life.

To evaluate what it is like for people to take studies on Mechanical Turk, we adopted the same standard. That is, we asked people, “Do you find most MTurk surveys more or less distressing than typical things you encounter in everyday life?” People provided their answers on a scale that ranged from 1 (Much less distressing) to 5 (Much more distressing). This data and a more complete discussion about the ethics of MTurk can be found in the newest book in SAGE’s Innovation in Methods Series Conducting Online Research on Amazon Mechanical Turk and Beyond.  

Approximately 10,000 people answered our question. As shown in Figure 1, just over 93% indicated that participating in studies on MTurk is equal to or less distressing than everyday life. To ensure this result was not an artifact of the question wording, we asked the question in two ways. In one version, the question referred to the overall experience of participating in research studies on MTurk; in the second version, the question referred specifically to the content of studies. In both cases, more than 93% of respondents indicated that MTurk studies were less distressing than typical everyday experiences.

What Are Common Sources of Stress for Gig Economy Workers on MTurk?

Even though a large majority of people reported experiencing little stress from MTurk, about 6% reported that MTurk was stressful. We followed up with people from this group a few weeks after they answered our polling question to understand what about their experience on Mechanical Turk causes stress.

The first question we asked people was whether they still found MTurk surveys more distressing than the things they encounter in everyday life. Interestingly, only sixty percent of people reaffirmed this position. The other 40% indicated that they no longer found MTurk more distressing than everyday life. Combined with the original poll, this result suggests that close to 4% of people reliably find MTurk distressing.

Next, we asked people what, specifically, they find distressing about MTurk surveys. The answer options were: a) The content, b) Other things I’m asked to do, c) Dealing with requesters, d) Finding HITs, and e) Something else. We allowed people to select multiple answers and for each answer we asked them to provide a brief open-ended explanation.

The most common answer was that HITs are hard to find. In fact, difficulty finding HITs was a more common complaint than low pay. Whereas 70% of people (from the 6% who found MTurk distressing) reported finding HITs distressing, only 25% reported low pay as distressing. 

Finally, we asked people whether they thought the benefits of MTurk outweigh the costs. Just because some people say some aspects of Mechanical Turk are upsetting does not mean that running studies on MTurk is unethical. After all, IRB’s sometimes approve studies that expose participants to a fair amount of risk because the benefits of doing so outweigh the costs. 

In response to our question, 75% of the 6% of people who found MTurk distressing reported that the benefits of Mechanical Turk outweigh the costs. That is, among the 4% of people who reliably reported that MTurk is more distressing than everyday life, a strong majority said the benefits still outweighed the costs. This finding may provide important context within which to interpret people’s reported distress. 

Conclusion

Technology and platforms like Mechanical Turk have changed the way academic researchers conduct research. IRB’s care about minimizing risk to participants and one form of risk is the stress or discomfort participants might experience while participating in research. The results of our polling and brief follow up survey suggest that few people find the experience of Mechanical Turk reliably distressing, and for the overwhelming majority of workers the cost/benefit analysis is positive.

To read more about the ethics of MTurk, see our paper that reports the results of a representative survey of people on MTurk.