How Do Most MTurk Workers Work?
By Aaron Moss, PhD & Leib Litman, PhD
A persistent cause of concern for researchers who conduct studies online is understanding what participants might be doing while completing their study. When participants are outside the lab, they cannot be observed and distracting aspects of the environment cannot be controlled by the research team. As a result, researchers are left to wonder: how much attention are participants giving my survey?
In this blog, we report on one small aspect of this issue by describing the work style adopted by workers on Amazon’s Mechanical Turk.
What Work Style Do Workers Favor?
To find out how MTurk workers like to work, we asked: “How would you describe your work style on MTurk?” Workers had four response options:
- I complete HITs one by one, but sometimes I take breaks while in the middle of a HIT
- I complete HITs one by one, each HIT in one sitting
- I complete multiple HITs at the same time in one sitting
- I complete multiple HITs at the same time while taking breaks
Of the 8,256 people who answered this question, 77.3% said they complete HITs one at a time, each in one sitting (see Figure 1). Roughly 12% of people said they complete multiple HITs at the same time in one sitting.
After learning that most workers complete HITs one at a time and in one sitting, we wondered whether variables like gender, age, ethnicity, or worker experience moderate this result. As shown in the tables and figures below, workers described their work style similarly, regardless of gender, age, or ethnicity.
Gender Results
Age Results
Race Results
Worker Experience Results
Although participants of various genders, ages, and ethnicities, all reported a preference for completing one HIT at a time and in one sitting, one notable difference emerged among workers with various levels of experience. As shown by the green bars in Figure 5, less experienced workers reported preferring to complete multiple HITs at the same time in one sitting at a higher rate than workers with more HITs completed (i.e., green bars decrease as the number of HITs completed increase). Workers with fewer than 100 HITs completed reported the highest preference for completing multiple HITs at one time in the same sitting.
Summary
Most researchers would like for workers on Amazon’s Mechanical Turk to devote their full attention to one study at a time and to not take breaks or look for other HITs until the study is complete. The good news from this data is that nearly 80% of workers say this is how they go about their work and these results do not vary much by gender, age, or ethnicity.
Interestingly, the one variable that did influence our results was worker experience. Workers with less experience reported completing multiple HITs at the same time more than workers with greater experience. What is not clear from this one question is whether less-experienced workers are actually jumping around multiple HITs at the same time or simply keeping an eye out to grab other HITs when they become available.
While the results of this single survey question are limited, they do provide at least some insight into how MTurk workers approach their work. Positively for requesters, most workers say they focus on one study at a time and complete HITs without taking a break. Knowing that workers are directing attention toward each study as they take it, is one step toward having greater confidence in data collected over the internet and outside of the lab.
References
Moss, A. J., & Litman, L. (2019, Mar. 01). How do most MTurk workers work? [blog post]. Retrieved from: https://www.cloudresearch.com/resources/blog/how-do-most-mturk-workers-work-2/
