The Universal Exclude List: A Tool for Blocking Bad Workers

By Aaron Moss, PhD & Leib Litman, PhD

By now, even casual users of MTurk have heard about recent concerns of “bots” or low quality data. We’ve written about the topic here and laid out evidence that suggests “bots” are actually foreign workers using tools to obscure their true location (here). Perhaps most importantly, we’ve created two tools to help keep these workers out of your studies. In this blog, we introduce a third tool: the Universal Exclude List.

Problem

Suppose you run a study on MTurk and some workers submit surveys with subpar data (i.e., low quality) or data that seems intentionally fraudulent (think the recent bot scare). Your IRB may not allow you to reject these workers, but you also want to make sure they don’t participate in any of your future studies. How can you easily ensure these workers never take another one of your studies?

Solution

The answer is our new and free Universal Exclude List. The Universal Exclude List allows you to create your own list of workers who will be automatically disqualified from your future studies. You can easily add or remove workers from The Universal Exclude List at any time.

To create your Universal Exclude List click on the Manage Workers tab from the main dashboard and then click Universal Exclude List.

The next page will display any workers currently in your list as well as a box where you can add workers to the list (you can add workers using either their MTurk Worker ID or the anonymized CloudResearch Worker ID). When adding a worker to the list, you have the option to give a rationale (see figure below).

When launching a study, all workers in your Universal Exclude List will be blocked by default. If you ever need to disable the Universal Exclude List, you can manually override it by unchecking the box as show in the figure below (this box is on Tab 6 “Worker Requirements” in the study setup).

Under the Hood

When using the Universal Exclude List, workers are excluded from your study using the MTurk Qualification system. Workers you want to exclude will be granted an exclude qualification that makes them ineligible for the study. If you have multiple exclude lists associated with your study (e.g., the Universal Exclude List AND an exclude list for workers from a previous study in the same line of research), your current study will contain a single exclude list with all workers you wish to exclude. Please note that exclude lists take precedence over include lists; if a worker is listed on an exclusion list and an inclusion list they will not be able to accept your study.

Why You Should Exclude Rather Than Block Workers

MTurk allows requesters to block specific workers. Blocking workers prevents them from doing work for the requester in the future. Critically, when workers are blocked by multiple requesters their account may be permanently disabled. This means getting blocked could prevent a worker from being able to use MTurk to make money. Doing this raises ethical concerns that are circumvented by the Universal Exclude List. Rather than blocking workers from being able to work on MTurk, our Universal Exclude List uses the MTurk qualification system to prevent specific workers from participating in your studies without affecting their ability to work for other requesters.