Including and Excluding Participants From Studies Run Through CloudResearch: What, When and How
By Richa Gautam & Aaron Moss, PhD
In this blog, we explain everything you could ever want to know about including and excluding participants from studies while using CloudResearch. In last week’s blog on longitudinal studies, we described our Include Workers feature, but this blog digs into the nitty-gritty and explains what our features are, when you might want to use them, and how they work.
What Are Include And Exclude Features?
Simply put, Include and Exclude features are tools you can use to determine which participants are eligible or ineligible to take part in your study. CloudResearch has multiple options for including and excluding participants from your studies. Some options are part of our Core Features (i.e., free) and some are part of our Pro Features (i.e., small cost associated). When using our Include and Exclude features, you can specify worker eligibility: a) at the worker level, using MTurk Worker IDs or CloudResearch anonymized IDs, or b) at the survey level, using the Survey ID of a study participants took in the past.
When Would I Need to Include or Exclude Participants?
There are several instances in which you might want to include or exclude certain participants from your study. Perhaps the most common circumstance for including workers is when you are running a longitudinal study and want to ensure only people who participated in Time 1 are able to participate in Time 2. In this instance, you could use an “Include List” to ensure only those participants who completed your first study are eligible to take your second study.
A common circumstance in which you might want to exclude workers is when you’re running multiple studies in a line of research on the same topic. Once participants have been exposed to the manipulations or measures in one of your studies, you might want to make sure they do not also participate in similar studies.
There are, obviously, several other reasons you may want to include or exclude participants from your studies. Regardless of your purpose, CloudResearch has several options for actually specifying which workers to include or exclude.
How can I Include participants?
MTurk
When using the MTurk Toolkit, you can Include workers by specifying specific studies workers must have previously completed, entering a list of Worker IDs, or creating a Worker Group. The Include feature is located in Tab 6, Worker Requirements, in the study setup.
To Include workers from a previously completed study, simply select which studies you would like to include and choose whether workers must have completed All or Any one of the studies selected.
The second way to Include workers is to paste MTurk Worker ID’s (or CloudResearch ID’s) into the Include workers box in the study setup.
Finally, another way to Include workers is to create a Worker Group. Worker groups allow you to paste several Worker ID’s into one manageable group that can be reused to Include workers across studies. This option is especially useful if you’re attempting to sample a specific demographic group (e.g., women under 40) after running a demographic screener to determine workers’ eligibility.
Using any feature to include workers means, by definition, all workers not on your Include list are excluded from your study. Thus, there is no need to also set an exclude list, place worker qualifications on your study (e.g., 95% approval rating, 100 HITs completed), or use location-based security features (e.g., Block Duplicate IP addresses). The only people eligible for your study are the people on your Include list.
Prime Panels
The Prime Panels self-service system does not currently have the option to Include participants from previous studies. If you’re interested in running a longitudinal study on Prime Panels, contact our customer support team (support@cloudresearch.com). One of our Project Managers can provide a quote and assist with running the study.
How can I Exclude participants?
MTurk
Similar to the options for Including workers, you can Exclude workers by specifying specific studies workers must have previously completed, entering a list of Worker IDs, or creating a Worker Group. Also, when excluding workers, you can use our Super Exclude feature, which excludes any worker who may have started one of your past studies but not finished it. This option ensures workers are completely unfamiliar with your study. Alternatively, the final option for excluding workers is to use a Survey Group.
To Exclude workers who completed a previous study, simply select which study or studies you want to exclude while working through the study setup options.
To manually exclude a list of workers, paste MTurk Worker ID’s (or CloudResearch ID’s) into the Exclude workers box in the study setup.
To exclude a group of workers, you can create a Worker Group or a Survey Group. The Worker Groups function is described above in the Include workers section. Survey Groups work by bunching a group of studies together and excluding workers who have accepted a HIT from any one of the studies in your Survey Group. Survey Groups is a more dynamic tool for excluding workers because it can Exclude workers from multiple studies running simultaneously. For more details, see here.
Finally, if you have identified several workers whom you would like to always exclude from your studies—perhaps because these workers consistently provide bad data—you can add their Worker IDs to the Universal Exclude List. You can find the Universal Exclude List in the Manage Workers dropdown menu. By pasting Worker ID’s into the Exclude box on the subsequent page, you can ensure the workers listed are blocked from all your studies.
Prime Panels
You can Exclude participants on Prime Panels by specifying specific studies workers have previously completed. To Exclude workers who completed a previous study, simply select which study or studies you want to exclude while working through the study setup options. Another way to exclude participants on Prime Panels is by adding your study to a Survey Group in the Study Settings section. However, due to limitations of the Prime Panels platform, you must add your study to the Survey Group before launching it. If you’re interested in adding a previously completed study to a Survey Group, please contact our customer service.
The Nitty-Gritty of Include and Exclude features
All of our Include and Exclude features on MTurk work with the use of MTurk’s Worker Qualifications. By granting Worker’s qualifications for some studies and not others, we can control who is eligible to take various HITs based on the criteria you specify. Although this system works well, it does have some limitations. For example, the options for Including Workers only work when Workers have completed AND been approved for your previous studies. Another limitation is that only 2,000 Worker IDs can be used at one time when Including or Excluding Workers by pasting Worker IDs into the setup box on Tab 6. To Include or Exclude groups larger than 2,000, you can create a Worker Group.
Although Prime Panels is a different online environment, our Exclude features on Prime Panels work similarly to those on MTurk. We use internal Worker IDs to exclude people who have completed your previous studies.
Three things to keep in mind:
- Exclusion criteria take precedence over inclusion criteria. Therefore, if a worker on your Include list is also on any Exclude list, they will not be eligible for the study.
- When Including workers at the survey level, you must specify whether you want workers who completed ALL of the studies selected or ANY of them. Selecting ALL will include workers who were part of every single study on the list, while selecting ANY will include workers who completed any one of the studies. Exclude logic is always ANY. If you ran three studies and want to exclude workers who completed all of them instead of all workers who completed any of them, you will have to specify their worker IDs.
- If you add several studies to a Survey Group, you cannot Include workers who were part of a study listed in the Survey Group. While Survey Groups may seem like a good way to organize your research, this feature is not meant as an organizational tool. Survey Groups work as exclusion criteria and since exclusion criteria take precedence over Inclusion criteria, Workers listed in your Survey Group will be blocked from your study.
Include and Exclude Features on CloudResearch
| Feature | Where to Find It | What It Does | Notes | Remember To... |
|---|---|---|---|---|
| Include Workers by previously completed Survey ID. | 1. Tab 6 in the MTurk Toolkit, Work Requirements. | Includes only workers who have completed and been approved for selected surveys. | 1. All workers who did not complete selected studies are automatically excluded.
2. Workers who have completed but not been approved for a previous study are not included. Not good for including workers from a live study | Select Any or All based on your purpose. |
| Include Workers by Worker ID | 1. Tab 6 in the MTurk Toolkit, Worker Requirements. | Includes only workers who IDs (Amazon or CloudResearch) you add. | 1. Limited to 2,000 Worker IDs | Include Amazon or CloudResearch IDs on separate lines. |
| Exclude Workers by previously completed Survey ID. | 1. Tab 6 in the MTurk Toolkit, Worker Requirements. 2. Prime Panels Study Settings Page | Excludes only workers who have completed and been approved for selected surveys. | 1. Automatically excludes all workers in selected studies and exclusion takes precedence over inclusion.
2. Workers who have completed but not been approved for a previous study are not excluded. Not good for excluding workers from a live study. 3. Only has Any logic. | |
| Super Exclude | 1. Tab 6 In the MTurk Toolkit, Worker Requirements. | Excludes all workers who started a study, even if they did not complete it. Helps ensure workers are entirely unfamiliar with your study. | Check the Super Exclude box. | |
| Exclude Workers by Worker ID. | 1. Tab 6 in the MTurk Toolkit, Worker Requirements. | Excludes only workers whose IDs (Amazon or CloudResearch) you add. | 1. Limited to 2,000 Worker IDs. | Include Amazon or CloudResearch IDs on separate lines. |
| Worker Group | 1. Tab 6 in the MTurk Toolkit, Worker Requirements.
2. In the drop-down menu under Manage Workers. | Includes or Excludes workers by ID (Amazon or CloudResearch ID). | 1. No limit on the number of IDs that can be added.
2. Must specify whether the group should be included or excluded. | Select the group while setting up your study. |
| Survey Group | 1.Tab 8 in the MTurk Toolkit, Pro Features.
2. In the drop-down menu under Manage Workers. 3. Prime Panels Study Settings page. | Excludes workers who belong to the group as soon as they complete and are approved for a study. Works well for live studies. | 1. Works for simultaneously running live studies and completed studies. | Specify the survey group when setting up your study. |
| Universal Exclude | 1. Tab 6 in the MTurk Toolkit, Worker Requirements. | Excludes workers whose IDs (Amazon or CloudResearch) are on the list from all your studies. | 1. Best used for excluding workers who consistently give poor responses. | Check the box to activate the list when setting up your study. |
As researchers run more studies on Mechanical Turk, the ability to select which workers are eligible or ineligible for studies becomes more important. In this blog, we’ve described CloudResearch’s tools for helping set inclusion and exclusion criteria. In the future, we will continue to develop tools that make research tasks like setting eligibility criteria easier for researchers.
