Running Longitudinal Studies on CloudResearch
By Sam Krumholtz & Aaron Moss, PhD
In this blog, we describe how to run a longitudinal study on MTurk, using CloudResearch. We also provide tips for maximizing worker participation and minimizing attrition.
Longitudinal research is important to the scientific community because it reveals how behavioral phenomena change over time. Nevertheless, researchers in some areas rarely run longitudinal studies because doing so requires a lot of resources. In a traditional research lab, longitudinal studies entail contacting an initial group of participants, maintaining contact information, begging participants to return for follow up, and managing data across time points. Even when the hassle of bringing participants to the lab is removed, online longitudinal studies can be challenging because simply re-contacting participants over email can take a great amount of time. Fortunately, CloudResearch has tools that make running longitudinal studies on MTurk less daunting and more efficient.
Setting Up a Longitudinal Study with CloudResearch’s MTurk Toolkit
Setting Up Your Initial Study
To set up a longitudinal study, select Create a Study from the header on the CloudResearch homepage and choose the MTurk Toolkit.
Next, select whether to run the study on your own MTurk account or as a fully managed MTurk study (if you do not have your own MTurk account).
On the next page, you can set up your study as you normally would.
When writing the Description and Custom Instructions for your HIT, explicitly state that the study is longitudinal or contains multiple waves of data collection. Doing so is one way to minimize attrition. Also, in the description area, describe the payment plan for the entire study (e.g., “You will be paid $1 for the initial study and $5 if you complete the second part of this study.”) Higher payment (at least $8 per hour) and a more transparent process (e.g., “You will be re-contacted in three weeks”), will result in lower attrition.
While setting up your initial study, it is imperative that you collect participants’ MTurk Worker IDs. Without Worker IDs, there will be no way to match participants’ data across time points. (Note: If you are worried about participant anonymity, CloudResearch can encrypt each Worker’s ID, allowing you to use the encrypted ID just as you would an MTurk ID, while giving workers an extra layer of privacy).
To collect Worker IDs you can: a) create a question within your survey that asks workers to enter their ID or b) use embedded data to automatically capture each Worker’s ID as they enter the survey. The embedded data option is preferable because it saves time and limits any human error when entering the ID.
Setting Up a Follow-Up Study
When running a follow-up study or successive waves of data collection, the Include, and Email Workers features are critical.
Include Feature
CloudResearch’s Include feature is fundamental to running longitudinal studies on MTurk. The Include feature allows only workers you place on the list to take your survey.
The Include feature is located in Tab 6, “Worker Requirements” in the study setup. There are two ways to use the Include Feature:
- At the study level by selecting previous surveys to include.
This method will only include workers who have completed All or Any of the surveys specified.
When All is selected, workers who have completed every study specified will be included. Think of this as an AND feature; workers who have completed survey 1 AND survey 2 AND survey 3 will be included in the current survey.When Any is selected, workers who have participated in any one of the surveys specified will be included. Think of this as an OR feature; workers who have completed survey 1 OR survey 2 OR survey 3 will be included in the current survey.
- At the worker level by manually entering MTurk Worker IDs.To manually include workers, first, you will need to retrieve Worker IDs from a previous study. On your study Dashboard, click on the study you want to obtain Worker IDs from and select “Download CSV” on the right side of the screen.
A .csv file will download from your web browser. Open the file and copy the MTurk IDs from the first column, AmazonIdentifier.
Next, paste the MTurk IDs into the manual include textbox.
**Please note that only workers who have completed one of your surveys in the past AND have been approved may be included.
Letting Workers Know the Follow-Up Study is Available
Once your follow up is ready to go, you can use CloudResearch’s Email tool to quickly and easily let all eligible workers know that the follow up is available.
To use the Email tool, go to your follow-up study on the dashboard and select the orange button Email Included Workers. The Email tool only appears on studies that use an Include list.
Once you select Email Included Workers, you will be taken to a page where you can draft a message to send to all workers. The default Subject includes the compensation of the HIT and the default message includes the compensation for the HIT, how long on average it has taken participants to finish, and a link to the HIT on MTurk.
Note: When customizing your message, do not delete the link to the HIT. Without this link, workers will be unable to navigate to your study.
Tips for Minimizing Attrition
To minimize attrition across time points, we recommend:
- Using the Study Description and Custom Instructions in your initial study to let workers know the study is longitudinal and to give them a sense of how the study will work.
- Using the Email Included Workers tool to let workers know when successive waves of the study go live.
- Paying more across each successive wave of data collection OR offering workers a bonus for completing all waves of data collection.
For complex longitudinal designs or any other questions on longitudinal methods not answered here, please contact our customer support team at support@cloudresearch.com.
