Running Dyadic Studies on Mechanical Turk Using CloudResearch

Studying pairs of people (e.g., married couples, friends, coworkers, etc) is becoming increasingly common in the social and behavioral sciences. Online participant populations, such as Mechanical Turk and other online panels, can potentially serve as a rich source of dyadic participants. However, conducting dyadic research online also faces multiple challenges that need to be overcome in order to obtain high quality results. This blog post will outline some of the challenges of running dyadic studies online, as well as the ways our MTurk Toolkit can best be used to run a dyadic study. We give recommendations for best practices based on our experience. Using the methods outlined in this blog, researchers have been able to successfully run numerous dyadic studies using the MTurk Toolkit.

Challenges

Several challenges present themselves when running dyadic studies online. Typically, recruitment tools, like MTurk, only provide access to individual participants and there is no simple way to check the relationship status of any two people. Additionally, many recruitment platforms have strict data collection policies that prevent collecting personally identifiable information (PII). This prevents directly asking participants to present their names, drivers licenses, marriage and business certificates or any other document that could be used to demonstrate their relationship status. Furthermore, when asking participants to “recruit” their partner or colleague it is difficult to verify that they have done so truthfully. This can obviously compromise the integrity of the data.

Using CloudResearch’s MTurk Toolkit to Run Dyadic Studies

Using CloudResearch’s MTurk Toolkit to run dyadic studies can help mitigate the challenges described in the above section.

Targeting the Right People

Specifically, using the panel section on the Toolkit can help ensure that you are targeting participants who have previously self-reported that they are in a relationship. We ask workers non-leading demographic questions at random points in time (dissociated with any specific study, so that they are not inclined to be dishonest if they feel that one answer may get them qualified over another). We then follow up weeks or months later and ask them the same or similar demographic questions to get updated information, and see if participants’ responses remain consistent. Once response consistency is determined, we make the option available to target those workers using the panel section.

Panel Targeting

Using panel targeting, your study will only show up for those qualified workers who meet the given demographic criteria. Once you can be confident that you are targeting individuals who have previously self identified as being part of a relationship, you can request that they send the survey link to their partner. In order to be able to link the couple’s responses, you should include a question in your survey asking for the participant’s worker ID. Instruct the worker to give their ID to their partner to use as a code, so that you can link the data between the responses.

Please note, while this process does not 100% guarantee workers will be honest (i.e., get their partner to fill out the second half of the survey), it does make lying less likely by removing the temptation for workers to say they qualify, when they actually do not. In addition, researchers can include questions to try to verify that two different people are completing the survey, provided those questions do not ask for PII.

Step-by-Step Guide

To run a dyadic panel study on the MTurk Toolkit (step by step):

1. Select Create a Study from the navigation bar on the top of the homepage. Then select MTurk Toolkit.
2. If you have an MTurk requester account that is linked to CloudResearch, select the Run a study on your own MTurk account option. If you do not have your own MTurk requester account or have one but do not want to use it for this study, you can select Managed Research and we can help run the study for you.
3. On Tab 1 of the study setup, you will see a variety of different panel options to choose from. As an example, if you were targeting married couples, you can use the marital status criteria to select participants who are currently married.
   

   

4. After selecting the panel criteria, it is very important to be descriptive and clear regarding the requirements of the study. This includes informing participants of the dyadic nature of the design, and instructing them to give their partner their worker ID so you can link their responses. This can be outlined in the Describe HIT tab (or in advanced options in the fully managed MTurk study option) –> Custom Instructions. Here are example instructions from a previous dyadic study using our MTurk Toolkit to study romantic relationships:
   

   

5. You can then feel free to add any security options, distribution methods (i.e. MicroBatch or HyperBatch), or pro features to a study of this nature, just like setting up a standard MTurk Toolkit HIT.
6. Select Save on the final tab.

You can then edit or launch your study from the dashboard whenever you are ready.

Best Practices

Based on our experience of running dyadic studies using the MTurk toolkit, we have a few suggestions for best practices to help ensure data collection goes as smoothly as possible.

Participant Compensation

Participants should be offered higher than average compensation to participate in dyadic studies. We recommend setting a pay rate of at least $10 per hour because these tasks require more from the worker than the typical task on MTurk. At a pay rate lower than that, the participant may not be motivated enough to participate in the task, or to go through the bother of pulling their partner in for the study. We have also seen that participation rates increase when a bonus is offered upon successful completion of the study.

Additionally, you will need to account for the participant’s partner’s time as well, and work that into the total compensation. For example, if your survey is 30 minutes long for both the participant and their partner, you should be paying a minimum of $10 (2 × ($10/hr × .5 hours) total. Compensation for both partners should be included in the base pay rate because there is no external way of compensating the participant’s partner, and thus, all compensation needs to be done on MTurk.

Piloting Your Study

We strongly recommend launching a pilot study (for around 5%-10% of the total complete count you are looking for) before launching your full study. This helps ensure you are able to match up partners responses, the data look good, participants are not getting confused by the unique workflow, etc. If participants do give some feedback as to how to make the study less confusing, you can incorporate that feedback into the full launch. The pilot will also allow you to catch any significant issues that may be preventing participants from successfully completing the HIT before the full launch. Once you are ready to launch the full study, you can then exclude the pilot from the full launch to prevent pilot participants from taking the study twice (or create a survey group).

For dyadic studies that are more complex than the system described above can handle, we’d be happy to help manage the study using our Managed Research service. We can create a custom solution best suited for your project.

For any questions, please contact us here. Follow us on twitter or facebook for the most recent news, exciting new feature announcements, blog posts, and more.

Citation

Krumholtz, S., Moss, A. J., & Litman, L. (2018, Nov. 8). Running dyadic studies on Mechanical Turk using TurkPrime [blog post]. Retrieved from: https://www.cloudresearch.com/resources/blog/running-dyadic-studies-on-mechanical-turk-using-turkprime