Demographic Consistency Over Time on Mechanical Turk

Problem

Ever wonder if workers are being honest with you when they answer a survey? Or, if you specify that your study should be taken only by women, whether some workers take the study even though they are not women?

Solution

Now, all studies that have the Demographic Question feature enabled, will receive 2 additional columns in the CSV download for each Worker:

1. gender
   
2. gender consistency score
   

The gender column specifies the worker’s gender based on the worker’s responses to that question in previous surveys. Each worker who completes a study may have taken TurkPrime studies many times before. With each study where the Demographic Question is enabled, TurkPrime asks Workers to optionally share some demographic information like gender so they can qualify for HITs that require those demographics. (Through TurkPrime Panels Lab Service, researchers run many studies available only to workers who satisfy specific demographic requirements.) It is the responses to these questions that is shared with you, the TurkPrime users.

The gender consistency score specifies the consistency with which the gender question was answered as specified. For example, if the Gender Consistency is 66%, that means the MTurk Worker responded that they were female only 66% of the time. If the field is blank, that means that TurkPrime has not received a response from that Worker when asked to identify their gender.

This may indicate an innocuous error on the Worker’s part, multiple workers sharing an MTurk account, or negligence on the part of worker. In any event, that information is now shared with TurkPrime users.