Does It Matter What Time of Day a Survey Is Sent Out?
If you have worked in behavioral research during the last decade, you know the Internet has changed things. Market researchers, pollsters, and academics have all turned to the Internet as a fast and efficient way to find research participants and learn what they think.
Among the advantages of collecting data online is speed. The Internet allows researchers to administer surveys, polls, and experiments exponentially faster than traditional methods such as mail surveys, telephone interviews, or bringing participants in-person. Although the speed of online data collection has primarily been celebrated as a strength, a new study from CloudResearch suggests that excessive speed may sometimes lead to bias within a study.
Specifically, extremely fast data collection may open a study up to time of day bias. Time of day bias occurs when the people who are available to take an online study in the morning are substantially different than those available to take a study in the evening. Because many sources of online participants make it easy for participants to take studies whenever it is convenient for them, this is not a trivial issue.
Differences Between Morning People vs. Evening People
Decades of research in the biological and psychological sciences support the basic idea that some people are more oriented towards mornings, and others are more oriented toward evenings. Morning-oriented people go to bed early, wake up early, and achieve peak mental and physical performance earlier in the day. Evening-oriented people go to bed late, wake up late, and achieve peak performance in the evening or at night. In addition to these differences, morning and evening-oriented people differ in a variety of personality and psychological variables.
Broadly speaking, morning people are happier and more conscientious and agreeable than evening-oriented people. Conversely, evening-oriented people are more open, extraverted, anxious, and impulsive than morning people.
In addition to these personality differences, evening-oriented people often show more signs of clinical symptomatology than morning-oriented people. For example, research shows that evening people are more prone to anxiety, depression, and unhealthy behaviors like disordered eating, disordered sleeping, Internet addiction, impulsivity, and procrastination than morning people.
Whether these differences are driven by biology, psychology, or lifestyle, they may interact with some of the phenomena studied by behavioral researchers. And, when they do, the possibility for bias increases.
Do These Differences Exist in Online Research Platforms?
To understand how much differences between morning and evening-oriented people affect the data collected in online research platforms, CloudResearch conducted two studies on Amazon’s Mechanical Turk (MTurk).
In many respects, MTurk is an ideal place to look for time of day differences in online research participants. People on MTurk are free to complete studies whenever they want. In addition, because researchers all over the world use MTurk, studies are posted and remain live around the clock. For these reasons, we expected people who work on MTurk to drift toward working during the time of day that is aligned with their circadian typology.
In the first study, we sampled people on MTurk at either 6:00 a.m. or 8:30 p.m. We had people complete a measure of preference for mornings or evenings and examined whether people active in the morning or evening differed on a variety of psychological and behavioral characteristics known to correlate with a preference for either mornings or evenings. We expected people active in the morning to be more morning-oriented than those active in the evening. In addition, we expected these time of day differences to translate into several personality and behavioral differences between morning and evening people.
The results of Study 1 are in Figure 1. There were differences between morning- and evening-oriented people on all variables. The only variable in which the difference was not significant was on perfectionism.
To increase our confidence in these differences, we conducted a second study in which we controlled for the day of the week participants completed the study and their local time zone. We also added an afternoon condition so that participants were sampled at either 5:00 a.m., 1:30 p.m., or 12:30 a.m. The results of Study 2 are shown in Figure 2. They largely support the existence of time of day differences in online platforms, and highlight a few additional ways that people taking surveys at night differ from those who take them during the day.
How Might Time of Day Bias Affect Studies?
Time of day bias may affect studies when researchers study a phenomenon that changes throughout the day or is correlated with variables that morning and evening-oriented people differ on. For example, consider the morning morality effect. As originally reported by Kouchaki and Smith (2014), the morning morality effect explains a general tendency for people to act more ethically early in the day and less ethically later in the day as resources for self-control are taxed.
Although the original research reported a general effect of time of day, subsequent research suggested that accounting for people’s circadian typology in addition to time of day might do a better job of predicting immoral behavior than time of day alone (Gunia, Barnes, & Sah, 2014).
Therefore, studies that investigate phenomena that vary across time of day and fail to account for variation in peoples’ circadian typology may yield incomplete or inaccurate findings.
Finding the Best Time of Day to Send a Survey
Another way time of day differences might affect online studies is by presenting the opportunity to sample specific groups of participants at different times of the day. For example, older adults tend to be more morning-oriented than younger adults. As a result, a purposeful sampling strategy to recruit older adults might include a plan to launch studies earlier in the morning when older people are likely to be active.
Conversely, purposefully sampling late at night might help researchers recruit participants who suffer from insomnia, Internet addiction, or anxiety. In this way, time of day differences represent an opportunity for online research.
What’s the Solution?
Our studies suggest there are instances in which researchers may want to slow down data collection in order to control the time of day, day of the week, or the local time zone in which participants complete a study. Changing sampling practices around the time when a study is launched may help researchers avoid biasing their study by participant age, conscientiousness, anxiety, level of depression, and other characteristics known to vary with a preference for mornings and evenings. Researchers should consider collecting data across different times of the day to ensure that they are not biasing their study in specific ways.
At CloudResearch, we have a variety of tools that make controlling the timing of your data collection possible. When using our MTurk Toolkit, it is easy to divide your study into multiple “batches” that are launched throughout the day. You can also use our MicroBatching tool to set a delay between when successive batches of your study are launched, slowing down your data collection and mitigating time of day bias.
If you really want to ensure time of day bias does not affect your study, our tools make it possible to spread data collection across all the days of the week and to target workers in specific time zones as we did in Study 2.
To learn more about these features and start planning your next study, check out our MTurk Toolkit.
