9 Strategies to Enhance Quality of Data in Online Research

By Aaron Moss & Leib Litman, PhD

The CloudResearch Guide to Data Quality, Part 4:
9 Ways to Enhance Data Quality of Online Research Surveys

Collecting quality data begins with selecting the right participants. A phrase common in many research circles is that participant selection should be “fit for purpose.” This means the participants a researcher selects for a study should be well suited for finding answers to the questions that motivated the study.

However, data quality does not end with participant selection. As the study progresses, researchers also need to be aware of different threats to data quality and should know how to manage these threats through data analysis. Here, we examine some threats to data quality and practical ways to minimize their impact on your data.

---

The Basics of Data Quality Assurance

Ensuring data quality is a continuous process that evolves as a research project progresses. After researchers select the right participants and the right measures for their study, there are several ways to ensure participants are attentive, honest and engaged with the study during data collection.

1. Recruiting the Right Participants

Selecting the right participants for a research project is a complicated decision that is often influenced by the research question and convenience. When researchers are selecting among different sources of online participants, it is worth considering how different platforms may help or hinder the pursuit of quality data.

Some platforms, like Mechanical Turk (MTurk), allow researchers to get greater participant engagement than other online panels. This means researchers should consider whether the length of their study, the nature of the task, and the level of commitment they are seeking from participants are well suited to the recruitment platform they decide to use.

2. Ensuring Participant Attention

How can researchers ensure people remain attentive during online studies? One way is to design interesting studies. However, even if the content of a study isn’t naturally engaging, researchers can ensure the study is well designed, instructions are clear, and that respondents understand the importance of paying attention and giving honest answers.

Beyond study design options, researchers can include multiple brief attention check questions. During data analysis, participants’ success on the attention check questions should be used to help evaluate the quality of data and to decide whether to remove participants from the dataset.

3. Verifying Participant Demographics

Participants in online panels sometimes falsify demographic information, especially when doing so allows them to qualify for studies with high rewards. To discourage this practice, researchers can selectively target participants who have been previously profiled, either by the panel provider or, if using MTurk, by the researcher.

Dissociating the demographic screener from the study removes the benefit to the participant in providing false information and increases the likelihood that researchers will receive accurate demographic information. Researchers should always verify demographic information by also asking participants to report important demographics within the study.

4. Screening or Discouraging Dishonest Survey-Takers

People participating in online studies are human; and, given the impersonal nature of online data collection, some of them may lie or try to deceive researchers.

Potential forms of deception in online studies include participants using the Internet to look up the answers to knowledge-based questions as well as participants who are not truthful about their demographic information in order to qualify for studies recruiting specific populations.

5. Avoiding Non-Naive Participants

Some online platforms, like MTurk, allow researchers to selectively sample participants with specific levels of experience on the platform. However, even when a research platform does not make this possible, researchers may ask participants if they have been exposed to similar manipulations and measures as those used within the study, even if the wording is not the same.

6. Ensuring That Participants Fully Understand the Survey’s Language

Some online platforms allow people from all over the world to participate in studies. Although researchers can sometimes selectively recruit participants from specific countries, ensuring language comprehension is another effective way to improve data quality.

On platforms like MTurk, researchers can create a qualification for people who pass a language comprehension test. In other online platforms, researchers may set up a screener to determine people’s language comprehension before they enter a survey.

---

Simple Steps to Improving Data Quality

1. Understand the Differences Between Online Platforms

At CloudResearch, we regularly conduct research to understand the dynamics of different online recruitment platforms. Our research is published in leading academic journals and can be used to help researchers make informed decisions about the platform best suited to the needs of their research project.

2. Make Informed Decisions When Evaluating Participant Attention

The validity of attention check questions is a rapidly evolving area of research and one where some assumptions are being overturned by evidence. At CloudResearch, we help researchers make sense of this changing information. We regularly evaluate different methods of assessing participant attention and look for ways to combine traditional assessments of attention with technology that measures what participants are doing while taking studies.

3. Sample Participants Who Have Been Previously Profiled

Verifying online respondents’ demographic information is difficult because researchers do not interact with participants. Nevertheless, researchers can use certain methods to establish confidence in people’s demographic information.

For example, at CloudResearch, we profile workers on MTurk by randomly asking demographic questions over time and examining the consistency of each person’s responses. People who consistently provide the same demographic information — when there is no incentive to lie — are likely telling the truth.

On other online platforms, participants are profiled when they join the platform. Then, over time, people are regularly asked follow-up questions to verify their demographic information.

---