Pros and Cons of Different Sampling Methods
Quick Navigation:
By Aaron Moss, PhD, Cheskie Rosenzweig, MS, & Leib Litman, PhD
Online Researcher’s Sampling Guide, Part 4:
Pros and Cons of Different Sampling Methods
Conversations about sampling methods and sampling bias often take place at 60,000 feet. That is, researchers like to talk about the theoretical implications of sampling bias and to point out the potential ways that bias can undermine a study’s conclusions. Although these conversations are important, it is good to occasionally talk about what sampling looks like on the ground. At a practical level, what methods do researchers use to sample people and what are the pros and cons of each?
Non-random Sampling Methods
Non-random sampling techniques lead researchers to gather what are commonly known as convenience samples. Convenience samples are often based on who it’s easy for the researchers to contact. However, most online research does not qualify as pure convenience sampling. Often, researchers use non-random convenience sampling methods but strive to control for potential sources of bias. Here are some different ways that researchers can sample:
Voluntary Sampling
Voluntary sampling occurs when researchers seek volunteers to participate in studies. Volunteers can be solicited in person, over the internet, via public postings, and a variety of other methods. A researcher using voluntary sampling typically makes little effort to control sample composition.
Pros and Cons:
- Feasibility: Finding volunteers is often a relatively fast and affordable way to collect data.
- Subject to bias: Voluntary sampling is highly susceptible to bias, because researchers make little effort to control sample composition. The people who volunteer for the study may be very different than those who do not volunteer.
An Example of Voluntary Sampling:
A common form of voluntary sampling is the customer satisfaction survey. After a business provides a service or good, they often ask customers to report on their satisfaction. Because the business is asking all customers to volunteer their thoughts, the sample is voluntary and susceptible to bias.
Who Uses Voluntary Sampling?
Within academia, researchers often seek volunteer samples by either asking students to participate in research or by looking for people in the community. Within industry, companies seek volunteer samples for a variety of research purposes. Because volunteer samples are inexpensive, researchers across industries use them for a variety of different types of research.
Snowball Sampling
Snowball sampling begins when researchers contact a few people who meet a study’s criteria. After those people complete the study, the researchers ask each person to recommend a few others who also meet the study criteria. By building on each participant’s social network, the hope is that data collection will snowball until the researchers reach enough people for their study.
Pros and Cons:
- Ability to reach small or stigmatized groups: By drawing on people’s social networks, snowball sampling can be an effective way to study hard-to-reach groups. Once researchers gain the trust of a few members of the group, those people can help the researchers recruit other people.
- Non-random: A snowball sample will likely provide results that are hard to generalize beyond the sample studied.
- Slow: Because snowball sampling relies on each participant to recommend others, the data collection process is typically slow when compared to other methods.
An Example of Snowball Sampling:
Snowball sampling is an effective way to find people who belong to groups that are difficult to locate. For example, psychologists may use snowball sampling to study members of marginalized groups, such as homeless people, closeted gay people, or people who belong to a support group, such as Alcoholics Anonymous. After gaining the trust of a few people, the researchers could ask the participants to recommend some other members of the group. By proceeding from one recommendation to the next, the researchers may be able to gain a large enough sample for their project.
Who Uses Snowball Sampling?
Snowball sampling is most common among researchers who seek to conduct qualitative research with hard-to-reach groups. Academic researchers might use snowball sampling to study the members of a stigmatized group, while industry researchers might use snowball sampling to study customers who belong to elite groups, such as a private club.
Quota Sampling
When researchers engage in quota sampling, they identify subsets of the population that are important to represent and then sample participants within each subset.
Pros and Cons:
- Representation: Quota sampling ensures representation of important groups within the population being studied.
- Mitigates confounds: Setting quotas within a study is a purposeful action that can help researchers eliminate potential confounds.
- Potential for bias: Because participants within each quota are not randomly drawn, it’s impossible to know how well they represent the groups in the population.
An Example of Quota Sampling:
If you wanted to study Americans’ beliefs about economic mobility, it would be important to sample people from different steps on the economic ladder. That is, you would want to make sure your sample included people who make a lot of money, people who make a moderate amount of money, and some people who make a little bit of money. To obtain this sample, you might set up quotas that are stratified by people’s income. That is what one researcher recently did using CloudResearch’s Prime Panels.
Who Uses Quota Sampling?
Quota sampling is extremely common in both academic and industry research. Sometimes, researchers set simple quotas to ensure there is an equal balance of men and women within a study. At other times, researchers want to represent several groups and, therefore, set up more extensive quotas that allow them to represent several important demographic groups within a sample.
Judgment Sampling
Judgment sampling occurs when a researcher uses his or her own judgment to select participants from the population of interest. The researcher’s goal is to balance sampling people who are easy to find with obtaining a sample that represents the group of interest. Hence, when using judgment sampling, researchers exert some effort to ensure their sample represents the population being studied.
Pros and Cons:
- Efficiency: Judgment sampling is often used when the population of interest is rare or hard to find. By exercising judgment in who to sample, the researcher is able to save time and money when compared to broader sampling strategies.
- Unsystematic: Judgment sampling is vulnerable to errors in judgment by the researcher, leading to bias.
An Example of Judgment Sampling:
Imagine a research team that wants to know what it’s like to be a university president. Compared to the entire population, very few people are or have been employed as the president of a university. Rather than rely on other sampling techniques that have a low probability of contacting university presidents, the researchers may choose a list of university presidents to contact for their study. By using their judgment in who to contact, the researchers hope to save resources while still obtaining a sample that represents university presidents.
Who Uses Judgment Sampling?
Researchers within industry and academia sometimes rely on judgment sampling. Whenever researchers choose to restrict their data collection to the members of some special group, they may be engaged in judgment sampling.
Random Sampling Techniques
Random sampling techniques lead researchers to gather representative samples, which allow researchers to understand a larger population by studying just the people included in a sample. Although there are a number of variations to random sampling, researchers in academia and industry are more likely to rely on non-random samples than random samples.
Simple Random Sampling
Simple random sampling is the most basic form of probability sampling. In a simple random sample, every member of the population being studied has an equal chance of being selected into the study, and researchers use some random process to select participants.
Pros and Cons:
- Strong external validity: Allows researchers to generalize results from the sample to the entire population being studied.
- Relative speed and efficiency compared to the census: A simple random sample allows researchers to learn about an entire population much faster and more efficiently than collecting data from every member of the population.
- Expensive: Contacting a large, randomly selected group of people requires lots of resources.
- Time consuming: Although this method is faster than conducting a census, gathering data from a large, random sample is often slow when compared to other methods.
- Not always possible: Researchers may wish to study a group for which there is no organized list (sampling frame) to randomly sample from.
An Example of Simple Random Sampling:
Researchers who want to know what Americans think about a particular topic might use simple random sampling. The researchers could begin with a list of telephone numbers from a database of all cell phones and landlines in the U.S. Then, using a computer to randomly dial numbers, the researchers could sample a group of people, ensuring a simple random sample.
Who Uses Simple Random Sampling?
Simple random sampling is sometimes used by researchers across industry, academia and government. The Census Bureau uses random sampling to gather detailed information about the U.S. population. Organizations like Pew and Gallup routinely use simple random sampling to gauge public opinion, and academic researchers sometimes use simple random sampling for research projects. However, because simple random sampling is expensive and many projects can arrive at a reasonable answer to their question without using random sampling, simple random sampling is often not the sampling plan of choice for most researchers.
Systematic Sampling
Systematic sampling is a version of random sampling in which every member of the population being studied is given a number. Then, researchers randomly select a number from the list as the first participant. After the first participant, the researchers choose an interval, say 10, and sample every tenth person on the list.
Pros and Cons:
- External validity: Allows generalization from the sample to the population being studied.
- Relative speed: Faster than contacting all members of the population or simple random sampling.
- Limited feasibility: This sampling method is not possible without a list of all members of the population.
An Example of Systematic Sampling
Colleges and universities sometimes conduct campus-wide surveys to gauge people’s attitudes toward things like campus climate. To conduct such a survey, a university could use systematic sampling. By starting with a list of all registered students, the university could randomly select a starting point and an interval to sample with. Contacting every student who falls along the interval would ensure a random sample of students.
Who Uses Systematic Sampling?
Systematic sampling is a variant of simple random sampling, which means it is often employed by the same researchers who gather random samples. Researchers engaged in public polling and some government, industry or academic positions may use systematic sampling. But, much more often, researchers in these areas rely on non-random samples.
Cluster Sampling
Cluster sampling occurs when researchers randomly sample people within groups or clusters the people already belong to.
Pros and Cons:
- External validity: The random nature of selecting clusters allows researchers to generalize from the sample to the entire population being studied.
- Speed: Faster and more efficient than sampling all groups or all people in the population.
- Not always possible: There are several groups researchers may want to study for which there is no organized list from which to randomly select participants.
An Example of Cluster Sampling:
Imagine that researchers want to know how many high school students in the state of Ohio drank alcohol last year. The researchers could study this issue by taking a list of all high schools in Ohio and randomly selecting a portion of schools (the clusters). Then, the researchers could sample the students within the selected schools, rather than sampling all students in the state. By randomly selecting from the clusters (i.e., schools), the researchers can be more efficient than sampling all students while still maintaining the ability to generalize from their sample to the population.
Who Uses Cluster Sampling?
Researchers who study people within groups, such as students within a school or employees within an organization, often rely on cluster sampling. By randomly selecting clusters within an organization, researchers can maintain the ability to generalize their findings while sampling far fewer people than the organization as a whole.
Multistage Sampling
Multistage sampling is a version of cluster sampling. Multistage sampling begins when researchers randomly select a set of clusters or groups from a larger population. Then, the researchers randomly select people within those clusters, rather than sampling everyone in the cluster.
Pros and Cons:
- External validity: Multistage sampling maintains the researchers’ ability to generalize from the sample to the entire population being studied.
- Relative speed: By sampling fewer people, multistage sampling is faster and more efficient than cluster sampling.
An Example of Multistage Sampling:
Researchers who want to study work-life balance and employee satisfaction within a large organization might begin by randomly selecting departments or locations within the organization as their clusters. If each cluster is large enough, the researchers could then randomly sample people within each cluster, rather than collecting data from all the people within each cluster.
Who Uses Multistage Sampling?
Similar to cluster sampling, researchers who study people within organizations or large groups often find multistage sampling useful. Multistage sampling maintains the researcher’s ability to generalize their findings to the entire population being studied while dramatically reducing the amount of resources needed to study a topic.
Stratified Sampling
Stratified sampling is a version of multistage sampling, in which a researcher selects specific demographic categories, or strata, that are important to represent within the final sample. Once these categories are selected, the researcher randomly samples people within each category.
Pros and Cons:
- External validity: Maintains the researcher’s ability to generalize from the sample to the entire population being studied.
- Representation: By selecting important groups to sample within before beginning data collection, the researchers can ensure adequate representation of small and minority groups.
An Example of Stratified Sampling:
Researchers at the Pew Research Center regularly ask Americans questions about religious life. To ensure that members of each major religious group are adequately represented in their surveys, these researchers might use stratified sampling. In doing so, researchers would choose the major religious groups that it is important to represent in the study and then randomly sample people who belong to each group. By using this technique, the researchers can ensure that even small religious groups are adequately represented in the sample while maintaining the ability to generalize their results to the larger population.
Who Uses Stratified Sampling?
Stratified sampling is common among researchers who study large populations and need to ensure that minority groups within the population are well-represented. For this reason, stratified sampling tends to be more common in government and industry research than within academic research.
CloudResearch connects researchers with a wide variety of participants. Using our Prime Panels platform, you can sample participants from hard-to-reach demographic groups, gather large samples of thousands of people, or set up quotas to ensure your sample matches the demographics of the U.S. When you use our MTurk Toolkit, you can target people based on several demographic or psychographic characteristics. In addition to these tools, we can provide expert advice to ensure you select a sampling approach fit for your research purposes. Contact us today to learn how we can connect you to the right sample for your research project.
