ACF - Administration for Children and Families

11/20/2024 | Press release | Distributed by Public on 11/20/2024 07:22

Probability and Nonprobability Samples in Surveys: Opportunities and Challenges

Probability and Nonprobability Samples in Surveys: Opportunities and Challenges

Publication Date:November 20, 2024

Introduction

Expand

On October 18-19, 2023, the Administration for Children and Families' (ACF) Office of Planning, Research, and Evaluation (OPRE) hosted the 2023 Methods Meeting, titled "Addressing Unit Missingness in Social Policy Survey Research." During the meeting, several presenters discussed challenges observed in surveys using probability sampling and introduced nonprobability sampling as an emerging trend in the survey research field. OPRE invited two survey research experts, Dr. J. Michael Brick of Westat and Dr. Michael Bailey of Georgetown University, to provide their perspectives on the use of probability and nonprobability sampling in surveys. The experts' responses, presented individually, provide an overview of probability and nonprobability sampling, factors that guide the selection of one approach over the other, opportunities and challenges associated with each approach, and potential applications of nonprobability sampling.

Purpose

Expand

This document serves as a reference to guide researchers interested in learning more about probability and nonprobability sampling. This resource contains two experts' responses to the emerging debate on the use of probability and nonprobability sampling in surveys:

  • Opt-in Panels: Error, Inference, and Qualityby Dr. J. Michael Brick

  • Coming to Grips with the Fact that Even Probability-Based Polls are Not Random Samples by Dr. Michael Bailey

Key Findings and Highlights

Expand

Opt-in Panels: Error, Inference, and Quality

  • Researchers are exploring alternatives to probability sampling to combat increasing survey data collection costs. This brief examines one of the most widely discussed alternative approaches, nonprobability sampling. Unlike probability sampling, nonprobability sampling does not require known selection probabilities for all units in a finite population. Thus, nonprobability sampling is not a concept that can be easily defined and evaluated.

  • Opt-in panels are popular in the United States and frequently used in polling and commercial surveys. There are two main sources of survey error in opt-in panels: representation and measurement.

    • Representation error occurs when the individuals in the respondent sample do not reflect the target population. Federal agencies rarely use opt-in panels, largely due to concerns that results from these surveys cannot be generalized to the universe of study. Weighting methods to produce unbiased estimates are difficult to assess.

    • Measurement error relates to the quality of the responses from participating individuals. Motivation for participating in an opt-in panel for example, may affect responses, leading to less accurate data.

  • Nonprobability samples can be useful for some purposes, such as exploratory research which can inform the next stage of survey development, or when populations are very homogeneous.

  • Probability samples can also have shortcomings. When real-world applications deviate from assumptions, there may be serious effects on inferences.

Coming to Grips with the Fact that Even Probability-Based Polls are Not Random Samples

  • Modern probability-based surveys are based on contacting a random sample. However, a random contact survey does not necessarily produce a random sample when response rates are very low. For this reason, we need to move beyond random sampling based theory to understand the properties of contemporary surveys.

  • Even when probability and nonprobability-based survey do not produce a random sample, they can be accurate if the people who respond are representative of those who do not respond once we account for demographics. However, samples may be systematically different from the population in ways that are not accounted for by weighting.

  • Rather than assume weighted survey data are random, we can think about characterizing error for any sample as random or not. When the propensity to respond is associated with a survey attribute, survey error can be large and increase relative to the population size. Research shows that random contact surveys, even those with poor response, are better at limiting this bias (Meng, 2018).

  • Probability-based sampling does not eliminate error in the low response rate environment, but it attenuates error and is the best we can do in the modern age of surveys.

Citation

Expand

Bailey, M. & Brick, J.M. (2024). Probability and Nonprobability Samples in Surveys: Opportunities and Challenges(OPRE Report 2024-182). Prepared by Westat Insight. U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation.

Glossary

Nonprobability sampling: A sampling approach that does not require known selection probabilities for all units in a finite population. Opt-in panels: A sampling approach that uses self-reported information from potential participants to select panelists for surveys. Probability sampling: A method of selecting units for observation such that each unit in a finite population has a known, positive probability of selection. Random contact-based surveys: A common probability sampling approach that involves communicating with a random sample of possible participants when fielding a survey. This does not necessarily lead to recruiting a random sample. Selection bias: The bias introduced when participants do not represent a true random sample of the population of interest.
Types:
OPRE Research Topics: