This workshop shares the ways in which survey weighting processes can and cannot be used to improve the representativeness of data on small and disaggregated populations within population surveys. The presentations cover the purpose of providing survey weights that account for specific subpopulations, things to consider when selecting a control population to use for calibration, and methods of accounting for small subgroups in weighting data.
Presenters:
Ninez A. Ponce, PhD, MPP, Director, UCLA Center for Health Policy Research
Brian Wells, PhD, Former Survey Methodologist, California Health Interview Survey
Tara Becker, PhD, Senior Public Administration Analyst, UCLA Center for Health Policy Research
About the National Network of Health Surveys' Advancing Health Equity Through Data Disaggregation Workshop Series
Disaggregated race/ethnicity data is needed to expose gaps in health equities and inform policies and programs and close those gaps. The National Network of Health Surveys, part of the UCLA Center for Health Policy Research, offers a series of workshops designed to improve the disaggregation of race and ethnicity measures in health data sources. Our goal is to boost the number of subpopulation categories made available to key constituencies working to improve health equity. This is especially important for representing communities that are often “hidden” in large health data sets.
Download the presentation slides.
Topics and Timestamps
Why is a weighting session included in data disaggregation series? (6:25)
- Survey weights are a critical component to the discussion of data disaggregation.
The purpose of weighting (8:56)
- Weights help reflect the complexity of sample design, and reduce bias reduction.
When are weights unnecessary? (10:44)
- Weights are not always necessary, but certain conditions must hold.
Why we need survey weights (11:51)
- Weights are used when the sample of respondent distribution is not aligned with the population distribution.
Conditions that affect representativeness (13:19)
- Sampling frame is incomplete/contain errors
- Unequal probability of being sampled
- Nonresponse error
General form of weights (13:49)
- Weight = selection probability x sample nonresponse x population adjustment
- What weighting does (27:19)
Selection probability (14:30)
- Understanding relationship between the sample and the framework.
- Selection probability and simple random sample designs (15:24)
- Selection probability and complex designs (16:08)
- Selection probability and data disaggregation (17:46)
- If a study is oversampling a small group, we want to account for this different from the population.
Sampling frame limitations (18:55)
- Sampling frame may underrepresent a subpopulation within the target population, exclude a subpopulation or cover more than the desired population.
- Discussion includes examples of each.
Adjusting for nonresponse (20:33)
- Nonresponse and data disaggregation (22:03)
- If a subgroup of interest responds to a survey at a lower (or higher) rate, you want to account for this difference. Discussion includes examples.
Limitations of sample-based adjustments (23:13)
- Sample based adjustments require knowing information about both respondents AND nonrespondents.
Population-based adjustments (23:55)
- Using information known about the population to make the respondent pool look like the population.
- Benchmark Comparison example (24:29)
- Population adjustments and data disaggregation (26:36)
- Regardless of what our final sample looks like, you want it to reflect the population.
Limitations of weighting (29:08)
- The effectiveness of weighting is constrained by survey methodology and content – it cannot make a sample representative of a missing subpopulation.
- Example of Limitations (30:50)
Weighting considerations and benchmark data (33:18)
- Why do we need a benchmark population? (33:45)
- Tells us what the population SHOULD look like.
- How is the benchmark population used? (34:26)
- Choosing benchmark data (34:55)
- Commonly used benchmark data (37:11)
- Practical example of benchmark data usage (38:20)
- Benchmark data from multiple sources (39:38)
- Practical examples of utilizing benchmark data from multiple sources (40:07)
Weighting dimensions
- What are weighting dimensions? (41:57)
- Set of characteristics that are used to standardize the sample data
- Choosing characteristics for weighting (42:14)
- Defining dimensions (43:13)
- Sample size constraints (44:22)
- Small samples may require collapsing categories, preventing adjustments for disaggregated categories
- Limitations of weighting dimensions (45:48)
- Weighting dimensions might not fully account for differential nonparticipation
Coding race and ethnicity (47:15)
- Measuring race and ethnicity (47:21)
- Constraints on coding race/ethnicity (48:45)
- Practical example (50:23)
- Weighting characteristics: A look at six federal surveys (51:18)
- Visualization of differences based on weighting (53:05)
- Potential solutions for small groups (55:50)
- Suggestions for coding of weighting dimensions and increasing sample size
- Weighting and disaggregated racial/ethnic data (58:25)
- Weighting methodologies matter (58:43)
- The process of weighting survey data (59:20)
