Jiangzhou Fu

This article evaluates the use of dynamic adaptive design methods to target outbound computer-assisted telephone interviewing (CATI) in the California Health Interview Survey (CHIS). CHIS 2022 implemented a dynamic adaptive design in which predictive models were used to end dialing early for some cases. 

For addresses that received outbound CATI follow-up, dialing was paused after three calls. A response propensity (RP) model was applied to predict the probability that the address would respond to continued dialing, based on the outcomes of the first three calls. Low-RP addresses were permanently retired with no additional dialing, while the rest continued through six or more attempts. Authors used a difference-in-difference design to evaluate the effect of the adaptive design on calling effort, completion rates, and the demographic composition of respondents. 

Findings: Authors find that the adaptive design reduced the mean number of calls per sampled unit by about 14% (relative to a modeled no-adaptive-design counterfactual) with a minimal reduction in the completion rate and no strong evidence of changes in the prevalence of target demographics. This suggests that RP modeling can meaningfully distinguish between addressed-based sample units for which additional dialing is and is not productive, helping to control outbound dialing costs without compromising sample representativeness.
 

​Summary: The California Health Interview Survey (CHIS) has employed an addressed-based sampling (ABS) frame with a mail push-to-web interview followed by a telephone nonresponse follow-up as the primary data collection approach since 2019. However, the nature of the self-administered web survey results in more survey breakoffs than the previous computer-assisted telephone interview (CATI). During CHIS 2021 data collection, the CHIS team observed that a large proportion of questions with high break-off incidence began with transition statements, such as “The following questions are about…” or “These next questions are about...”. Therefore, experiments were warranted to test whether eliminating transition statements leads to a reduction in survey breakoffs during CHIS 2022.

This study evaluates an experiment conducted in CHIS 2022, where respondents were evenly split and randomly assigned to two conditions: (1) a treatment group where transition statements were removed from the selected twenty-six questions; (2) a control group with the original question wording, including transition statements.

Findings: Data demonstrate that eliminating transition statements results in substantive survey break-offs reductions. Aggregated breakoffs from the 26 questions have decreased by 44.2%. For individual questions, reduction rates range from 14% to 82%. Results also show that removing the transition statements converted sufficient partials to fully completes and slightly shortened interview length. Consequently, all transition statements except an outlier have been removed for the remainder of the CHIS 2022 and transition statements will be less likely to be included in new survey question development for the CHIS.

Read the Publication:

• Report: Impacts of Transition Statements in Survey Questions on Survey Break-off: Evidence from a Survey Experiment