Methods/Statistics
Mode effects on survey item measurement: A systematic review of the experimental evidence Georgia Tomova* Georgia Tomova Tomova Tomova University College London
Survey data are increasingly collected using mixed-mode designs (e.g., face-to-face and online). However, item measurement may differ between modes, introducing ‘mode effects’, a form of systematic measurement error. While the literature discusses the theoretical mechanisms behind this in detail, empirical evidence on the occurrence and magnitude of mode effects is fragmented. Further, some statistical approaches for handling mode effects, such as quantitative bias analysis, remain underused due to the need for such external evidence. Motivated by these issues, we conducted a systematic review of experimental studies of mode effects. We searched multiple bibliographic databases and the grey literature, with backward and forward citation screening. Eligible studies had random allocation of survey mode, sampled from the general population (or age-, sex-, or region-specific strata thereof), and reported mode effect estimates on item measurement. We extracted comprehensive details on the study design, sampling, mode effect estimates, and reporting. Ninety experiments published from 1967 to 2024 met the inclusion criteria, comprising 4,113 mode effect estimates for 3,545 variables. Mode effects were generally small (typically under 0.2 SD). However, larger mode effects were more commonly observed when modes differed by direct interviewer involvement or by question delivery (i.e., visual vs aural), and when variables were sensitive (e.g., sexual behaviour, social life), aligning with pre-existing theory on the causes of mode effects. Generally, mode effects were variable-, mode-, and population-specific. However, reporting quality varied substantially. Post-randomisation non-response and compliance were commonly not reported or not addressed, and may have therefore introduced bias. We collated all mode effect estimates into a free online database to inform future study design and analysis, and provide a set of recommendations to improve the reporting of future research.