Examination of the demographic representativeness of a cross-sectional mobile phone survey in collecting health data in Colombia using random digit dialling

Abstract

Objectives

As mobile phone ownership becomes more widespread in low-income and middle-income countries, mobile phone surveys (MPSs) present an opportunity to collect data on health more cost-effectively. However, selectivity and coverage biases in MPS are concerns, and there is limited information about the population-level representativeness of these surveys compared with household surveys. This study aims at comparing the sociodemographic characteristics of the respondents of an MPS on non-communicable disease risk factors to a household survey in Colombia.

Design

Cross-sectional study. We used a random digit dialling method to select the samples for calling mobile phone numbers. The survey was conducted using two modalities: computer-assisted telephone interviews (CATIs) and interactive voice response (IVR). The participants were assigned randomly to one of the survey modalities based on a targeted sampling quota stratified by age and sex. The Quality-of-Life Survey (ECV), a nationally representative survey conducted in the same year of the MPS, was used as a reference to compare the sample distributions by sociodemographic characteristics of the MPS data. Univariate and bivariate analyses were performed to evaluate the population representativeness between the ECV and the MPSs.

Setting

The study was conducted in Colombia in 2021.

Participants

Population at least 18 years old with a mobile phone.

Results

We completed 1926 and 2983 interviews for CATI and IVR, respectively. We found that the MPS data have a similar (within 10% points) age–sex data distribution compared with the ECV dataset for some subpopulations, mainly for young populations, people with none/primary and secondary education levels, and people who live in urban and rural areas.

Conclusions

This study shows that MPS could collect similar data to household surveys in terms of age, sex, high school education level and geographical area for some population categories. Strategies are needed to improve representativeness of the under-represented groups.

Correspondence to Dr Deivis Nicolas Guzman-Tordecilla; [email protected]

STRENGTHS AND LIMITATIONS OF THIS STUDY

• This is the first study in Latin America that has examined the representativeness of mobile phone surveys with random digit dialling using a nationally representative household survey as a referent.

• Due to budgetary constraints, sample sizes were not met for any of the computer-assisted telephone interview age–sex quota.

• We have used ‘age–sex’ stratification for improving and ensuring high response rates across all age groups among men and women.

Introduction

Health surveys are important tools for monitoring disease burden and disease risk factors and resource allocation, and evaluating health policies.¹ In low-income and middle-income countries (LMICs), household surveys have traditionally been the gold standard means of data collection. However, household surveys require face-to-face (FTF) interviews, multiple attempts to gather information from selected households and often travel to hard-to-reach places.^{1 2} Consequently, they are conducted infrequently due to high costs related to personnel and transportation.¹ Moreover, conducting household surveys are more challenging in security risk areas or when epidemiological (eg, pandemics) conditions change.^{3 4} Alternative or supplemental data collection methodologies have emerged to address some of these challenges.^{4 5}

High-income countries have implemented mobile phone surveys (MPS) to collect health and demographic data at the population level because of their lower cost and the population’s universal access to mobile phones and landlines.^6–9 Currently, LMICs have shown a high penetration rate of mobile phones.¹⁰ For example, in Latin America, 70% of people own mobile phones.¹¹ In Colombia, there are 133 mobile phone subscribers per 100 adults.¹²

Taking advantage of the high penetration of mobile phones globally and in LMICs, research shows there are increasing efforts to introduce MPS, which includes short message service, interactive voice response (IVR), and computer-assisted telephone interviews (CATI), survey modalities^{13 14} for more cost-effective and rapid data collection.^{4 15}

However, there are concerns of population representativeness of MPS in countries where coverage of mobile phones is not yet universal.¹⁶ Moreover, some population groups may be more willing to participate in a survey compared with others. It is possible to know the extent of non-response rates by demographic characteristics in household surveys and accordingly use weighting method to adjust the estimates, but that may not be feasible for MPS. Few studies compared the representativeness of MPS to standard health FTF surveys in LMICs, particularly from middle-income countries.^{5 17–19} The sparse evidence published in LMICs about this topic is mainly from low-income countries.^{5 20–23} As far as we know, published studies in middle-income countries that compare the sociodemographic characteristics of MPS using IVR and CATI are scant.^{3 24–26}

This study aims to examine the population-level representativeness of MPS by comparing selected sociodemographic characteristics of respondents of an MPS, which collected data on non-communicable disease (NCD) risk factors, with those who completed a similar FTF survey—Quality-of-Life Survey (ECV)—in Colombia. Studies have shown that age, sex, education and geographical residence variables are key determinants of selection and coverage biases in MPS.^{16 27} We have used ‘age–sex’ stratification for improving and ensuring high response rates across all age groups among men and women.¹⁶ Our specific interest is to compare the sample distributions of education level and geographical areas across the eight age–sex strata of the MPS. Information on the extent of representations will help the country to develop appropriate sample weighting for improving representativeness of the MPS data.

Methods

Study setting

The MPS was conducted under the Bloomberg Data for Health Initiative,²⁸ which aims to develop and optimise MPS to collect data on NCD risk factors in Colombia and elsewhere. Due to regulations related to access to personal data of individuals in Columbia,²⁹ there is no sampling frame available that lists working (activated) mobile phone numbers. We used the random digit dialling technique to generate potential mobile phone numbers.³⁰ We used the first three digits corresponding to the prefixes from the mobile network operators for Colombia, ranging from 300 to 351, and the remaining seven numbers were randomly generated. We distributed the prefixes of the mobile phone numbers evenly. Potential mobile phone numbers were generated only once using STATA V.14.³¹ The phone numbers were not repeated across CATI and IVR modalities.

CATI data collection

A call centre based in Bogota, Colombia, was hired to conduct CATIs. Calls were performed between 08:00 and 17:00 local time, from Monday to Saturday, by six call agents or interviewers with at least 1 year of CATI experience. For each potential phone number, a maximum of three call attempts were made in case the respondents missed the call or could not answer. Randomly generated phone numbers were uploaded to a custom dialling software platform that directs the call to a call centre agent. Individuals were eligible for the survey if they were at least 18 years old and the sample size had not yet been reached for each of the eight age–sex strata (male/female, ages 18–29, 30–44, 45–59 and 60+ years old).

All call centre agents received training to conduct the CATI survey and were supervised by a field coordinator. The survey instrument and CATI software were piloted prior to full data collection. Data collection took place from 15 January to 13 April 2021. Only those who provided their informed consent were administered the survey.

IVR data collection

The IVR survey involved a series of pre-recorded questions in Spanish.³² The audios were professionally recorded by a nationally recognised female journalist and then loaded onto a self-service platform, engageSPARK.³³ The survey response options were programmed to be answered using the mobile phone keypad. For example, ‘If you are male, press 1. If you are female, press 3’.

The pilot phase consisted of several trials to test the performance of the platform, understanding of the questions, and audio quality and comprehension. The pilot phase was conducted in early January 2021. Adjustments were made to some of the question scripts and optimising audios to improve the quality and comprehension recorded. IVR surveys were launched between 08:00 and 17:00 local time, from Monday to Saturday in batches of 250 000 numbers at a time from 26 January to 23 April 2021.

MPS questionnaire

The questionnaire applied in both modalities (IVR and CATI) was the same, except for minor changes specific to each survey modality for the sake of clarity. The survey included several components: (1) informed consent; (2) demographic characteristics; (3) NCD modules (tobacco use, alcohol use, dietary intake, physical activity and blood pressure); and (4) delivery of incentives. NCD modules are groups of questions on NCD behavioural risk factors covered by existing risk factor surveys, such as the WHO STEPS survey and the Global Adult Tobacco Survey.^{34 35} All the answer options were multiple choice or binary, except for questions on age, food consumption and physical activity.

Calls through all modalities were free to respondents. All participants who completed the survey and had a prepaid mobile phone plan were automatically sent an airtime incentive of COP 5000 to their mobile phones (approximately equivalent to US$1.37 at the time). Postpaid mobile phone plans do not accept top-up credits, but they are a minority of all lines available in Colombia.³⁶

National household survey questionnaire

We used the data from the ECV (for its initials in Spanish) as a reference for the representativeness of population for the reference period to compare the characteristics of our data. The ECV survey was conducted during 2021 when the MPS was implemented. ECV collects information on the living conditions of Colombians, including variables related to housing, education, health, labour force, etc. ECV interviewed 257 589 subjects using FTF encounters. The survey included data on men and women aged 0–112 years old from rural and urban areas. For our study, we restricted the ECV sample to those 18 years of age or older. Details on the sampling and representativeness, validity and reliability of the ECV can be consulted elsewhere.³⁷

Respondent characteristics

Sociodemographic information was included from those who completed at least one NCD module of the MPS. The MPS included four sociodemographic variables. Age (years old) was measured as a continuous variable and then was grouped into four categories (18–29, 30–44, 45–59 and 60+). Education level (none/primary, secondary, and university or higher), geographical area (urban vs rural) and sex (men vs women) were measured as ordinal and binary variables.

The same sociodemographic variables and measures were selected for the ECV survey. However, respondents below the age of 18 years were excluded from the ECV to enable a proper comparison of the variables’ distribution between the two surveys. This was because the MPS only collected data from respondents who were at least 18 years old.

Also, we categorised the results from calls of the MPS-based standards set in the American Association for Public Opinion Research.³⁸ A complete interview was defined as answering at least four of the five NCD modules, and a partial interview was defined as answering at least one, but no more than four, NCD modules. Break-offs were defined as those who were age–sex eligible with less than one NCD module completed.

Sample size

The MPS used stratified random sampling design with eight age–sex strata men and women by age groups 18–29, 30–44, 45–59, and 60 and older) in each modality (IVR and CATI). We estimated a quota sample of 385 observations for each age–sex strata, with an expected prevalence of the risk factor of 50% (p=0.5), a margin of error of 5% (δ=0. 05) and 5% type I error (α=0. 05) and as recommended by WHO.³⁹ We generated the potential mobile phone numbers using the digit dialling technique, as mentioned previously. The age–sex quota stratification was used to improve representations of some population groups in the MPS surveys, such as older and female populations.^{20 23}

Statistical analysis

We used several approaches to compare the sociodemographic characteristics of the surveys (CATI– IVR and ECV). First, we performed a univariate analysis from the call disposition codes. Second, we conducted a bivariate analysis between age–sex strata by mobile phone modality to identify whether sampling quotas were met. Third, we estimated and compared 95% CIs of selected sociodemographic characteristics between the MPS and FTF (ECV) for checking their statistical comparability. We considered that the surveys have dissimilar distributions if the CIs do not overlap.⁴⁰ Because the sampling designs of the MPS and FTF surveys were different and the FTF was based on extremely large sample size (n=257 589) compared with MPS (n=2581 and 1837 for IVR and CATI, respectively), we did not combine the datasets for formal statistical tests. Also, we estimated differences in sample distribution proportions between MPS versus ECV data. We used STATA V.14 to run all the analyses.

Patient and public involvement

No members of the public or patients were involved in this study.

Results

Call outcomes

For the CATI survey, 105 050 numbers were dialled (table 1 and figure 1). Approximately 21% (n=21 949) of all calls made were sent to an inactive line, and 69% (n=72 331) did not pick up the call. Of the remaining 10% who picked up the call, 8798 people hung up the call; 15 refused to participate in the study; 80 were under 18 years old; 41 were partial interviews; and 1837 were complete interviews. The average survey length was 23 min and 10 s (table 2).

Figure 1. Flowchart of call outcomes for CATI and IVR. It starts with the quantity of dialled mobile phone numbers and ends with the completed MPS. CATI, computer-assisted telephone interview; IVR, interactive voice response; MPS, mobile phone survey; NCD, non-communicable disease.

Table 1

Call outcomes for the random digit dial mobile phone samples

Call outcomes	IVR		CATI
	n	%	n	%
Complete interview	2581	0.21	1837	1.75
Partial Interview	402	0.03	41	0.04
Break-off	341	0.03	38	0.04
Refusal	151	0.01	15	0.01
Unknown, uncertain age–sex eligible	615 926	50.01	8759	8.34
Unknown, no response	381 523	30.97	37 260	35.47
Unknown, voicemail	NA	0	35 071	33.39
Ineligible: age <18	630	0.05	80	0.07
Ineligible: line not registered	228 256	18.53	21 949	20.89
Ineligible: quota met	1969	0.16	0	0
Total calls	1 231 779	100	105 050	100

Note: Data are n and percentages are presented for each of the modalities of the mobile telephone surveys. NA for IVR indicates that no data was reported for that category.

CATI, computer-assisted telephone interview; IVR, interactive voice response.

Table 2

Cost, time and call outcome rates for mobile phone surveys

Response rates	IVR	CATI
Contact rate 1	0.35%	2.33%
Response rate 2	0.30%	2.26%
Refusal rate 1	0.05%	0.06%
Cooperation rate 1	74.27%	95.13%
Cooperation rate 2	85.84%	97.26%
Calls per complete interview*	341	57
Average survey length	17 min 23 s	23 min 10 s
Cost/complete survey	US$35.51	US$6.36 (excludes person time)

*For IVR, restricted analysis until the first age–sex quota was filled; 498 249 phone calls, 1459 complete interviews. The cost to CATI did not include the salary of call agents.

CATI, computer-assisted telephone interview; IVR, interactive voice response.

For the IVR survey, 1 231 779 numbers were dialled (table 1 and figure 1). Approximately 18% (n=228 256) of all calls made were sent to an inactive line; 31% (n=381 523) did not pick up the call; and 50% of the responders hung up the call. Of the remaining observations, 102 refused to participate in the study; 630 were under 18 years old; 402 were partial interviews; and 2581 were complete interviews. The average survey length was 17 min and 23 s (table 2).

Contact, response and cooperation rates were higher for CATI, 2.33%, 2.26% and 95.15%, respectively, compared with IVR, 0.35%, 0.30%, and 74.27% respectively (table 2).

Sociodemographic characteristics of MPSs

For the CATI survey, we did not meet quota sampling in any age–sex group. The two age–sex strata closer to filling the quota were women between 18–29 years old (n=295) and 30–44 years old (n=370) (table 3). Approximately 78% (n=1465) of the sample had completed high school or higher, and 78% lived in an urban area (table 4). For the IVR survey, we met five quotas: men aged 18–29 and 30–44 years and women aged 18–29, 30–44 and 45–59 years (table 3). Approximately 82% (n=2446) of the sample had completed high school or higher, and 82% lived in an urban area (table 4).

Table 3

Quota met by mobile phone modality

Survey modality	Age groups (years)	Male		Female
		Complete surveys	Exceeded	Complete surveys	Exceeded
IVR	18–29	386	367	389	815
	30–44	386	190	391	571
	45–59	291	0	387	26
	60+	150	0	120	0
CATI	18–29	210	0	295	0
	30–44	252	0	370	0
	45–59	175	0	241	0
	60+	131	0	160	0

Note: Table 3 presents the number of surveys reached by each of the age–sex strata for the mobile telephony modalities in the study. It also reports the categories for which quotas were met. This table does not have three observations for the CATI modality since they did not identify themselves as female or male.

CATI, computer-assisted telephone interview; IVR, interactive voice response.

Table 4

Comparison of face-to-face and MPS samples by age strata among (A) women and (B) men

(A) Women
Age group	Sociodemographic variables	ECV 2021 (%) (95% CI)	CATI 2021	IVR 2021	CATI versus ECV 2021	IVR versus ECV 2021	CATI+IVR
			(N=1114)	(N=1507)	Proportion difference (%)	Proportion difference (%)	Total sample of MPS for women, % (n)
			% (n) (95% CI)	% (n) (95% CI)	95% CIs overlap (yes/no)	95% CIs overlap (yes/no)
18–29	Education
	None/primary	7.6 (7.1 to 8.2)	11.7 (36) (8.6 to 15.9)	7.5 (34) (5.4 to 10.4)	−4.1 (yes)	−0.1 (yes)	9 (70)
	Secondary	81.2 (80.1 to 82.1)	72.6 (223) (67.3 to 77.4)	74.0 (334) (69.8 to 77.9)	−8.6 (no)	−7.2 (no)	74 (557)
	University or higher	11.2 (10.3 to 12.2)	15.6 (48) (11.9 to 20.2)	18.4 (83) (15.1 to 22.3)	+4.4 (yes)	+7.2 (no)	17 (131)
	Location
	Rural	22 (21.4 to 22.7)	26.4 (81) (21.7 to 31.6)	21.8 (98) (18.2 to 25.8)	+4.4 (yes)	−0.2 (yes)	24 (179)
	Urban	78 (77.3 to 78.6)	73.6 (226) (68.4 to 78.3)	78.2 (352) (74.2 to 81.8)	−4.4 (yes)	+0.2 (yes)	76 (578)
30–44	Education
	None/primary	16.9 (16.3 to 17.6)	16.5 (64) (13.1 to 20.6)	15.1 (68) (12.1 to 18.7)	−0.4 (yes)	−1.8 (yes)	16 (132)
	Secondary	62.9 (61.7 to 63.9)	56.3 (218) (51.3 to 61.2)	62.5 (282) (57.9 to 66.9)	−6.6 (no)	−0.4 (yes)	60 (500)
	University or higher	20.2 (19.2 to 21.3)	27.1 (105) (22.9 to 31.8)	22.4 (101) (18.8 to 26.5)	+6.9 (no)	+2.2 (yes)	24 (206)
	Location
	Rural	20.5 (19.9 to 21.1)	22.5 (87) (18.6 to 26.9)	18.1 (81) (14.8 to 21.9)	+2 (yes)	−2.4 (yes)	20 (168)
	Urban	79.5 (78.9 to 80.1)	77.5 (300) (73.1 to 81.4)	81.8 (366) (78.0 to 85.2)	−2 (yes)	+2.3 (yes)	80 (666)
45–59	Education
	None/primary	36.1 (35.1 to 37.2)	35.6 (87) (29.8 to 41.9)	23.5 (108) (19.8 to 27.6)	−0.5 (yes)	−12.6 (no)	28 (195)
	Secondary	50.5 (49.3 to 51.7)	43.4 (106) (37.3 to 49.8)	54.9 (252) (50.3 to 59.4)	−7.1 (yes)	+4.4 (yes)	51 (358)
	University or higher	13.3 (12.4 to 14.3)	20.9 (51) (16.2 to 26.5)	21.6 (99) (18.0 to 25.6)	+7.6 (no)	+8.3 (no)	21 (150)
	Location
	Rural	19.0 (18.4 to 19.6)	25.4 (62) (20.3 to 31.3)	14.4 (66) (11.5 to 17.9)	+6.4 (no)	−4.6 (no)	18 (128)
	Urban	80.9 (80.4 to 81.6)	74.6 (182) (68.7 to 79.7)	85.6 (391) (82.0 to 88.5)	−6.3 (no)	+4.7 (no)	82 (573)
60+	Education
	None/primary	61.1 (59.8 to 62.4)	43.6 (71) (36.1 to 51.3)	39.7 (56) (31.9 to 48.1)	−17.5 (no)	−21.4 (no)	42 (127)
	Secondary	30.3 (29.1 to 31.5)	30.7 (50) (24.0 to 38.3)	41.8 (59) (33.9 to 50.2)	+0.4 (yes)	+11.5 (no)	36 (109)
	University or higher	8.6 (7.81 to 9.43)	25.8 (42) (19.6 to 33.1)	18.4 (26) (12.8 25.8)	+17.2 (no)	+9.8 (no)	22 (68)
	Location
	Rural	19.4 (18.7 to 20.0)	22.6 (37) (16.7 to 29.6)	13.5 (19) (8.7 to 20.3)	+3.2 (yes)	−5.9 (yes)	18 (56)
	Urban	80.6 (79.9 to 81.3)	77.4 (127) (70.3 to 83.2)	86.5 (122) (79.7 to 91.3)	−3.2 (yes)	+6 (yes)	82 (249)
(B) Men
Age groups	Socioeconomic variables	ECV 2021 (%) (95% CI)	CATI survey	IVR survey	CATI versus ECV 2021	IVR versus ECV 2021	CATI+IVR
			(n=764)	(n=1476)	Proportion difference (%)	Proportion difference (%)	Total sample of MPS for men % (n)
			% (n) (95% CI)	% (n) (95% CI)	95% CIs overlap (yes/no)	95% CIs overlap (yes/no)
18–29	Education
	None/primary	10.9 (10.3 to 11.6)	6.8 (15) (4.1 to 11.0)	10.1 (47) (7.7 to 13.2)	−4.1 (yes)	−0.8 (yes)	9 (62)
	Secondary	79.9 (78.8 to 80.9)	73.2 (161) (66.9 to 78.7)	71.2 (331) (66.8 to 75.1)	−6.7 (no)	−8.7 (no)	72 (492)
	University or higher	9.2 (8.3 to 10.2)	20 (44) (15.2 to 25.9)	18.7 (87) (15.4 to 22.5)	+10.8 (no)	+9.5 (no)	19 (131)
	Location
	Rural	22.6 (21.9 to 23.3)	20.4 (45) (15.5 to 26.2)	20.3 (95) (16.9 to 24.3)	−2.2 (yes)	−2.3 (yes)	20 (140)
	Urban	77.4 (76.7 to 78.1)	79.6 (176) (73.8 to 84.5)	79.7 (372) (75.7 to 83.1)	+2.2 (yes)	+2.3 (yes)	80 (548)
30–44	Education
	None/primary	21.5 (20.7 to 22.3)	18.8 (49) (14.5 to 24.1)	21 (98) (17.5 to 24.9)	−2.7 (yes)	−0.5 (yes)	20 (147)
	Secondary	61.5 (60.4 to 62.6)	49.2 (128) (43.2 to 55.3)	58.0 (271) (53.5 to 62.4)	−12.3 (no)	−3.5 (yes)	55 (399)
	University or higher	16.9 (15.9 to 18.0)	31.9 (83) (26.5 to 37.9)	20.9 (98) (17.5 to 24.9)	+15 (no)	+4 (yes)	25 (181)
	Location
	Rural	22.2 (21.6 to 22.9)	20.4 (53) (15.9 to 25.7)	19.8 (92) (16.4 to 23.6)	−1.8 (yes)	−2.4 (yes)	20 (145)
	Urban	77.7 (77.1 to 78.4)	79.6 (207) (74.2 to 84.1)	80.2 (373) (76.3 to 83.6)	+1.9 (yes)	+2.5 (yes)	80 (580)
45–59	Education
	None/primary	38.9 (37.8 to 40.2)	21.6 (40) (16.2 to 28.2)	21.4 (75) (17.5 to 26.1)	−17.3 (no)	−17.5 (no)	22 (115)
	Secondary	48 (46.7 to 49.3)	47.6 (88) (40.4 to 54.8)	53.3 (186) (48.0 to 58.5)	−0.4 (yes)	+5.3 (yes)	51 (274)
	University or higher	13.0 (12.0 to 14.0)	30.8 (57) (24.5 to 37.9)	25.2 (88) (20.9 to 30.1)	+17.8 (no)	+12.2 (no)	27 (145)
	Location
	Rural	24.3 (23.5 to 25.0)	15.1 (28) (10.6 to 21.1)	13.2 (303) (10 to 17.2)	−9.2 (no)	−11.1 (no)	42 (331)
	Urban	75.7 (74.9 to 76.5)	84.9 (157) (78.8 to 89.4)	86.8 (303) (82.8 to 89.9)	+9.2 (no)	+11.1 (no)	58 (460)
60+	Education
	None/primary	58.1 (56.7 to 59.5)	38.6 (51) (30.6 to 47.3)	25 (43) (19.0 to 32.1)	−19.5 (no)	−33.1 (no)	31 (94)
	Secondary	29.1 (27.8 to 30.5)	19.7 (26) (13.7 to 27.5)	45.9 (79) (38.5 to 53.5)	−9.4 (no)	+16.8 (no)	34.5 (105)
	University or higher	12.8 (11.7 to 13.9)	41.7 (55) (33.5 to 50.4	29.1 (50) (22.7 to 36.4)	+28.9 (no)	+16.3 (no)	34.5 (105)
	Location
	Rural	26.3 (25.4 to 27.2)	23.6 (31) (17.08 to 31.80)	15.7 (27) (10.9 to 21.9)	−2.7 (yes)	−10.6 (no)	19 (58)
	Urban	73.7 (72.8 to 74.6)	76.3 (100) (68.2 to 82.9)	84.3 (145) (78 to 89)	+2.7 (yes)	+10.6 (no)	81 (245)

Note: Table 4 reports the distribution of sociodemographic variables by MPS age–sex strata. Also reported is the difference between proportions between MPS versus ECV (eg, CATI vs ECV 2021, IVR vs ECV 2021). We considered that the surveys have dissimilar distributions if the confidence intervals do not overlap between MPS and ECV (yes or no). The ECV data were only for those 18 years of age or older. Since ECV sample weights were used, the ‘n’ values are not reported for ECV.

CATI, computer-assisted telephone interview; ECV, Quality-of-Life Survey; IVR, interactive voice response; MPS, mobile phone survey.

In comparing female participants from ECV and CATI samples, we found that systematically, all proportions of difference were within 10% points for each age strata for education and geographical variables except for distribution of the least and most educated in those aged 60 years and above. The distribution of women with none or primary education was similar between ECV and CATI for age groups 30–44 (16.9% (95% CI 16.3% to 17.6%) vs 16.5% (95% CI 13.1% to 20.6%)) and 45–59 (36.1% (95% CI 35.1% to 37.2%) vs 35.6% (95% CI 29.8% to 41.9%)). Similar distributions for those with university or higher education were observed only in the 18–29 strata (11.2% (95% CI 10.3% to 12.2%) vs 15.6% (95% CI 11.9% to 20.2%)). For other age strata, CATI had higher levels of university education than ECV (table 4).

In comparing female participants from ECV and IVR samples, we also found that systematically, all proportions of difference were within 10% points for each age group for education and geographical variables except for distributions of the least educated in those 45–59 and 60+ and those with secondary education in the oldest age group. IVR respondents were similar to ECV respondents for primary education in age groups 18–29 (7.5% (95% CI 5.4% to 10.4%) vs 7.6% (95% CI 7.1% to 8.2%)) and 30–44 (15.1% (95% CI 12.1% to 18.7%) vs 16.9% (95% CI 16.3% to 17.6%)). The distribution of those with highest education was similar only for 30–44 year-olds. Like CATI, remaining female-age distributions for university education were higher in IVR than ECV (table 4).

Geographical location was similar between ECV and MPS distributions for all age–sex strata except for CATI and IVR respondents aged 45–59 (table 4).

For male participants, there were more instances, as compared with female participants, where the absolute differences in proportions between the ECV sample and MPS sample were greater than 10%. The distribution of men with none or primary education was similar between CATI and ECV for age groups 18–29 (6.8% (95% CI 4.1% to 11.0%) vs 10.9% (95% CI 10.3% to 11.6%)) and 30–44 (18.8% (95% CI 14.5% to 24.1%) vs 21.5% (95% CI 20.7% to 22.3%)). For those with university or higher education, there were no similar proportions for any of the age strata (table 4).

In comparing male participants from IVR and ECV, there were similar differences of proportions for those with none or primary education for age-group 18–29 years (10.1% (95% CI 7.7% to 13.2%) vs 10.9% (95% CI 10.3% to 11.6%)) and 30–44 years (21.0% (95% CI 17.5% to 24.9%) vs 21.5% (95% CI 20.7% to 22.3%)). Respondents with university or higher education were only similar in the age group 30–44 years (20.9% (95% CI 17.5% to 24.9%) vs 16.9% (95% CI 15.9% to 18.0%)) (table 4).

Location was similar for male participants in both CATI and IVR, as compared with ECV for age groups 18–29 and 30–44 years. Location was also similar for oldest CATI but not IVR respondents (table 4).

Overall, we found that the MPS (CATI and IVR) reported higher education levels than the ECV, especially for the university or higher education levels. The age groups with the higher representative in the MPS samples were people between 18–29 years old and 30–44 years old, and the most representative sex in the sample was female (table 4). We also observed that most of the MPS responders were from urban areas, similar to the ECV data (table 4). Additionally, in the 45–69 age group, the CATI surveys reported better performance related to the number of categories with similar proportion distribution to the ECV (table 4).

Discussion

To our knowledge, this is the first study in Latin America that has examined the representativeness of random digit dialling technique MPS surveys using a nationally representative household survey as a referent. One of the main findings of this study was that MPS could collect similar data as household survey in terms of education level and geographical area across several age–sex strata.

Evidence from LMICs has shown that mobile phones are an effective tool to collect health data, especially from younger, male, educated and urban populations.^{20 21 23 41} Although we have used age strata quota for tracking and improving responses among older population, our study found that people older as those over 45 years old were still difficult to reach in our data collection procedure. Most of the participants in this study (CATI–IVR) were people between 18 years old and 44 years old. The previous finding is consistent with the results from prior research conducted in Colombia, which explored the perception and feasibility of implementing MPS.³² That study found that respondents perceive these types of surveys as inconvenient since they can take a long time and interfere with their daily activities since the CATI survey was implemented during business hours (08:00–17:00) and took about 23 min to complete the survey. Consequently, it is possible that surveying the population during that time frame impacted the survey completion quotas in the population over 45 years old, which is an economically active population in Colombia.⁴² Furthermore, the same study mentioned previously also indicated that some adults have more free time to answer this kind of survey,³² which could explain why some survey completions were achieved in the adult population.

Different from other studies published in LMIC,^{4 20 23 43} the current study found that women were the most represented population group in our data. This may be related to the fact that women are more willing to answer this type of survey.⁴⁴ This could be due to the fact that in Colombia, women are more likely to be homemakers, while men are typically employed outside of the home. This might result in men having less time to answer calls, leading to a lower response rate than women. On the other hand, our study supports the assertion that those with higher education levels are the ones who most respond to this type of survey. Our data illustrate that more than 80% of the sample that answers the MPS obtained a secondary or higher education. This is reasonable since some studies in Colombia found that one of the main barriers to implementing MPS is technological literacy,^{32 45} which is more likely in people with a lower level of education.

Additionally, we find that most of the surveys came from urban areas. This is consistent with the population distribution presented by household survey used as a reference in this study and data from the 2018 census. This may also be due in part to the fact that there is better mobile phone coverage in urban areas than in rural areas. Moreover, a study in Colombia found that people who live in rural areas usually do not pick up calls from unknown mobile phone numbers.⁴⁵ Also, our findings showed that the digit dialling technique was an effective technique for generating mobile phone numbers to collect data with CATI and IVR modalities. This is similar to other previous findings in other LMICs, where the Random digit dialing (RDD) technique successfully collected data on health comparable to national health surveys.²⁰

We also found that more sample quotas were achieved for IVR because 10 times more numbers were dialled in IVR than CATI. That difference was due to time and money constraints. With unlimited resources and continuing with CATI calls, we would likely have completed some or all of the CATI quotas. The higher response and cooperation rates observed in CATI are likely due to the presence of the human interviewers and their ability to build trust with the respondents. However, it should be noted that the CATI survey after 45 years reported the greatest number of similarities with ECV than IVR. On the other hand, deploying MPS surveys might be relatively inexpensive compared with other types of FTF surveys.¹ For example, the survey for IVR in our study cost, on average, about US$36, and CATI cost US$6 without including the salary of the call agents for CATI.

One of the limitations of this study was the small number of demographic variables available in the mobile phone sample. Although this may limit the capacity to explore in-depth the differences between mobile phone and FTF samples, education and location are the main variables collected by household surveys in Colombia. Another limitation to report is the fact that we do not use sample weights for cell phone surveys as other studies of this nature have done in the past,^{22 23} but this might be more important if we were generating population-level estimates for a given indicator such as current tobacco smokers. In addition, CATI calls were made during working hours, which could affect the availability to pick-up or respond to calls. To mitigate this limitation, calls were also made on Saturdays since many people do not work that day in Colombia. Finally, conducting this study during the COVID-19 pandemic created an additional challenge.

Conclusions

This study shows that MPS could collect similar data as household survey in terms of education level and geographical area across several age–sex strata, especially in populations under 45 years old. Also, the findings have implications for health by providing insights into the feasibility of using MPSs to collect data on NCD risk factors. More work is needed to reach better representativeness of under-represented groups. Future studies should consider exploring other data collection strategies or alternating MPS modalities to improve its representativeness.

The authors thank the EngageSpark team for their support in implementing interactive voice response surveys and the Edumetrika team for the computer-assisted telephone interview surveys.

Data availability statement

Data are available upon reasonable request.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

This study involves human participants and was approved by the research and ethics committee of the Public Health Institute of Pontificia Universidad Javeriana (0001 from 2020) and the Johns Hopkins Bloomberg School of Public Health Institutional Review Board (#00007318). The participants gave informed consent to participate in the study before taking part.

Footnote

Twitter @NicolsGuzmnTor1, @0000-0003-0689-487X

Contributors Conception or design of the work: DNG-T, AIVO, DGG, AT-Q, CAS-B, VM, REP, JA, SA and GWP. y. DGG is the guarantor. Data collection: DNG-T, AIVO, DGG, VM, AT-Q and CAS-B. Data analysis and interpretation: DNG-T, AIVO, DGG, SA, AT-Q, CAS-B, REP, JA and GWP. Drafting the article: DNG-T, AIVO, DGG, AT-Q and CAS-B. Critical revision of the article and final approval of the version to be submitted: DNG-T, AIVO, DGG, AT-Q, CAS-B, REP, JA, SA, GWP and VM.

Funding This work was funded by Bloomberg Philanthropies (grant number 131761). This funding agency had no role in the preparation of this article.

Competing interests None declared.

Patient and public involvement Patients and/or the public were not involved in the design, conduct, reporting or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

References

¹ World Health organization. Noncommunicable diseases country profiles 2011. Geneva; 2011. Available: https://apps.who.int/iris/handle/10665/44704Thacke

² Hancioglu A, Arnold F. Measuring coverage in MNCH: tracking progress in health for women and children using DHS and Mics household surveys. PLoS Med 2013; 10: e1001391. doi:10.1371/journal.pmed.1001391

³ Phadnis R, Wickramasinghe C, Zevallos JC, et al. Leveraging mobile phone surveys during the COVID-19 pandemic in Ecuador and Sri Lanka: methods, Timeline and findings. PLoS One 2021; 16: e0250171. doi:10.1371/journal.pone.0250171

⁴ Song Y Phadnis R, Favaloro J, et al. Using mobile phone data collection tool, Surveda, for Noncommunicable disease surveillance in five Low- and middle-income countries. OJPHI 2020; 12. doi:10.5210/ojphi.v12i2.10574

⁵ Gibson DG, Pereira A, Farrenkopf BA, et al. Mobile phone surveys for collecting population-level estimates in low-and middle-income countries:A literature review. J Med Internet Res 2017; 19: e139. doi:10.2196/jmir.7428

⁶ Liu B, Brotherton JML, Shellard D, et al. Mobile phones are a viable option for surveying young Australian women: A comparison of two telephone survey methods. BMC Med Res Methodol 2011; 11: 159. doi:10.1186/1471-2288-11-159

⁷ Iachan R, Pierannunzi C, Healey K, et al. National weighting of data from the behavioral risk factor surveillance system (BRFSS). BMC Med Res Methodol 2016; 16: 155.: 155. doi:10.1186/s12874-016-0255-7

⁸ Brick JM, Edwards WS, Lee S. Sampling telephone numbers and adults, interview length, and weighting in the California health interview survey cell phone pilot study. Public Opinion Quarterly 2007; 71: 793–813. doi:10.1093/poq/nfm052

⁹ International Telecommunication Union. The World in 2016: ICT Facts and Figures. 2016.

¹⁰ Bank W. World Development Report 2016: Digital Dividends. Washington DC, 2016.

¹¹ GSM Association. La Economía Móvil en América Latina. 2020.

¹² The International Telecommunication Union. Statistics-individuals using the Internet [Internet], 2022. Available: https://www.itu.int/en/ITU-D/Statistics/Pages/stat/default.aspx

¹³ Data for Health. Bloomberg philanthropies, 2021. Available: https://www.bloomberg.org/public-health/strengthening-health-data/data-for-health/

¹⁴ World Health Organization (WHO). mHealth New horizons for health through mobile technologies. Switzerland, 2011.

¹⁵ Nagpal K, Mathur MR, Biswas A, et al. Who do phone surveys Miss, and how to reduce exclusion: recommendations from phone surveys in nine Indian States. BMJ Glob Health 2021; 6 (Suppl 5): e005610. doi:10.1136/bmjgh-2021-005610

¹⁶ Labrique A, Blynn E, Ahmed S, et al. Health surveys using mobile phones in developing countries: automated active strata monitoring and other statistical considerations for improving precision and reducing biases. J Med Internet Res n.d.; 19: e121. Available https://pubmed.ncbi.nlm.nih.gov/28476726/ doi:10.2196/jmir.7329

¹⁷ Greenleaf AR, Gibson DG, Khattar C, et al. Building the evidence base for remote data collection in low- and middle-income countries: comparing reliability and accuracy across survey modalities. J Med Internet Res 2017; 19: e140. doi:10.2196/jmir.7331

¹⁸ Sibai AM, Ghandour LA, Chaaban R, et al. Potential use of telephone surveys for non-communicable disease surveillance in developing countries: evidence from a national household survey in Lebanon. BMC Med Res Methodol 2016; 16: 1–8. doi:10.1186/s12874-016-0160-0

¹⁹ World Bank. Listening to LAC: using mobile phones for high frequency data collection. Final Report 2013.

²⁰ L’Engle K, Sefa E, Adimazoya EA, et al. n.d. Survey research with a random digit dial national mobile phone sample in ghana: methods and sample quality. PLoS ONE; 13: e0190902. doi:10.1371/journal.pone.0190902 Available: https://doi.org/10.1371/journal.pone.0190902 doi:10.1371/journal.pone.0190902

²¹ Greenleaf AR, Gadiaga A, Choi Y, et al. Automated and interviewer-administered mobile phone surveys in Burkina Faso: Sociodemographic differences among female mobile phone survey respondents and Nonrespondents. JMIR Mhealth Uhealth 2020; 8: e17891. doi:10.2196/17891

²² Greenleaf AR, Gadiaga A, Guiella G, et al. Comparability of modern contraceptive use estimates between a face-to-face survey and a cellphone survey among women in burkina faso [ Internet ]. PLoS ONE 2020; 15: e0231819. doi:10.1371/journal.pone.0231819 Available: https://doi.org/10.1371/journal.pone.0231819

²³ Larmarange J, Kassoum O, Kakou E. Feasibility and Representativeness of a random sample mobile phone survey in Côte D’Ivoire. Population (Paris) 2016; 71: 121.

²⁴ Ferreira AD, César CC, Malta DC, et al. Validity of data collected by telephone survey: A comparison of Vigitel 2008 and the "Saúde em Beagá" survey. Rev Bras Epidemiol 2011; 14 Suppl 1 (SUPPL. 1): 16–30.: S1415-790X2011000500003. doi:10.1590/s1415-790x2011000500003

²⁵ Moura EC, Claro RM, Bernal R, et al. Exequibilidade do USO de Entrevistas POR telephone Celular E POR Telefone Fixo no Monitoramento de Fatores de Risco E Proteção para Doenças Crónicas. Cad Saude Publica 2011; 27: 277–86.

²⁶ World Bank. Peru - the World Bank listening to LAC (L2L), 2022. Available: https://microdata.worldbank.org/index.php/catalog/2022

²⁷ Keusch F, Bähr S, Haas G-C, et al. Coverage error in data collection combining mobile surveys with passive measurement using Apps: data from a German national survey. Sociol Methods Res 2023; 52: 841–78. Available https://journals.sagepub.com/doi/10.1177/0049124120914924 doi:10.1177/0049124120914924

²⁸ Johns Hopkins Bloomberg School of Public Health. Bloomberg philanthropies data for health Initiative at JHSPH | Johns Hopkins | Bloomberg school of public health, 2022. Available: https://publichealth.jhu.edu/departments/international-health/research-and-practice/centers-and-research-groups/research-groups/bloomberg-philanthropies-data-for-health-initiative-at-jhsph

²⁹ Silva J, Solano D, Fernandez C, et al. Privacy preserving, protection of personal data, and big data: a review of the Colombia case. Procedia Computer Science 2019; 151: 1213–8. doi:10.1016/j.procs.2019.04.174

³⁰ Waksberg J. Sampling methods for random digit dialing. Journal of the American Statistical Association 1978; 73: 40–6. doi:10.1080/01621459.1978.10479995

³¹ The Stata. Using STATA’s random-number generators, part 2, drawing without replacement, 2022. Available: https://blog.stata.com/2012/08/03/using-statas-random-number-generators-part-2-drawing-without-replacement/

³² Torres-Quintero A, Vega A, Gibson DG, et al. Adaptation of a mobile phone health survey for risk factors for Noncommunicable diseases in Colombia: a qualitative study. Glob Health Action 2020; 13: 1809841. doi:10.1080/16549716.2020.1809841

³³ engageSPARK. engageSPARK - SMS, voice, Whatsapp, and Airtime campaigns in 200+ countries. 2022. Available: https://www.engagespark.com/

³⁴ WHO. Stepwise approach to Ncd risk factor surveillance (steps). 2021. Available: https://www.who.int/teams/noncommunicable-diseases/surveillance/systems-tools/steps

³⁵ OPS/OMS. Global adult tobacco survey (GATS), 2021. Available: https://www3.paho.org/hq/index.php?option=com_content&view=article&id=13431:gats&Itemid=42359&lang=es

³⁶ Ministerio de de Tecnologias de la informacion y las Comunidades. Telefonía Móvil [Internet], 2022. Available: https://colombiatic.mintic.gov.co/679/w3-propertyvalue-47274.html

³⁷ Departamento Administrativo Nacional de Estadística. Encuesta Nacional de Calidad de Vida (ECV), 2021. Available: https://www.dane.gov.co/index.php/estadisticas-por-tema/salud/calidad-de-vida-ecv/encuesta-nacional-de-calidad-de-vida-ecv-2021

³⁸ THE AMERICAN ASSOCIATION FOR PUBLIC OPINION RESEARCH. Standard Definitions Final Dispositions of Case Codes and Outcome Rates for Surveys, 2016: 1–81 Available. https://www.aapor.org/Standards-Ethics/Standard-Definitions-(1).aspx

³⁹ World Health Organization (WHO). Who steps surveillance manual, 2020. Available: https://cdn.who.int/media/docs/default-source/ncds/ncd-surveillance/steps/steps-manual.pdf?sfvrsn=c281673d_7

⁴⁰ Austin PC, Hux JE. A brief NOTE on overlapping confidence intervals. J Vasc Surg 2002; 36: 194–5. doi:10.1067/mva.2002.125015 Available: https://pubmed.ncbi.nlm.nih.gov/12096281/

⁴¹ Greenleaf AR. Cell phone surveys among women of reproductive age in Burkina Faso: identifying sources of error. Johns Hopkins Bloomberg School of Public Health, 2018.

⁴² Departamento Administrativo Nacional de Estadística-DANE. Empleo. DANA; 2021. Available: https://www.dane.gov.co/files/faqs/faq_ech.pdf

⁴³ Lau CQ, Cronberg A, Marks L. In search of the optimal mode for mobile phone surveys in developing countries. A comparison of IVR, SMS, and Cati in Nigeria. Surv Res Methods 2019; 13: 305–18. Available: https://ojs.ub.uni-konstanz.de/srm/article/view/7375

⁴⁴ Glass DC, Kelsall HL, Slegers C, et al. A telephone survey of factors affecting willingness to participate in health research surveys. BMC Public Health 2015; 15: 1–11. doi:10.1186/s12889-015-2350-9

⁴⁵ Rodriguez-Patarroyo M, Torres-Quintero A, Vecino-Ortiz AI, et al. Informed consent for mobile phone health surveys in Colombia: a qualitative study. Journal of Empirical Research on Human Research Ethics 2021; 16: 24–34. doi:10.1177/1556264620958606