Comparability of modern contraceptive use estimates between a face-to-face survey and a cellphone survey among women in Burkina Faso

Abstract

Introduction

The proliferation of cell phone ownership in Sub-Saharan Africa (SSA) presents the opportunity to collect public health indicators at a lower cost compared to face-to-face (FTF) surveys. This analysis assesses the equivalence of modern contraceptive prevalence estimates between a nationally representative FTF survey and a cell phone survey using random digit dialing (RDD) among women of reproductive age in Burkina Faso.

Methods

We analyzed data from two surveys conducted in Burkina Faso between December 2017 and May 2018. The FTF survey conducted by Performance Monitoring and Accountability (PMA2020) comprised a nationally representative sample of 3,556 women of reproductive age (15–49 years). The RDD survey was conducted using computer-assisted telephone interviewing and included 2,379 women of reproductive age.

Results

Compared to FTF respondents, women in the RDD sample were younger, were more likely to have a secondary degree and to speak French. RDD respondents were more likely to report using modern contraceptive use (40%) compared to FTF respondents (26%) and the difference remained unchanged after applying post-stratification weights to the RDD sample (39%). This difference surpassed the equivalence margin of 4%. The RDD sample also produced higher estimates of contraceptive use than the subsample of women who owned a phone in the FTF sample (32%). After adjusting for women’s sociodemographic factors, the odds of contraceptive use were 1.9 times higher (95% CI: 1.6–2.2) in the RDD survey compared to the FTF survey and 1.6 times higher (95% CI: 1.3–1.8) compared to FTF phone owners.

Conclusions

Modern contraceptive prevalence in Burkina Faso is over-estimated when using a cell phone RDD survey, even after adjusting for a number of sociodemographic factors. Further research should explore causes of differential estimates of modern contraceptive use by survey modes.

About the Authors:

Abigail R. Greenleaf

Contributed equally to this work with: Abigail R. Greenleaf, Saifuddin Ahmed, Caroline Moreau

Roles Data curation, Formal analysis, Investigation, Methodology, Resources, Software, Supervision, Validation, Visualization, Writing – original draft

* E-mail: [email protected]

Affiliation: Department of Population Family and Reproductive Health, Johns Hopkins University, Baltimore, MD, United States of America

ORCID logo http://orcid.org/0000-0002-8261-7025

Aliou Gadiaga

Roles Conceptualization, Investigation, Project administration, Resources

¶‡ These authors also contributed equally to this work.

Affiliation: Institut Supérieur des Sciences de la Population, University of Ouagadougou, Institut Supérieur des Sciences de la Population, Ouagadougou, Burkina Faso

Georges Guiella

Roles Conceptualization, Funding acquisition, Project administration

¶‡ These authors also contributed equally to this work.

Affiliation: Institut Supérieur des Sciences de la Population, University of Ouagadougou, Institut Supérieur des Sciences de la Population, Ouagadougou, Burkina Faso

Shani Turke

Roles Conceptualization, Investigation, Writing – review & editing

¶‡ These authors also contributed equally to this work.

Affiliation: Department of Population Family and Reproductive Health, Johns Hopkins University, Baltimore, MD, United States of America

Noelle Battle

Roles Conceptualization, Investigation, Writing – review & editing

¶‡ These authors also contributed equally to this work.

Affiliation: Department of Population Family and Reproductive Health, Johns Hopkins University, Baltimore, MD, United States of America

Saifuddin Ahmed

Contributed equally to this work with: Abigail R. Greenleaf, Saifuddin Ahmed, Caroline Moreau

Roles Conceptualization, Formal analysis, Funding acquisition, Supervision, Validation, Writing – original draft

Affiliation: Department of Population Family and Reproductive Health, Johns Hopkins University, Baltimore, MD, United States of America

Caroline Moreau

Contributed equally to this work with: Abigail R. Greenleaf, Saifuddin Ahmed, Caroline Moreau

Roles Conceptualization, Formal analysis, Funding acquisition, Methodology, Supervision, Writing – original draft

Affiliations Department of Population Family and Reproductive Health, Johns Hopkins University, Baltimore, MD, United States of America, Gender, Sexual and Reproductive Health, Centre for Research in Epidemiology and Population Health (CESP), Villejuif, France

Introduction

Largely due to inadequate vital registration systems, data from population-based surveys such as the Demographic and Health Surveys (DHS) and UNICEF’s Multiple Indicator Cluster Survey (MICS) serve as the primary data source in low and middle-income countries to monitor progress towards the Sustainable Development Goals and track changes in population health [1].

Although the DHS and MICS have provided invaluable data for the past 30 years, there are two main challenges associated with face-to-face (FTF) surveys: cost and extensive field time to attain the needed population coverage [2, 3]. The challenges of FTF surveys in low- and middle- income countries (LMIC) are contrasted with the three primary advantages of phone surveys identified in the 1990s that remain pertinent today: speed of data collection, cost efficiency, and ability to supervise interviewers throughout data collection [4]. Cost reduction is a major consideration in LMIC; one study conducted in Honduras showed a decrease from $40 USD per interview using FTF to $17 USD using a cellphone survey with interactive voice response (IVR) [5].

Cell phone ownership grew exponentially in Sub-Saharan Africa (SSA) in the early 2000s, driven mainly by urbanization and low cost of purchasing a cell phone and airtime [6, 7]. There are currently over 444 million cell phone subscriptions in SSA and over 634 million subscriptions are anticipated by 2025 [6]. SSA is expected to reach 50% mobile phone penetration, or the percent of unique mobile phone users within a specific population, in 2021 and 52% penetration by 2025 [6, 8]. In this context, demographers are considering the use of cell phone surveys to track national health indicators [9–11]. However, phone survey estimates need to be compared with FTF results before substituting data collection modes, to ensure phone survey estimates are valid. Threats to survey data quality and validity can be categorized into five types of survey error: frame and non-response errors, which relate to survey representation, and specification, measurement and processing errors, which relate to the quality of data collected. The manifestation of these errors differs by data collection mode and sampling approach [12]. Cell phone surveys are traditionally collected using one of three modes–computer assisted telephone interviews (CATI), IVR or short message service (SMS). The focus of this study is survey errors when comparing CATI to FTF surveys.

Few studies comparing cell phone surveys to FTF studies have been conducted in LMIC [13, 14]. Those that have compared cell phone and FTF survey results used a census or a FTF reference population-based survey such as DHS, rather than a concurrent study with the same questions in the cell phone survey. Two recent random digit dial (RDD) studies, an IVR survey in Ghana about general health [10] and a CATI survey in Cote d’Ivoire about HIV risk behaviors [15] compared their population with the sociodemographic composition of recent national FTF surveys. In Ghana, two-thirds of RDD respondents were male and more than half were 15–24 years of age whereas according to the 2017 census, only 48% of the population is male and 30% ages 15–24 [10]. In the Cote d’Ivoire study, the composition of the RDD sample was different than the DHS sample distribution, with over-representation of urban individuals and males [15].

The aim of this analysis is to examine whether an RDD cell phone survey produces equivalent estimates of modern contraceptive use to a reference FTF survey conducted in the general female population after applying post-stratification weights to account for RDD sample distortion.

Materials and methods

Methods

The PMA2020 FTF survey has exempt status (IRB #00000287, exempt as “public health practice”, defined by DHHS regulations 45 CFR 46.102), as determined by John Hopkins’ IRB. Approval was granted in July 2014 by the Comité d’éthique pour la recherche en santé (IRB #2014-7-81). Consent was obtained orally. For the phone survey, ethical approval was granted by the IRB at the Johns Hopkins Bloomberg School of Public Health (IRB No. 00007961). The ISSP team submitted a protocol to their ethical committee for the RDD survey (IRB No. 2018-3-036).

We used two datasets for this analysis. The first dataset is the Burkina Faso PMA2020 Round 5 (R5) FTF survey, designed to track key family planning indicators under the Family Planning 2020 initiative [16]. PMA2020 R5 implemented a two-stage stratified cluster design to select a national sample of women of reproductive age in Burkina Faso. This design starts with a probability proportional to size selection of 83 geographic clusters stratified by urban and rural areas followed by a random selection of about 35 households within each sample cluster. Detailed sampling methods and procedures are available in a previous publication [17]. PMA2020 R5 Burkina Faso was conducted between November 20, 2017- January 20, 2018 and included a total of 2,811 households (98.5% response rate) and 3,659 females (97.8% response rate) [18].

Data collection was performed by a network of trained female interviewers who conducted FTF interviews with members of selected households and with all eligible females 15–49 years from the selected households. Interviewers recorded responses directly on cell phones and uploaded the data when cellular network was available.

The second dataset was the RDD CATI survey, which took place four months after PMA2020 R5 survey. To conduct the RDD CATI survey, PMA2020 Baltimore and Institute Supérieur des Sciences de la Population (ISSP) staff trained five call center supervisors for three days. Supervisors had at least a Bachelor degree and all had previous survey research experience. Twenty-five interviewers were trained for four days and 20 interviewers were retained for data collection. All interviewers spoke French and at least one of four local languages, except for one interviewer who spoke exclusively French. Data collection was led by ISSP and took place from April 13 to May 17, 2018 in a call center located in an NGO building in Ouagadougou. Interviewers worked in two shifts, the first group from approximately 12–4 pm, the second group from 4–8 pm.

CATI was chosen over IVR based on a previous PMA2020 study which found higher response rates and less sample distortion in a CATI sample than a IVR sample [19]. RDD sampling was chosen because the mobile network operators in Burkina Faso do not share lists of valid phone numbers. To create the phone numbers, Viamo, an international mobile technology survey company, [20] used a list of the 25 existing prefixes provided from the three mobile network operators in Burkina Faso. Viamo then randomly generated the remaining six digits to create a list of phone numbers.

Results from the aforementioned follow-up phone survey in Burkina Faso also informed the sampling strategy of the RDD cross-sectional survey, which included quotas by age and area of residence to improve representativeness and reduce design effects [21]. We established the quota groups by comparing respondents and non-respondents from a phone follow-up study of women who participated in the previous national FTF survey conducted by PMA2020 in 2017 (Round 4). Women who responded to the follow-up phone interview were more likely to live in an urban setting, had a higher level of education and were more likely to be over the age of 20. We defined quota groups based on residence and age for ease of implementation and used the distribution of these characteristics in the PMA2020 Round 4 female sample to calculate the percent of women in each of the four quota groups (see Table 1).

[Figure omitted. See PDF.]

Table 1. The number of completed surveys needed by quota group.

https://doi.org/10.1371/journal.pone.0231819.t001

In the absence of active phone number lists, a significant proportion of phone numbers generated by RDD are invalid. A pilot study conducted in February 2018 estimated that 58% of RDD generated numbers were not assigned to a subscriber, or were invalid. We defined a phone number as valid if the outcome of the call, as recorded by any of the three mobile phone companies in Burkina Faso had one of three following outcomes (1- No Answer, 2- Normal Clearing or 3- Normal Unspecified) [22].

To improve RDD efficiency in light of the pilot results, we sent out an IVR “validation/pre-notification” phone call to all generated RDD numbers to eliminate invalid phone numbers. Calls were placed one to seven days before being called by an interviewer, with the following message in Moore: “Thank you for responding to our call. We will call you this week for a study. Please, pick up the phone when we call. Have a good day!”

For the first seven days of data collection, we called phone numbers that were identified with any one of the three call statuses during the validation calls: 1) No Answer, 2) Normal Clearing or 3) Normal Unspecified. However, after the first seven days of data collection, due to an insufficient number of completed interviews per day, we narrowed the definition of a valid phone number and no longer included phone numbers marked as ‘No Answer’ during the validation call. The narrower definition meant that interviewers were only calling phone numbers that were answered during the validation calls.

When a female respondent answered the call, she could either complete the survey at that time or be called back up to six times. If a respondent answered that did not speak the same language as the interviewer, she would be called back the same day by an interviewer that spoke the respondent’s language. If a respondent explicitly refused the study, she was not called back. Men were not allowed to pass the phone to a female in their household. Women that completed the survey were sent the equivalent of $1 US dollar phone credit the day after completing the interview.

Questionnaires

The PMA2020 Burkina Faso FTF female questionnaire includes standardized questions that are largely based on the DHS. The questionnaire typically takes less than 40 minutes to complete [23]. The female survey collects sociodemographic information including cell phone ownership, and reproductive health measures, including current contraceptive use as described in the next section [17].

The RDD questionnaire was limited to 19 questions that allowed comparison of modern contraceptive use estimates with the FTF survey. Four questions helped establish the eligibility of the respondent, followed by 5–6 demographic questions, 5 questions assessing awareness of contraceptive methods, and 3–4 questions on contraceptive use. The RDD questions were identical to the FTF questions with a few adaptations for phone administration. The RDD survey was available in French and four local languages: Dioula, Fulfulde, Gourmantchema and Moore. The questionnaire is available as a supporting file.

Measures—Defining call outcomes for RDD survey using AAPOR final disposition codes

We used the 9th edition American Association for Political Opinion Research (AAPOR) final disposition case codes to classify call outcomes. [24] The invalid phone numbers identified during the screening process were not assigned a disposition code. The 13 disposition codes were divided into four groups (Not Eligible; Unknown Eligibility–non-interview; Eligible–non-interview; Interview) and are detailed in Table 2.

[Figure omitted. See PDF.]

Table 2. Final disposition codes for RDD survey, among valid phone numbers.

https://doi.org/10.1371/journal.pone.0231819.t002

Non-eligible respondents were categorized into four codes. Respondents were ineligible if they were male, or were female but >49 years and <15 years or did not speak one of the five survey languages. The fourth group consisted of women who spoke one of the survey languages and were between the ages of 15–49 but were ineligible due to quota restrictions.

Unknown eligibility was captured in four disposition codes including calls that were never picked up that were classified as “No answer” and calls answered by a voice mailbox, classified as “Telephone answering device”. Respondents who answered but for whom age, gender or area of residence was not known were classified as “No screener completed”. Finally, respondents who spoke one of the five survey languages but who did not speak the same language as the interviewer and were not reached during subsequent attempts were classified as “Other (language not matched with interviewer)”.

The next group “eligible, not interviewed”, was divided into three codes and consisted of women ages 15–49 who spoke one of the five survey languages and were not excluded due to quota restrictions. The first was refusal before consent. The second was refusal at consent and the third was “break-off”, corresponding to a consenting respondent who completed less than 50% of questions.

The final group included women who completed the interview, classified as a partial interview when 50–80% of questions were answered and a complete interview when 80% of questions or more were answered.

Based on the aforementioned call disposition codes, we created four key call outcome indicators aligned with AAPOR standards–response rate, cooperation rate, refusal rate and contact rate (Table 3). To improve on the specificity of these outcome measures, AAPOR also recommends calculating rates that exclude an estimated number of unknown eligibility phone numbers from the denominator for response and contact rates. Based on pilot data collected in February 2018, we estimated that 20% of calls with unknown eligibility would in fact include an eligible woman. Applying this correction, we defined corrected response rates (Response rates 3 & 4) and a corrected contact rate (Contact rate 2) excluding 80% of attempted calls with unknown eligibility from the denominators.

[Figure omitted. See PDF.]

Table 3. Call outcome rates for RDD survey based on final disposition distributions.

https://doi.org/10.1371/journal.pone.0231819.t003

Measures—Independent & dependent variables

The outcome of interest was a binary measure of modern contraceptive use, based on two questions that were asked identically in the two surveys. The first question asked whether the respondent or her partner was currently using a form of contraception (“Are you or your partner currently doing something or using any method to delay or avoid getting pregnant?”). If the respondent responded affirmatively, she was asked to specify the type of method used (“Which method or methods are you using?”). If the respondent identified a modern method (as specified below), she was classified as a user of modern contraception.

Traditionally, measures of modern contraceptive use include all modern contraceptives available in a country. In Burkina Faso, 12 modern contraceptive methods are available: male and female sterilization, implant, Intra Uterine Device (IUD), injectables, pill, emergency contraception, male condom, female condom, diaphragm, foam/jelly and the lactation amenorrhea method. However, the RDD survey only collected data on the five most common methods used in Burkina Faso, based on PMA2020 R5 estimates (covering 98.8% of modern method use): implants, injectables, pills, condoms and IUDs. Thus we limited the definition of modern contraceptive use to these five methods for both the FTF and RDD surveys in this study [18]. In addition, we constructed a five-category indicator of method mix, including the following contraceptive methods: implant, IUD, injectables, pills, and condoms.

We selected independent measures related to modern contraceptive use as cited in the literature as well as factors related to phone ownership, that were collected in both surveys to conduct our analysis [25–27]. The independent variable of interest was mode of data collection. The FTF survey was the reference group; RDD the comparison group. Women’s sociodemographic characteristics included age grouped in 5-year increments, current union status (in union–i.e., currently married or living with a partner vs. not in union), parity (ever had a birth yes-no). residential area (urban vs. rural), educational attainment (none, primary, or secondary and higher) and language of survey (Moore, French, Dioula, Fulfulde, or Gourmantchema).

Missing data.

The RDD data had item non-response due to internet outages at the call center (electricity brown-outs).We used the hot deck method [28] to impute missing values for three variables: age (43 missing values, 1.8%), residence (10 missing values, 0.4%) and education (10 missing values, 0.4%), assuming data were missing completely at random.

Analysis

We first describe RDD call outcomes to evaluate response rate, cooperation rate, refusal rate and contact rate. To examine distributions of the aforementioned independent variables, we conducted univariate analysis, looking at patterns of response in the FTF and RDD samples. Comparisons were made using RDD respondents who completed at least 50% of the questionnaire (N = 2,379) and all women in the FTF sample who represent the target population (n = 3,659), as well as a subsample of FTF respondents who own a cell phone who represent the sample frame of the RDD survey phone (n = 2,027). The R5 data were adjusted for sampling weights, which address disproportionate two-stage cluster sampling and non-response rates. [29] To account for sample distortion based on age, area of residence and level of education, we created post-stratification weights for the RDD sample, using R5 as the reference population.

We examined the equivalence of modern contraceptive use prevalence estimates in the weighted RDD and FTF samples, setting an equivalence margin to +/- 4%. The null hypothesis assumed a difference of more than 4% between the two survey estimates. We report the 90% confidence interval for the difference in point estimates, which simulates performing two one-sided tests. We also report a p-value from an adjusted Wald test [30].

We further compared modern contraceptive use by mode of data collection by conducting multivariable logistic regression adjusting for additional sociodemographic factors. We first assessed bi-variate relationships between each co-variate and modern contraceptive use among the pooled FTF-RDD data. We then conducted multivariable logistic regression to assess the odds of modern contraceptive use by survey mode, adjusting for covariates. We also compared the RDD and FTF phone owner sample using multivariable logistic regression. Analysis was performed using weighted RDD and FTF data. We conducted analyses in Stata version 15 (StataCorp 2017) and determined statistical significance using an alpha of less than 0.05.

Results

Call outcomes

Approximately 202,295 unique phone numbers were screened of which 21% were deemed valid. The 42,726 valid phone numbers were called by interviewers over the course of a month and constitute our sample size for all RDD survey response outcome analyses. Overall, 45% of the 42,726 valid phone calls were categorized as ineligible, mostly due to a man answering the call (36%) and approximately 6% of calls were non-eligible because of quota restrictions (Table 2). Another 49% of calls fell in the “unknown eligibility” category, the majority of which were call no answer (43%). Less than 1% of eligible women did not complete the survey.

Altogether, 2,379 women completed 50% or more of the survey questions and comprised our RDD study population for contraceptive prevalence analysis (n = 54 were partial completers).

Survey outcome rates

The minimum response rate (Response rate 1), which includes only interviews where more than 80% of questions were answered and includes all attempted calls in the denominator was 9.9% (Table 3). This percentage rose to 10.1% when including partial survey completion (50–80%). When excluding 80% of unknown eligibility calls from the denominator, response rates increased substantially, to 68% when counting complete interviews in the numerator (Response rate 3) and 70% when also counting partial interviews (Response rate 4).

The cooperation rate, which includes only eligible calls in the denominator, was 94.7% when counting complete interviews (Cooperation rate 1), and 97.1% when also counting partial interviews (Cooperation rate 2). The refusal rate was 3.1%.

Comparison of FTF & RDD samples

Among the 3,659 women who completed PMA2020 R5 FTF survey, 78% were rural, 72% were married and 75% had given birth (Table 4). The average age of the respondents was 28.6 years (standard error = 0.24). Close to two-thirds of women (63%) had never been to school; thus only one in five had a secondary education or higher. Interviews were most often conducted in Moore (44%) while 10% of women completed the survey in French and 20% completed the interview in another language. 46% of women (N = 1,671) in the FTF survey indicated that they owned a phone. A greater percent of female phone owners lived in an urban area (35%) than the full FTF sample (20%). 30% of FTF phone owners had secondary education or higher and more than half completed the survey in Moore (53%) while 19% completed the survey in French.

[Figure omitted. See PDF.]

Table 4. Characteristics of women by survey mode and cell phone ownership.

https://doi.org/10.1371/journal.pone.0231819.t004

Compared to the target population of FTF respondents, women in the RDD sample were younger, a greater percent had a secondary degree and responded in French. Distribution by region also differed with greater representation of women in the Center region (where the capital, Ouagadougou, is located) in the RDD sample. Characteristics of women in the RDD sample more closely reflected the sample FTF phone owners. After applying post-stratification weights, age and educational differences disappeared but regional differences increased.

Contraceptive use by survey mode

Contraceptive use among FTF, phone owner and RDD weighted samples.

A quarter of women (26% (95% Confidence Interval (CI): 22.7%– 29.6%)) in the FTF survey reported contraceptive use versus 40.2% (95% CI: 38.2%– 42.4%) in the unweighted RDD sample. The post-stratification weights had little impact on the RDD estimate: 38.7% contraceptive use (95% CI: 36.7%– 40.8%). The 12.7% difference was greater than the 4% equivalence margin leading to conclude RDD and FTF estimates are not equivalent. The difference between RDD and FTF phone owners was smaller but remained substantial (40.2% to 31.7%).

Method mix differed by mode of data collection. Altogether 46% of contraceptive users in the FTF sample used a short-acting method (injectables, pills, or condoms) and 54% used long acting reversible contraception (LARC), including IUD or implant (Fig 1) compared to 59% and 41% in the RDD sample respectively. The RDD sample estimates more closely resembled the phone FTF method mix of 55% short-acting method and 45% LARC.

[Figure omitted. See PDF.]

Fig 1. Modern method mix among current users among full FTF sample, FTF cell phone owners, and RDD respondents* (%).

*The most effective method currently used, if multiple methods were reported. % estimates are adjusted for sampling weight.

https://doi.org/10.1371/journal.pone.0231819.g001

After adjusting for confounding covariates, results from the multivariable logistic regression model indicated that women in the weighted RDD sample had almost twice the odds of reporting modern contraceptive use compared to women in the FTF sample (Odds Ratio (OR): 1.9, 95% CI: 1.6–2.1) (Table 5). Significant differences also remained when comparing RDD and FTF phone owners (OR: 1.6 95% CI: 1.3–1.8).

[Figure omitted. See PDF.]

Table 5. Odds of reporting modern contraceptive use by women’s characteristics and survey mode, using full FTF sample and FTF phone owner sample.

https://doi.org/10.1371/journal.pone.0231819.t005

Discussion

This study assessed the feasibility of conducting RDD phone surveys for monitoring family planning metrics in Burkina Faso and the associated bias in estimating national modern contraceptive prevalence. We found that six percent of valid phone numbers resulted in a complete interview. The RDD sample resulted in 14% point over-estimation of modern contraceptive use, which remained substantial and significant, after post-stratification adjustments of the RDD sample and further adjustment for confounding (OR: 1.9, 95% CI: 1.6–2.2). Whereas previous RDD studies in LMIC have compared their phone-based estimates with non-concurrent studies that were managed by exterior survey organizations, our study used the same questionnaire for both the FTF and RDD study, within a reasonable time frame, allowing us to more confidently compare the outcomes. This study is among the first in SSA to offer this direct comparison of modes and to look specifically at women.

As expected from previous studies on phone ownership [7, 31] and from our previous work comparing respondents to non-respondents among phone owners [19], the RDD sample was distorted in favor of more educated women. In addition, the RDD sample over-represented women speaking Moore and living in Ouagadougou. The application of quotas, which excluded only six percent of eligible respondents, helped reduce age and residence distortions.

We nonetheless applied post-stratification weights to address distortion according to educational, age and urban/rural residence. The weights had little impact on the difference in contraceptive prevalence rates between RDD and FTF (14.2% point difference using unweighted RDD estimates versus 12.7% point difference using weighted RDD estimates). A trio of articles from Brazil, all using data from VIGITEL, an annual and continuous phone survey in the 26 state capitals monitoring a host of non-communicable diseases, compared RDD samples with concurrent FTF samples [32–34]. Two of the articles used post-stratification weights and reported the weights reduced the difference between phone survey estimates and FTF survey estimates [32, 33].

To better understand the reasons for the RDD over-estimation of modern contraceptive use in our study, we assessed the impact of frame bias by comparing RDD estimates to FTF phone owner estimates. The difference between the RDD unweighted sample and the FTF subsample of phone owners was significantly reduced (8.5% point difference) compared to the 14.2% difference observed with the full FTF sample, suggesting frame bias may contribute for more than half of the over-estimation of modern contraceptive prevalence in the RDD sample. These results are in line with previous analyses of female phone owners in Burkina Faso, showing greater use of contraception among cell phone owners compared to non-owners [26]. They are also consistent with the findings of one of the aforementioned Brazilian studies showing little difference between RDD estimates and FTF estimates among phone owners [32].

Nonetheless, the difference in estimates of modern contraceptive use between FTF phone owners and RDD respondents (unweighted) suggests additional sources of bias. The bias is likely caused by differences in population composition (non-response) or differences in survey response by mode of data collection (measurement error). The RDD sample showed greater representation of women with secondary education or higher, women under 34 and French- and Moore-speaking women compared to the FTF sample. However, after adjusting for these factors in the multivariable analysis, the odds of contraceptive use remained significantly higher in the unweighted RDD sample compared to the FTF phone owner sample (OR: 1.6 95% CI: 1.3–1.8). Unfortunately, we were not able to collect community-level variables that are frequently associated with contraceptive use, such as quality of and access to health services, cultural norms and fertility practices.

This study has a number of strengths. It is among the first in SSA to compare health estimates from concurrent surveys using different modes of data collection. As such, it provides a firsthand investigation of the opportunities and challenges of using phone surveys in a context of rapid demographic change. The use of similar questionnaires limited measurement error while the almost concurrent timing of the surveys also improved comparability of survey estimates. The sample sizes were large enough to allow equivalence testing of modern contraceptive use prevalence with a relatively low margin of equivalence of 4%. Furthermore, this study is among the first in LMIC to use CATI during RDD; traditionally IVR is used with RDD sampling in LMIC. The results, which show that cell phone surveys among women in West Africa do not create estimates comparable to a concurrent FTF survey, provide a warning to those who are eager to replace FTF surveys with cell phone surveys.

The study also has a number of limitations, including the small number of demographic variables available in the RDD sample, which limited our capacity to systematically investigate differences between the RDD sample and the FTF target population. As a result, post-stratification weights were limited to a few demographics, leaving out potential unobserved differences that could better explain the difference between RDD and FTF modern contraceptive use estimates. Although we used multiple weights to correct for the difference, the modern contraceptive use estimate from the RDD survey remained higher than the estimate from the FTF survey. The study could not explain all the factors that may cause the differences in estimates between two surveys. Another limitation was not allowing men to pass the phone to a female respondent as 84% of households own a phone but fewer females own their own phone. Although the decision to not pass the phone was made to reduce the complexity of weighting (i.e. avoiding weighting the sample for women in a household that were not sampled), the volume of calls we had to place due to men answering the majority of calls (36.4% (N = 15,570) of calls picked up by men) made the project very challenging to implement and may have impacted data quality.

Conclusion

An RDD survey in Burkina Faso did not yield an estimate of modern contraceptive use that was equivalent to FTF reference estimate, even after applying post-stratification weights. Over-estimation of modern contraceptive use in our RDD survey originated both from a truncated sample frame, excluding non-cell phone owners, and also from non-response and measurement error, which need further examination as cell phone ownership expands in the SSA.

Supporting information

[Figure omitted. See PDF.]

S1 Data.

https://doi.org/10.1371/journal.pone.0231819.s001

(DOCX)

Acknowledgments

The authors would like to acknowledge three groups of people that made this research possible: first, the female survey respondents, who were willing to participate in a new form of data collection. Second, the interviewers and call center supervisors who showed great perseverance and dedication during a challenging data collection. Finally, we would like to show our appreciation for Viamo, the organization that created the software for data collection.

Citation: Greenleaf AR, Gadiaga A, Guiella G, Turke S, Battle N, Ahmed S, et al. (2020) Comparability of modern contraceptive use estimates between a face-to-face survey and a cellphone survey among women in Burkina Faso. PLoS ONE 15(5): e0231819. https://doi.org/10.1371/journal.pone.0231819

References

1. Choi Y, Short Fabic M. Monitoring Progress in Equality for the Sustainable Development Goals: A Case Study of Meeting Demand for Family Planning. Glob Health Sci Pract. 2018;6(2):390–401. Epub 2018/07/01. pmid:29959277

2. Chandy L, Zhang C. Stuffing data gaps with dollars: What will it cost to close the data deficit in poor countries?: Brookings Institute; 2017 [cited 2018 5 Sept]. https://www.brookings.edu/opinions/stuffing-data-gaps-with-dollars-what-will-it-cost-to-close-the-data-deficit-in-poor-countries/.

3. Greenleaf AR, Vogel L. Interactive Voice Response Technology for Data Collection in Sub-Saharan Africa. Toronto, Canada: Viamo, 2018.

4. Kempf AM, Remington PL. New Challenges for Telephone Survey Research in the Twenty-First Century. Annual Review of Public Health. 2007;28:14. pmid:17094769

5. Ballivian A, Azevedo JP, Durbin W, Rios J, Godoy J, Borisova C, et al. Listening to LAC: Using Mobile Phones for High Frequency Data Collection. Washington, DC: The World Bank, Poverty EaGT, Latin America and Carribean region; 2013.

6. GSMA. The Mobile Economy Sub-Saharan Africa 2018. London: 2018.

7. Poushter J, Oates R. Cell Phones in Africa: Communication Lifeline. Washington, DC: Pew Research Center, 2015.

8. Intelligence G. Measuring mobile penetration. 5 New Street Square, New Fetter Lane, London EC4A 3BF: GSMA, 2014.

9. Hyder AA, Wosu AC, Gibson DG, Labrique AB, Ali J, Pariyo GW. Noncommunicable Disease Risk Factors and Mobile Phones: A Proposed Research Agenda. J Med Internet Res. 2017;19(5):e133. Epub 2017/05/10. pmid:28476722

10. L’Engle K, Sefa E, Adimazoya EA, Yartey E, Lenzi R, Tarpo C, et al. Survey research with a random digit dial national mobile phone sample in Ghana: Methods and sample quality. PLoS One. 2018;13(1):e0190902. pmid:29351349.

11. Gibson DG, Pariyo GW, Wosu AC, Greenleaf AR, Ali J, Ahmed S, et al. Evaluation of Mechanisms to Improve Performance of Mobile Phone Surveys in Low- and Middle-Income Countries: Research Protocol. JMIR Res Protoc. 2017;6(5):e81. pmid:28476729.

12. Biemer PP. Total Survey Error: Design, Implementation, and Evaluation. Public Opinion Quarterly. 2011;74(5):817–48.

13. Greenleaf AR, Gibson DG, Khattar C, Labrique AB, Pariyo GW. Building the Evidence Base for Remote Data Collection in Low- and Middle-Income Countries: Comparing Reliability and Accuracy Across Survey Modalities. J Med Internet Res. 2017;19(5):e140. Epub 2017/05/10. pmid:28476728

14. Gibson DG, Pereira A, Farrenkopf BA, Labrique AB, Pariyo GW, Hyder AA. Mobile Phone Surveys for Collecting Population-Level Estimates in Low- and Middle-Income Countries: A Literature Review. J Med Internet Res. 2017;19(5):e139. Epub 2017/05/10. pmid:28476725

15. Larmarange J, Kassoum O, Kakou E, Fradier Y, Lazare S. Feasibility and Representativeness of a Random Sample Mobile Phone Survey in Côte d’Ivoire. Population. 2016;71(1):10.

16. Scoggins S, Bremner J. FP2020: Catalyzing Collaboration 2017–2018 Family Planning 2020: 2018 Contract No.: Oct 12.

17. Zimmerman L, Olson H, Group PMAPI, Tsui AO, Radloff S. PMA2020: Rapid Turn-Around Survey Data to Monitor Family Planning Service and Practice in Ten Countries. Stud Fam Plann. 2017;48(3):293–303. Epub 2017/09/09. pmid:28885679.

18. PMA2020. PMA2020—Burkina Faso- Round 5. Baltimore, MD: Johns Hopkins Univeristy 2018 March 10 2018. Report No.

19. Greenleaf AR, Gadiaga A, Turke S, Battle N, Ahmed S, Moreau C, et al. Comparison of remote data collection modes to monitor family planning progress in Burkina Faso: representativeness, data quality, and cost Baltimore, MD: Bill & Melinda Gates Institute for Population and Reproductive Health, Johns Hopkins Bloomberg School of Public Health, 2019.

20. VotoMobile. What is Voto? 2013 [cited 2016 Nov 20]. https://www.votomobile.org/files/WhatIsVOTO7.pdf].

21. Labrique A, Blynn E, Ahmed S, Gibson D, Pariyo G, Hyder AA. Health Surveys Using Mobile Phones in Developing Countries: Automated Active Strata Monitoring and Other Statistical Considerations for Improving Precision and Reducing Biases. J Med Internet Res. 2017;19(5):e121. pmid:28476726

22. Areski B. Hang Up Cause Code Table FreeSwitch; 2014 [cited 2018 June 12]. https://freeswitch.org/confluence/plugins/servlet/mobile#content/view/3964945.

23. Jiang S. Use Item-Level Timestamp Paradata for Multi-Country Comparison. Comparative Survey Design and Implementation Workshop; Limerick, Ireland2018.

24. Research TAAfPO. Standard Definitions: Final Dispositions of Case Codes. Washington, DC The American Association for Public Opinion Research 2016.

25. Maiga A, Hounton S, Amouzou A, Akinyemi A, Shiferaw S, Baya B, et al. Trends and patterns of modern contraceptive use and relationships with high-risk births and child mortality in Burkina Faso. Glob Health Action. 2015;8:29736. Epub 2015/11/13. pmid:26562142

26. Greenleaf AR, Ahmed S, Moreau C, Guiella G, Choi Y. Cell phone ownership and modern contraceptive use in Burkina Faso: implications for research and interventions using mobile technology. Contraception. 2019;99(3):170–4. Epub 2018/11/24. pmid:30468721

27. Cecatti JG, Camargo RP, P R.D., Giavarotti T, Souza JP, Parpinelli MA, et al. Computer-assisted telephone interviewing (CATI): using the telephone for obtaining information on reproductive healt. Cad Saúde Pública. 2011;27(9):8.

28. Andridge RR, Little RJ. A Review of Hot Deck Imputation for Survey Non-response. Int Stat Rev. 2010;78(1):40–64. Epub 2010/04/01. pmid:21743766

29. Zimmerman L, Ahmed S. PMA2020 Weight Creation Memo—v4 Baltimore, MD2017 [cited 2018 June 11]. https://www.pma2020.org/sites/default/files/PMA2020_Creation%20of%20household%20and%20female%20weights_Generic_v4_2017.05.18_LZ-web.pdf.

30. Walker E, Nowacki AS. Understanding equivalence and noninferiority testing. J Gen Intern Med. 2011;26(2):192–6. Epub 2010/09/22. pmid:20857339

31. GSMA. Bridging the gender gap: Mobile access and usage in low and middle-income countries. 2015.

32. Ferreira A, Cesar C, de Souza AD, Proietti C, Bernal F, Caiaffa R. Validity of data collected by telephone survey: a comparison of VIGITEL 2008 and the ‘Saúde em Beagá’ survey. Rev Bras Epidemiol. 2011;14(1):16–30.

33. Moura EC, Moreira CR, Bernal R, Ribeiro J, Malta DC, Neto OM. A feasibility study of cell phone and landline phone interviews for monitoring of risk and protection factors for chronic diseases in Brazil. Cad Saúde Pública. 2011;27(2):10.

34. Francisco PMSB, de Azevedo Barros MB, Segri NJ, Porto Alves MCG, Cesar CLG, Malta DC. Comparison of estimates for the self-reported chronic conditions among household survey and telephone survey–Campinas (SP), Brazil. Rev Bras Epidemiol. 2011;14(3):5–15.