Digital Assessment Tools Using Animation Features to Quantify Alcohol Consumption: Systematic App Store and Literature Review

Abstract

Background: Accurate and user-friendly assessment tools for quantifying alcohol consumption are a prerequisite for effective interventions to reduce alcohol-related harm. Digital assessment tools (DATs) that allow the description of consumed alcoholic drinks through animation features may facilitate more accurate reporting than conventional approaches.

Objective: This review aims to identify and characterize freely available DATs in English or Russian that use animation features to support the quantitative assessment of alcohol consumption (alcohol DATs) and determine the extent to which such tools have been scientifically evaluated in terms of feasibility, acceptability, and validity.

Methods: Systematic English and Russian searches were conducted in iOS and Android app stores and via the Google search engine. Information on the background and content of eligible DATs was obtained from app store descriptions, websites, and test completions. A systematic literature review was conducted in Embase, MEDLINE, PsycINFO, and Web of Science to identify English-language studies reporting the feasibility, acceptability, and validity of animation-using alcohol DATs. Where possible, the evaluated DATs were accessed and assessed. Owing to the high heterogeneity of study designs, results were synthesized narratively.

Results: We identified 22 eligible alcohol DATs in English, 3 (14%) of which were also available in Russian. More than 95% (21/22) of tools allowed the choice of a beverage type from a visually displayed selection. In addition, 36% (8/22) of tools enabled the choice of a drinking vessel. Only 9% (2/22) of tools allowed the simulated interactive pouring of a drink. For none of the tools published evaluation studies were identified in the literature review. The systematic literature review identified 5 exploratory studies evaluating the feasibility, acceptability, and validity of 4 animation-using alcohol DATs, 1 (25%) of which was available in the searched app stores. The evaluated tools reached moderate to high scores on user rating scales and showed fair to high convergent validity when compared with established assessment methods.

Conclusions: Animation-using alcohol DATs are available in app stores and on the web. However, they often use nondynamic features and lack scientific background information. Explorative study data suggest that such tools might enable the user-friendly and valid assessment of alcohol consumption and could thus serve as a building block in the reduction of alcohol-attributable health burden worldwide.

Trial Registration: PROSPERO International Prospective Register of Systematic Reviews CRD42020172825; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42020172825

Introduction

Background

Alcohol-related injuries and diseases are major causes of morbidity and mortality worldwide, although, at least in theory, they are fully preventable [1]. The well-directed implementation of monitoring, prevention, and treatment programs requires accurate assessment tools to quantify the users’ alcohol consumption. To date, consumption assessments are generally based on standardized self-report questionnaires or brief interviews. At the population level, they form the foundation for public health monitoring, quantification of alcohol-attributable harm, and evaluation of alcohol policies. At the individual level, they constitute the cornerstone of effective harm reduction strategies such as screening and brief intervention (SBI) programs. SBI programs link the routine administration of a screening tool to identify harmful or hazardous drinking, often a questionnaire, to a tailored brief intervention, most commonly comprising a short motivational interview or structured advice [2]. They have been shown to be highly effective in reducing excessive drinking among adults [3,4] and are recommended in national and international policy guidelines for reducing alcohol-attributable harm [5,6]. However, the implementation of SBI programs in public health systems remains low [7,8]. Relevant implementation barriers include a perceived lack of knowledge or skills among health care professionals and environmental context factors such as time restrictions and limited resources [9].

Although the measures used in epidemiological surveys differ between countries and regions [10-12], to date, most of them ultimately require the counting of standard drinks consumed. Routine screening tools for primary care such as the Alcohol Use Disorders Identification Test (AUDIT) developed by the World Health Organization [13] and its abbreviated form AUDIT–Consumption (AUDIT-C) [14] also rely on this concept. The standard drink, defined as a beverage volume containing a fixed amount of pure alcohol, facilitates the comparison and assessment of alcohol quantities across different beverage types with varying alcohol content. However, this concept is problematic for 2 main reasons. First, standard drink sizes differ considerably between countries, cultures, and settings, with national definitions varying even within Europe from 8 g of pure alcohol in the United Kingdom to 20 g in Austria [15]. In fact, the majority of countries worldwide do not have an official definition [15]. In addition, relevant AUDIT items are often not adapted to account for differing national standard drink sizes, as required in the AUDIT manual [13,16]. Second, even in countries where the standard drink concept is officially used to standardize the size of retail alcohol, consumers are often not acquainted with the concept and many are unable to convert their consumption correctly [17,18]. For instance, when asked to pour their usual drink and subsequently estimate the number of standard drinks it contained, primary health care patients in the United Kingdom over- or underestimated their actual drink size by at least 0.5 standard drinks in more than half of the cases [19]. In a study conducted among health care professionals in the United States, < 20% of the interviewed clinicians could correctly state the volume of a standard drink of liquor [20]. Alongside other known biasing factors, such as memory and social desirability bias or underreporting because of alcohol-related stigma, this might contribute to the considerable underestimation of the total alcohol recorded through official statistics by approximately 50% in nationally representative surveys [21-24].

Evidence suggests that the assessment results of digital and traditional administration modes are comparable in epidemiological surveys as well as in screening situations [25-27]. Promises of digitally administered tools, such as increased standardization and time efficiency, adaptability of the assessment flow based on user input, and seamless integration with electronic health records [28], may thus help address central SBI implementation barriers [29]. Importantly, digital assessment tools (DATs) can replace the standard drink concept by using individualized, interactive animation features to assess the type and amount of alcohol consumed. Related research fields such as nutrition epidemiology have already recognized the usefulness of visualization features to improve the quantification of consumption [30,31].

Currently, there is a growing body of literature focusing on the effectiveness and availability of evidence-based alcohol reduction apps [32,33]. These apps often contain a screening part quantifying the user’s consumption, which might be text based [34] or based on interactive animations [35]. However, to the best of our knowledge, the current availability of interactive animation features in alcohol DATs has not been systematically evaluated. There is also no systematic review of the effects of such features on assessment feasibility, acceptability, and validity.

Research Questions

This review seeks to answer the following two questions with a focus on DATs quantifying alcohol consumption (alcohol DATs), which use animation to support users in describing their consumption:

What freely available animation-using alcohol DATs exist in the English or Russian language, and what are their core characteristics?

To what extent have such tools been scientifically evaluated in terms of feasibility, acceptability, and validity?

Methods

Study Design

This systematic review was performed in 2 parts. In part 1, an app store search and a web-based search were conducted to identify existing freely available alcohol DATs. Part 2 comprised a systematic literature search to identify studies that evaluated the feasibility, acceptability, and validity of animation-using alcohol DATs.

The study protocol was published in PROSPERO (International Prospective Register of Systematic Reviews; registration number: CRD42020172825) [36]. We adhered to the standards set out in PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 [37]. Where applicable, we also followed the recommendations for methodological reporting of systematic searches in app store environments proposed by Grainger et al [38].

Part 1: Existing DATs

Eligibility Criteria

This review focused on animation-using alcohol DATs, defined as tools that allow the assessment and quantification of the user’s alcohol consumption via an electronic display device (a PC, laptop, or a mobile device). Aiming to include any alcohol DATs using interactive visualizations as opposed to purely text-based quantification tools, a broad definition of the term animation was chosen. Specifically, tools were considered to be using animations if they included ≥1 of the following features: (1) selection of a drink or a beverage type from a number of visually displayed options; (2) selection of a drinking vessel from a number of visually displayed glasses and, in some instances, bottles; and (3) simulated interactive pouring of a drink—that is, continuously adjusting the beverage level displayed in the chosen drinking vessel. The availability of each of the listed features was recorded to classify the complexity of the animation used. In addition, tools had to allow for the quantification of the user’s alcohol consumption over a defined reference period or occasion, be available in English or Russian, and be accessible free of charge. English-language apps form the largest language group among all apps available in the iOS App Store and the Google Play Store [39]. The mentioned app stores represent approximately 95% of the app market share worldwide [40] and offer ≥95% of the available apps free of charge [41,42]. Given the language background of the authors, the review additionally focuses on Russian-language apps. Russia has one of the highest proportions of alcohol-attributable mortality worldwide, and digital health interventions might become part of the promising prevention efforts currently taken and underway [43]. Our search for Russian-language alcohol DATs aims to identify relevant Russian-language alcohol DATs and gauge the potential of repeating the systematic search in additional languages in the future.

Search Strategy

The German app store versions of Google Play Store and iOS App Store were searched in June and July 2020, with English as the preferred app language. As app store search functions do not allow the systematic combination of search terms, 4 independent searches were performed on each platform, using the search terms alcohol, alcohol screening, alcohol test, and drinking. We recorded the first 250 results per platform using the search term alcohol. Given the high overlap between search results and decreasing relevance after the first 50 to 70 results, a maximum of 100 search results were screened for each of the other search terms. The Google search engine was searched in August 2020 using three sets of search terms (alcohol screening online, alcohol test online, and drinking test online). A total of 90 websites were included in the screening.To further explore the extent of regional adaptation in alcohol DATs and potential content differences between national app stores, we conducted additional searches in January and February 2021 in the Russian version of the Google Play and iOS App Store, with Russian as the preferred app language, using translated search terms. All search results were screened. The Russian Google search engine was searched in February 2021.

Screening and Selection of Tools

The URLs and titles of all identified app store entries or websites were recorded. After removing duplicates with identical URLs, the remaining app store descriptions and websites were screened for eligibility. A random sample of 25 English app store entries was independently screened by a second reviewer, and agreement was quantified to ensure the objectivity of the eligibility criteria. After screening, potentially eligible mobile tools were downloaded and completed on mobile devices (for English searches: Huawei Honor 9 Lite LLD-L31, Android version 9 and iPhone SE (2016), iOS version 14.0.1; for Russian searches: Samsung Galaxy Tab A 7.0 SM-T285 8 Gb, Android version 9 and Apple iPad (2018), iOS version 11.2). Web-based tools were completed on the web via the Safari and Google Chrome browsers to determine eligibility.

Data Extraction

The following data were extracted from the app store entries and linked websites and through testing the apps or web-based tools: general information (tool name, developer, responsible organization, link to website, scientific background or development process, country of publisher, year of the last update, and number of downloads), content features (reference period, underlying questionnaire, feedback on the user’s consumption quantity, use of standard drink concept, target group, and characteristic additional features), and animation features (availability of abovementioned features and options to adjust further drink characteristics).

Part 2: Studies on Feasibility, Acceptability, and Validity

Eligibility Criteria

Part 2 of the review aimed to identify (1) validation studies comparing animation-using alcohol DATs with established assessment methods (eg, paper–pencil, interview, or web-based questionnaires such as the AUDIT [13], AUDIT-C [14], Alcohol, Smoking, and Substance Involvement Screening Test [44], Alcohol Timeline Followback [45]; drinking diaries; standardized clinical interviews; or alcohol biomarkers) and (2) studies reporting on feasibility or acceptability of animation-using alcohol DATs. Eligibility was restricted to completed and fully reported studies. The same eligibility criteria for animation-using alcohol DATs were used as in part 1. When it was not possible to determine whether the eligibility criteria for using animations were met or when a study used a sample of participants aged < 15 years, the study was excluded. Studies conducted among general and specialized populations, such as patient populations, were eligible. No geographical, language, or time restrictions were applied.

Search Strategy

A systematic literature search was performed using Embase, MEDLINE, PsycINFO, and Web of Science. Search terms (Multimedia Appendix 1) were adapted to the requirements of each web-based database with regard to medical subject headings and wildcards. The searches covered publications from January 2000 to August 2020.

Screening and Selection of Studies

After removing duplicates, titles and abstracts were screened by a first reviewer, and a subsample of 100 records was independently screened by a second reviewer. In a second step, the full texts were obtained to decide about final inclusion.

Data Extraction

Information on general study characteristics (title, authors, year of publication, and type of study), study methods (setting, design, comparator, sample and recruitment strategy, period of data collection, outcomes, and outcome measurement), main findings, and information on the tested alcohol DATs were extracted. Owing to the high heterogeneity of the study designs, no standardized risk of bias assessment was performed.

Where possible, evaluated DATs were accessed and assessed against the same criteria as the DAT identified in the systematic app store search.

Results

Part 1: Existing DATs—Tools Identified and Included

Overview

A total of 1062 app store entries and 171 web entries were identified through app store and web searches (Figure 1). The searches in the Russian language yielded a much lower number of results than the English-language searches. After removing duplicates, of the 1233 total entries, 874 (70.88%) entries were screened for eligible alcohol DATs. Agreement between the reviewers was 92% for exclusion decisions after screening. A total of 54 mobile tools and 38 web-based tools were considered and tested for final inclusion. Finally, 35% (19/54) of mobile tools and 8% (3/38) of web-based tools were eligible. Of the 19 included tools, 16 (84%) were available in English only, 3 (16%) were available in both English and Russian without adaptations in content [46-48], and none were available in Russian only. All included mobile tools were available in German app stores; all but 16% (3/19) of mobile tools [49-51] could also be downloaded from Russian app stores.

Figure 1. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart of tool selection.

Content Characteristics

The core characteristics of the 22 alcohol DATs are summarized in Table 1 (detailed in Multimedia Appendix 2) [46-65]. Of the 19 mobile tools, 9 (47%) were available for both iOS and Android operating systems, 5 (26%) were published only in the iOS app store, and 5 (26%) were published only in the Google Play Store. The year of the last update ranged from 2014 to 2020, with 55% (12/22) of tools updated in 2020 or the previous year. The download numbers (only available for Android apps) ranged from ≥100 [49] to 50,000 [48,52]. Approximately 36% (8/22) of tools were developed in the United Kingdom [51-57], and 5% (1/22) each in Ireland [49], Canada [58], Russia [47], France [48], Denmark [46], Germany [59], and Japan [50]. The country of origin could not be identified for 18% (4/22) of tools [60-62,66]. There was no information available regarding any regional or cultural adaptation, and for tools available in both Russian and English, no cultural adaptations were evident. Publishing institutions included public actors [51,53,58], registered charities in the field of alcohol use prevention and general health [52,56,63], and private companies [46-48,54,55,59]. No information about the legal status of the publishing institution could be identified for 32% (7/22) of tools [49,50,57,61,62,66]. For only 14% (3/22) of tools, a scientific background and development process was mentioned [51,58,59]. With the exception of 5% (1/22) of tools designed for health care professionals [51], all tools targeted the general adult population, with a focus on individuals wanting to monitor or cut down their alcohol consumption.

Out of 22 identified tools, 3 (14%) were primarily designed to assess risky drinking in a one-time screening [51,53,54] and led to a structured feedback section, including or enabling (1) an estimate of the user’s alcohol-related health risk, (2) a comparison of the individual consumption to a relevant guideline or reference group, and (3) additional information on the standard drink concept and alcohol-related health risks. A total of 14% (3/22) of the identified tools were designed to deliver individualized programs to reduce or quit drinking [46,58,59] and started with a brief prospective [59] or retrospective [46,58] assessment of the user’s baseline consumption, followed by a tailored reduction scheme. All remaining tools relied on real-time assessment and were designed to either estimate the users’ blood alcohol concentration (blood alcohol concentration calculators; 4/22, 18% of tools) [50,60,63,66], count the number of standard drinks at a drinking occasion (standard drink counters; 1/22, 5% of tools) [48], or keep track of the alcohol consumed over a longer period (drinking diaries; 8/22, 36% of tools) [47,49,52,56,57,61,62]‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬.

Although none of the tools relied on the standard drink concept in the assessment part, most (13/22, 59% of tools) referred to this concept in their results and feedback sections [48,51-58,61,63,66,67]. In addition to alcohol-related health risk and consumed alcohol quantity, 45% (10/22) of tools reported money spent on alcohol, calories consumed, and hypothetical money or calories saved by cutting down drinking [48,50,52-56,58,61,63]‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬.

Similar to the mobile tools, all 3 included web-based alcohol DATs addressed the general adult population. They were provided by nonprofit organizations from Ireland [64,65] and the United Kingdom [56], with copyright claims absent [56] or dating to the current (2020) [65] or past year [64]. All organizations provided contact details of support services helping to cut down drinking. Out of 22 identified tools, 2 (9%) [56,64] were digital versions of the World Health Organization’s AUDIT. The functionality of standard drink calculation was directly embedded into AUDIT-C item 3 (“How many units of alcohol do you drink on a typical day when you are drinking?”), preserving the questionnaire’s original item structure. Both led to a detailed feedback section, including AUDIT score and risk category, information on standard drinks, and calories consumed on a typical day. The third tool converted the user’s reported consumption into standard drinks [65] and provided additional feedback items, including a comparison of the user’s alcohol consumption to a weekly low-risk drinking guideline [68].‬‬‬‬

Table 1. DATs^a quantifying alcohol consumption (alcohol DATs) in the English language: core characteristics of the included tools (N=22).

Tool name (year of last updateb; country)		Animation features				Adjust drinksc (n=13)		User feedback			Extra features
		Drinksd (n=21)	Vesselse (n=9)	Pourf (n=2)			Unit of consumptiong		Additional feedbackh
Mobile app: 1-time assessment of risky drinking
	Drinks Meter (2020; United Kingdom) [54]	✓	✓		✓		Standard drinks		Physiology or nutrition	Text-based AUDITi; drink pourer tool
	Know Your Numbers (2017; United Kingdom) [51]	✓	✓				Standard drinks		—j	Alcohol unit guide
	Know Your Units (2017; United Kingdom) [53]	✓			✓		Standard drinks		Physiology or nutrition	Beverage-specific sound animations
Mobile app: individualized program to reduce or quit drinking
	MeSelfControl (2016; Germany) [59]	✓	✓		✓		Alcohol quantity		—	—
	ReduceYour Drinking (2015; Denmark) [46]	✓					Alcohol quantity		—	Text-based DATs; available in Russian
	Saying When (2016; Canada) [58]	✓	✓	✓			Standard drinks		Positive effect	Explanation of standard drink concept
Mobile app: BACkcalculator
	alcCalc (2014: Japan) [50]	✓					Alcohol quantity		Physiology or nutrition	—
	Alcohol Diary (2019; not provided) [67]	✓					Standard drinks		—	—
	Alcohol meter (2019; not provided) [60]			✓	✓		Alcohol quantity		Physiology or nutrition	—
	DrinkWatch Unit Checker (2016; United Kingdom) [63]‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬	✓		✓	✓		Standard drinks		Physiology or nutrition; negative effect	—
Mobile app: drinking diary
	AlcoExpert (2019; Russia) [47]	✓			✓		Alcohol quantity		Physiology or nutrition; negative effect	Photorealistic drink images; available in Russian
	Alcofy (2020; not provided) [62]	✓	✓		✓		Alcohol quantity		Physiology or nutrition	—
	DrinkCoach (2020; United Kingdom) [56]	✓			✓		Standard drinks		Physiology or nutrition; positive effect	Visualized drinking scene; link to animation-enhanced AUDIT
	DrinkControl (2020; not provided) [61]	✓			✓		Standard drinks		Negative effect	Photorealistic drink images
	Dry Days (2020; United Kingdom) [55]	✓	✓		✓		Standard drinks		Positive effect	—
	Drynk (2020; Ireland) [49]	✓			✓		Standard drinks		—	BAC calculator
	Simple Alcohol Unit Tracker (2020; United Kingdom) [57]	✓					Standard drinks		Negative effect	—
	Try Dry (2020; United Kingdom) [52]	✓	✓		✓		Standard drinks		Physiology or nutrition; positive effect	AUDIT-Cl
Mobile app: SD counter
	Wise Drinking (2019; France) [48]	✓	✓		✓		Standard drinks		Physiology or nutrition	Available in Russian
Web-based tool: 1-time assessment of risky drinking
	DrinkCoach Alcohol Test (not provided; United Kingdom) [56]	✓	✓				Standard drinks; AUDIT risk score		Physiology or nutrition	Visually enhanced AUDIT; linked to the DrinkCoach mobile tool
	HSE Self-assessment tool (2019; Ireland) [64]	✓					Standard drinks; AUDIT risk score		Physiology or nutrition	Visually enhanced AUDIT
Web-based tool: SD counter
	Drinkaware Drinks Calculator (2020; Ireland) [65]	✓					Standard drinks		Physiology or nutrition; negative effect	Drink selection depends on chosen drinking context

^aDAT: digital assessment tool.

^bAt time of data extraction (2020).

^cNonvisually adjust drink characteristics.

^dChoose drinks from visual selection.

^eChoose vessels from visual selection.

^fSimulated interactive pouring of drinks.

^gStandard drinks, alcohol quantity (pure ethanol consumed [eg, in g or L]), and AUDIT risk score.

^hPhysiology- or nutrition-related feedback (eg, calories, ingested sugar, alcohol quantity equivalent in volume of beer or vodka, burger equivalent, exercise time to burn calories, typical symptoms at intoxication level, time until sober); negative effect of consumption (eg, money spent, heavy drinking days, drinking days per week); positive effect of reduced consumption (eg, money saved, sober days).

ⁱAUDIT: Alcohol Use Disorders Identification Test.

^jNot available.

^kBAC: blood alcohol concentration.

^lAUDIT-C: Alcohol Use Disorders Identification Test–Consumption.

Use of Animation

The 3 animation features defined in the eligibility criteria represent different levels of animation complexity. The distribution of these animation features in the identified alcohol DATs is summarized in Textbox 1.

With the exception of 9% (2/22) of tools using photorealistic images [47,61], all tools presented a selection of abstract and often simplified drink icons. Examples of the assessment screens are shown in Figure 2. More than half of the tools (12/22, 55%) offered only 1 animation feature (selection of a drink or a beverage type from a number of visually displayed options). The number of drinks to choose from differed considerably. Tools with fewer choices ( < 10 drink icons) [46,49,50,57,63,64,66] did not allow for any individualization of the chosen drink, whereas tools with more choices (16-29 drink icons) [47,56,61] enabled the user to individually adjust certain drink characteristics, including standard units of alcohol [56], drinking vessel size [61], volume consumed [47], and alcohol content of the consumed beverage. The Know Your Units tool [53] featured a virtual bar animation [53], allowing the user to drag a predefined drink icon from a shelf onto a bar table, where it was emptied out, accompanied by a beverage-specific sound animation.

Most other tools (8/22, 36%) relied on a 2-step process to describe consumed drinks. After choosing a beverage category, users could choose their glass or bottle from a beverage-specific selection. In the group of mobile tools, the choice of available beverage categories and vessels per category differed from basic (3-7 beverage categories; ≤3 vessel icons per category) [48,59,62] to detailed (6-7 beverage categories; 4-10 vessel icons per category) [52-55]. All but 25% (2/8) of these tools [48,59] displayed all individualization steps on 1 overview screen (eg, Figure 2, Try Dry). All tools allowed for nonvisual adjustment of beverage alcohol content [48,52-55,59] or consumed beverage quantity [59,62]. The DrinkCoach web-based tool [56] lets users choose from 12 beverage categories, as well as 3 to 7 vessels per beverage category, but did not allow for further adjustment of drink characteristics.

Only 9% (2/22) of the included tools featured the simulated interactive pouring of a drink; that is, continuously adjusting the beverage level displayed in a drinking vessel [58,60]. One of these tools used a nonchangeable standard vessel icon and a standard-colored beverage for the animation [60]. The other tool allowed users to choose the beverage and the vessel before pouring their drink (Figure 2, Saying When) [58]. The poured volume was displayed in real time during the pouring action, in milliliters as well as in standard drinks. The color of the liquid matched that of the chosen beverage. Further features to enhance the 3D character of the pouring experience, such as shadows, sound animations, or pouring-induced movement of the liquid surface were not identified.

Identified animation features in mobile and web-based alcohol digital assessment tools.

Identified animation features

Most of the included tools (21/22, 95%) offered the selection of a drink or a beverage type from a number of visually displayed options (1-step visual description).

Less than half of the tools (9/22, 41%) additionally offered the selection of a drinking vessel from a number of visually displayed glasses and, in some instances, bottles (2-step visual description).

Only 9% (2/22) of tools allowed the simulated interactive pouring of a drink; that is, continuously adjusting the beverage level displayed in the chosen drinking vessel.

Textbox 1. Identified animation features in mobile and web-based alcohol digital assessment tools.

Figure 2. Screenshots of drink input sections in mobile digital assessment tools quantifying alcohol consumption. From left to right: Simple Alcohol Unit Tracker [58] and Know Your Units [54] (both 1-step visual description); Try Dry [53] (2-step visual description); Saying When [59] (2-step visual description with adjustment of the vessel fill height).

Part 2: Identification of Feasibility, Acceptability, and Validity Studies

Overview

A total of 1585 records were identified through a systematic literature review search in Embase, MEDLINE, PsycINFO, and Web of Science (Figure 3). Removal of duplicates left 73.56% (1166/1585) of records for the title and abstract screening. In the random sample of 100 records screened by 2 reviewers, the agreement was 92% for inclusion decisions. Of the 81 full-text articles assessed for eligibility, 5 (6%) met the inclusion criteria.

Figure 3. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart of study selection. eSBI: electronic screening and brief intervention.

Study Characteristics

All included studies used a 1-arm study design with convenience sampling to explore the feasibility, acceptability, or validity of the alcohol DAT in question, or several of these concepts (Table 2). Of the 5 studies, 3 (60%) were conducted in Australia [69-71], 1 (20%) in Canada [72], and 1 (20%) study used a combined sample of participants recruited in Canada and Switzerland [73]. Data collection took place between 2015 and 2017; 40% (2/5) of papers [70,71] did not report the period of data collection.

Participants were recruited at primary health care and addiction centers [69,71] and through researcher networks [70], advertisements on university campus [70,72], social media, and internet forums [70,73]. Of 5 studies, 2 (40%) recruited participants from the general population [70,72]; 20% (n=1) of studies focused on adults with risky alcohol use [73]; and 40% (n=2) of studies used quotas to include nondrinkers, nondependent drinkers, and dependent drinkers [69,71]. Sample sizes ranged from 50 [72] to 671 [70] participants, with balanced proportions of men and women.

Table 2. Overview of included studies (N=5).

Reference			Country; time of data collection (tested alcohol DATa)		Study sample (age in years)		Recruitment		Main findings on acceptability and criterion or convergent validity
Acceptability and feasibility studies
	Lee et al [69]	Australia; 2016-2017 (Grog Survey app)		246 patients (18-78) with and without problematic alcohol use; 5 field research assistants (—b)		Primary health care and addiction center		Acceptability: 97% of patients rated alcohol DAT as easy to use or okay to use (rather than hard to use); staff suggested a high potential for the app to be used in primary health care settings, noted that participants appeared engaged and required minimal assistance
	Bertholet et al [73]	Switzerland and Canada; 2015 (Alcooquizz)		130 participants (mean 32.8, SD 10) with problematic alcohol use		Social media and internet forums		Acceptability: Low self-reported frequency of alcohol DATs use during the 3-month study period (only 53.6% of participants reported using it more than once); moderate rating for appreciation and usefulness of the alcohol DAT (mean 6/10 points, IQR 5-8)
Validation studies
	Lee et al [71]	Australia; 2019c (Grog Survey app)		238 participants (18-78) with and without problematic alcohol use		Primary health care and addiction centers		Criterion and convergent validity: Moderate (Spearman correlation between alcohol DAT and clinical interview for consumption quantity: r=0.68; P < .01); compared with interviews, alcohol DAT recorded higher numbers of standard drinks consumed per drinking occasion (median 17.0, IQR 10.5-27.9 and median 15.4, IQR 9.6-23.2) Criterion validity: Equal or better correlation of the presence of self-reported withdrawal tremors with the self-reported quantity of alcohol consumption in the alcohol DAT (r=0.40; P < .05) than with consumption estimate in the clinical interview (r=0.32; P < .05)
	Poulton et al [70]	Australia; 2018c (CNLab-A)		671 participants (16-56) with unknown alcohol use		Researcher networks and social media and internet forums		Convergent validity: Acceptable or high, with a significantly higher percentage of drinking days (P=.007) and total alcohol intake (P < .001) assessed by EMAd alcohol DAT compared with 21-day TLFBe; alcohol DAT recorded significantly higher hourly alcohol intake compared with AUQf (P=.002); no significant difference between AUQ and DAT in estimated weekly average consumption (P=.13)
	Vanderlee et al [72]	Canada; 2016 (Beverage Frequency Questionnaire)		50 participants (16-30) with unknown alcohol use		Advertisement on university campus		Convergent validity: High correlation with 7dFRg for number of drinks (Pearson r=0.58; P < .001) and consumed volume (r=0.78; P < .001) Acceptability: Good comprehensiveness assessed through cognitive interviewing (78% of participants reported no trouble in selecting a beverage image).

^aDAT: digital assessment tool.

^bNot available.

^cYear of study, as the year of data collection is not available.

^dEMA: ecological momentary assessment.

^eTLFB: Alcohol Timeline Followback.

^fAUQ: Alcohol Use Questionnaire.

^g7dFR: 7-day food record.

Characteristics of the Evaluated Tools

The evaluated alcohol DATs included 3 mobile apps and 1 web-based tool (Table 3), none of which had been identified in this review through the systematic app store and web search. Only 25% (1/4) of tools were publicly available in the German or Russian app stores [73]. They were designed to screen for risky alcohol use and collect consumption data at the population level [69,71], enable real-time assessment of alcohol intake [70], deliver a program to reduce drinking [73], and conduct epidemiological research [72].

Of the 4 tools, 2 (50%) presented a low number of visually displayed drink choices ( < 10 drink icons) [70,73]; 25% (1/4) of tools offered the additional choice of a drinking vessel (16 vessel icons in 4 alcoholic beverage categories) [72]. The Grog Survey app offered a wide range of region- and culture-specific beverages and drinking vessels and the additional feature of pouring a drink [71,75].

Table 3. Scientifically evaluated DATs^a quantifying alcohol consumption: overview of core characteristics.

Tool name (year of study)		Animation features			Adjust drinksb	User feedback			Extra features
		Drinksc	Vesselsd	Poure		Unit of consumptionf	Additional feedbackg
eSBIhfor problematic alcohol use (mobile app)
	Alcooquizz (2017) [73]	✓				Risk score	Physiology or nutrition; negative effect	Comparison to reference group
Ecological momentary assessment alcohol DAT (mobile app)
	CNLab-A (2018) [70]	✓			✓	N/Ai	—j	—
One-time assessment of risky drinking (mobile app)
	Grog Survey app (2019) [71,74]	✓	✓	✓		AUDITk risk score	—	Visualizations partly use user-generated drinks
One-time alcohol consumption assessment for epidemiological research (web-based)
	Beverage Frequency Questionnaire (2018) [72]	✓	✓			N/A	—	Also assesses consumption of nonalcoholic drinks

^aDAT: digital assessment tool.

^bNonvisually adjust drink characteristics.

^cChoose drinks from visual selection.

^dChoose vessels from visual selection.

^eSimulated interactive pouring of drinks.

^fStandard drinks, alcohol quantity (pure ethanol consumed [eg, in g or L]), AUDIT risk score, and DAT designed for epidemiological research, did not report the results to the user.

^gPhysiology or nutrition-related feedback (eg, calories, ingested sugar, and alcohol quantity equivalent in volume of beer or vodka, burger equivalent, exercise time to burn calories, typical symptoms at intoxication level, and time until sober); negative effect of consumption (eg, money spent, heavy drinking days, and drinking days per week); positive effect of reduced consumption (eg, money saved; sober days)

^heSBI: electronic screening and brief intervention.

ⁱN/A: not applicable; DAT designed for epidemiological research, did not report the results to the user.

^jNot available.

^kAUDIT: Alcohol Use Disorders Identification Test.

Findings Regarding Acceptability and Feasibility

Of the identified 5 studies, 2 (40%) focused on the acceptability and feasibility of the evaluated alcohol DAT (Table 2) [69,73]. Both used participant rating scales, rating appreciation and usefulness [73] and ease of use, respectively [69]. One of the tools, which was offered to study participants to be used at their discretion during a 3-month period, recorded the self-reported frequency of use [73]. In the second study, conducted in a health care setting, quantitative and qualitative staff observations were taken into account.

User evaluations of alcohol DATs were moderate to favorable. The animation-using personal feedback module of the first tool received an average participant rating of 6/10 in both the appreciation and the usefulness scales [73]. However, the self-reported frequency of use was low. The second tool was rated as easy to use or okay to use rather than hard to use by 97% of the study participants. Staff observations concluded that it could be completed with or without minimal assistance across different age groups [69].

Findings Regarding Validity

In total, 60% (3/5) of studies aimed to explore the validity of the respective alcohol DAT [70-72]. Established assessment methods to quantify alcohol consumption, such as clinical interviews or the alcohol Timeline Followback questionnaire, were used as comparators. One of the studies additionally evaluated the correlation between physical signs of addiction and the self-reported quantity of alcohol consumption [71].

The reported convergent validity was moderate in one of the studies [71] and moderate to high in a second study [72]. In 40% (2/5) of studies, the alcohol DAT recorded higher alcohol consumption than the established assessment method [70,72]. In one case, comparing an alcohol DAT designed for real-time drinking assessment with a 21-day retrospective assessment, this difference was statistically significant for the percentage of drinking days and the total alcohol intake but not for the number of heavy drinking occasions [70]. In the other study, the number of standard drinks consumed per drinking occasion did not significantly differ between the alcohol DAT and the established assessment method [71]. Furthermore, consumption estimates recorded in the alcohol DAT predicted physical signs of addiction as good or better than a clinical interview [71].

Discussion

Principal Findings

This systematic review is the first on DATs using animation features to support the quantitative assessment of alcohol consumption, a novel approach in the emerging field of digital health. Only 9% (2/22) of the alcohol DATs identified in part 1 of the review used animation in the sense of dynamically animated images that can be modified through user interaction (pouring a drink). Most animation features were implemented in a simplistic manner and did not exploit the full visualization potential of the available technology. The addition of dynamic visual hints, such as foam, bubbles, or visible movement of the beverage, could potentially help users recall their drinking habits in greater detail, which is thought to enhance the accuracy of reporting [76,77]. The results indicate that these features remain underused and that there is ample room for exploration and development.

In the identified alcohol DATs, relevant information regarding the responsible organization, scientific background, and development process was often incomplete or unavailable, which prevented a well-founded quality assessment. A larger degree of transparency is urgently required to fully exploit the potential of animation-using alcohol DATs. Similarly, none of the included tools provided information on the cultural or regional adaptation of the offered beverages and drinking vessels [15,78] or the approach and data sources used for this process. Additional searches with Russian search terms, aiming to identify relevant Russian-language alcohol DAT and gauge the potential of repeating the systematic search in additional languages in the future, showed a high availability of English-language apps in Russian app stores. However, they did not yield evidence of efforts to account for different cultural contexts in different language versions of the same app. Moreover, to the best of our knowledge, none of the tools identified in the first part of the review had been scientifically evaluated, underlining the lack of evidence for animation-using alcohol DATs publicly available in app stores [79].

The second part of this review identified 5 exploratory studies on the feasibility, acceptability, and validity of 4 animation-using alcohol DATs. These data showed fair to high convergent validity between established consumption assessment methods and animation-using alcohol DATs, whereas some alcohol DATs were shown to record higher quantities of alcohol consumption than the established measure. Considering the worldwide underestimation of self-reported alcohol consumption [22-24], these results could arguably be interpreted as a sign of improved assessment accuracy [21]. Animation-using alcohol DATs might thus contribute to reducing the well-known bias of standard surveys.

Strengths and Limitations

To not miss any relevant alcohol DAT using interactive visualizations, as opposed to purely text-based quantitation tools, a broad definition of the term animation was chosen, encompassing any apps that allow an image-based interaction with the user to quantify personal alcohol consumption. In part 1, systematic searches were conducted in Android and iOS app stores and via the Google search engine. These sources do not provide access to tools that are published in smaller stores, such as Amazon App Store, Samsung Apps, or Windows Store, or on open-source platforms, such as Github [80]. Alcohol DATs developed for health institutions or researchers may also have been missed, as they often use ways of dissemination not covered in this review [81]. We restricted our search to apps available free of charge. More than 95% of the apps in the Android market [41] and > 99% of all downloaded and installed apps [42] are estimated to meet this criterion.

The specific limitations of part 1 stem from the characteristics of nonscientific search engines and app stores as search environments. The providers of the platforms searched for this review do not disclose their search algorithms [38,82]. Search parameters, such as language and region settings and customization based on previous search behavior, are known to influence the choice and order of results, reducing the replicability of searches. Copyright regulations and the differing contents of national app stores further limit the selection of apps available for review. Separate searches in all available national app stores would not have been feasible with the available resources. On the basis of the team’s locations, we searched the German and Russian app stores, which showed a high content overlap with US and UK app stores [39]. The searches allowed for the identification of alcohol DATs from several countries. Obtaining true global or regional representativeness is beyond the scope of this review.

Moreover, digital app stores can be considered very unstable sources of information. Their contents change quickly over time and although for research articles stable identifiers, such as the digital object identifier number, have been developed, so far, there is no equivalent for mobile apps. Analyzing app store data through systematic searches is a relatively novel approach; thus, accepted reporting guidelines are not yet available. However, first recommendations have been developed [38,82], which guided the reporting in this review.

Many studies identified in part 2 focused on the evaluation of electronic SBI programs to reduce alcohol consumption and provided no information on the use of animation features in the alcohol DAT that was tested. If this information could be obtained through a web search, the study was excluded from the review. Therefore, it is possible that some studies evaluating animation-using alcohol DATs were falsely excluded.

Comparison With Prior Work and Future Research Perspectives

Today, > 318,000 health apps are available on the app stores [83], most of which are not recognizably evidence based [79,84]. Many apps, especially in the field of alcohol use, even promote harmful behavior [85]. Efforts have been undertaken to develop frameworks for app quality evaluation [86,87], as well as provide systematic evaluations of the apps available in specified health fields [88-90]. There are promising data on both the efficacy of health apps to reduce harmful alcohol consumption [32,33] and on the effect of interactive elements and gamification in health apps [91-93]. This review adds a separate evaluation of the availability and effects of interactive animation features on alcohol DAT.

Further research is needed to evaluate the differences between regional app markets within and beyond the English- and Russian-language markets. To facilitate the implementation of animation-using alcohol DAT in existing health care systems, target group–specific evaluations, analyzing the perspective of different age groups, and professional versus patient experiences with animation-using alcohol DAT would be highly valuable. The cognitive and psychological mechanisms underlying the effects of animation features also warrant further evaluation. For future research and tool development, the field of alcohol assessment might benefit from deepening the dialog with nutritional and dietary studies that have already started the development of interactive tools using more elaborate animation features that showed high validity and user-friendliness [30,94,95].

Conclusions

Research in the field of DATs is rapidly advancing. This is especially true for the area of mental health assessment tools, platforms, and resources and seems particularly urgent in light of the current COVID-19 pandemic [96-98]. By facilitating the collection of internationally comparable data as part of population-based surveys and improving the delivery of electronic SBIs for hazardous and harmful alcohol use, animation-using alcohol DATs might contribute to reducing alcohol-attributable health burden in the future. However, the potential of using animation features for the quantification of individual alcohol intake in DATs has not been fully exploited to date and has received little scientific attention. Further research is needed to explore the extent to which such features could improve the accuracy and user-friendliness of the assessment and identify the underlying mechanisms. However, although mostly using nondynamic animation features and often deficient in scientific background information, first animation-using alcohol DATs are available in app stores and on the web, and the explorative study data generated so far support their novel approach.

Acknowledgments

Financial support for the project was provided through a grant from the Government of Germany and the Russian Federation in the context of the World Health Organization (WHO) European Office for the Prevention and Control of Noncommunicable Diseases. These funding sources had no role in the design of this study, analyses, interpretation of the data, or decision to submit the results. Carina Ferreira-Borges and Maria Neufeld are staff members of the WHO Regional Office for Europe. The authors alone are responsible for the views expressed in this publication, and these do not necessarily represent the decisions or stated policy of the WHO.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Keyword set (Ovid search).

References

GBD 2016 Alcohol and Drug Use Collaborators. The global burden of disease attributable to alcohol and drug use in 195 countries and territories, 1990-2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet Psychiatry 2018 Dec;5(12):987-1012

Heather N. Can screening and brief intervention lead to population-level reductions in alcohol-related harm? Addict Sci Clin Pract 2012;7:15

Angus C, Thomas C, Anderson P, Meier PS, Brennan A. Estimating the cost-effectiveness of brief interventions for heavy drinking in primary health care across Europe. Eur J Public Health 2017 Apr 01;27(2):345-351.

Tansil KA, Esser MB, Sandhu P, Reynolds JA, Elder RW, Williamson RS, Community Preventive Services Task Force. Alcohol electronic screening and brief intervention: a community guide systematic review. Am J Prev Med 2016 Nov;51(5):801-811

SAFER: preventing and reducing alcohol-related harms. World Health Organization. 2018. URL: https://www.who.int/substance_abuse/safer/msb_safer_framework.pdf?ua=1 [accessed 2021-12-13]

US Preventive Services Task Force, Curry SJ, Krist AH, Owens DK, Barry MJ, Caughey AB, et al. Screening and behavioral counseling interventions to reduce unhealthy alcohol use in adolescents and adults: US preventive services task force recommendation statement. J Am Med Assoc 2018 Nov 13;320(18):1899-1909.

Anderson P, Wojnar M, Jakubczyk A, Gual A, Reynolds J, Segura L, et al. Managing alcohol problems in general practice in Europe: results from the European ODHIN survey of general practitioners. Alcohol Alcoholism 2014;49(5):531-539

Anderson P, Braddick F, Reynolds J, Gual A. Alcohol policy in Europe: evidence from AMPHORA. 2nd ed. In: Alcohol Measures for Public Health Research Alliance (AMPHORA). Brussels, Belgium: European Commission; 2012.

Rosário F, Santos MI, Angus K, Pas L, Ribeiro C, Fitzgerald N. Factors influencing the implementation of screening and brief interventions for alcohol use in primary care practices: a systematic review using the COM-B system and Theoretical Domains Framework. Implement Sci 2021 Jan 07;16(1):6

Sierosławski J, Foster J, Moskalewicz J. Survey of European drinking surveys. Alcohol survey experiences of 22 European countries. Drugs Edu Prevent Policy 2013 May 31;20(5):383-398.

Greenfield TK. Ways of measuring drinking patterns and the difference they make: experience with graduated frequencies. J Subst Abuse 2000;12(1-2):33-49.

Dawson DA. Methodological issues in measuring alcohol use. Alcohol Res Health 2003;27(1):18-29

Babor T, Higgins-Biddle J, Saunders J, Monteiro M. The alcohol use disorders identification test: guidelines for use in primary care, second edition. World Health Organization. 2001. URL: https://apps.who.int/iris/handle/10665/67205 [accessed 2021-12-13]

Bush K, Kivlahan DR, McDonell MB, Fihn SD, Bradley KA. The AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening test for problem drinking. Ambulatory Care Quality Improvement Project (ACQUIP). Alcohol Use Disorders Identification Test. Arch Intern Med 1998 Sep 14;158(16):1789-1795.

Kalinowski A, Humphreys K. Governmental standard drink definitions and low-risk alcohol consumption guidelines in 37 countries. Addiction 2016 Jul 13;111(7):1293-1298.

Babor T, Robaina K. The Alcohol Use Disorders Identification Test (AUDIT): a review of graded severity algorithms and national adaptations. Int J Alcohol Drug Res 2016 Jul 19;5(2):17-24.

Kerr W, Stockwell T. Understanding standard drinks and drinking guidelines. Drug Alcohol Rev 2012 Mar;31(2):200-205

Sprague DJ, Vinson DC. Patient perceptions of risky drinking: knowledge of daily and weekly low-risk guidelines and standard drink sizes. Subst Abus 2017;38(3):253-256.

Boniface S, Kneale J, Shelton N. Actual and perceived units of alcohol in a self-defined "usual glass" of alcoholic drinks in England. Alcohol Clin Exp Res 2013 Jun 20;37(6):978-983

Welsh C, Earley K, Delahanty J, Wright K, Berens T, Williams A, et al. Residents' knowledge of standard drink equivalents: implications for screening and brief intervention for at-risk alcohol use. Am J Addict 2014;23(2):194-196.

Del Boca FK, Darkes J. The validity of self-reports of alcohol consumption: state of the science and challenges for research. Addiction 2003 Dec;98 Suppl 2:1-12.

Boniface S, Shelton N. How is alcohol consumption affected if we account for under-reporting? A hypothetical scenario. Eur J Public Health 2013 Dec;23(6):1076-1081.

Probst C, Shuper PA, Rehm J. Coverage of alcohol consumption by national surveys in South Africa. Addiction 2017 Apr 20;112(4):705-710.

Kilian C, Manthey J, Probst C, Brunborg GS, Bye EK, Ekholm O, et al. Why is per capita consumption underestimated in alcohol surveys? Results from 39 surveys in 23 European countries. Alcohol Alcohol 2020 Aug 14;55(5):554-563

Campbell N, Ali F, Finlay AY, Salek SS. Equivalence of electronic and paper-based patient-reported outcome measures. Qual Life Res 2015 Aug;24(8):1949-1961.

Gwaltney CJ, Shields AL, Shiffman S. Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: a meta-analytic review. Value Health 2008;11(2):322-333.

Harris SK, Knight JR, Van Hook S, Sherritt L, Brooks TL, Kulig JW, et al. Adolescent substance use screening in primary care: validity of computer self-administered versus clinician-administered screening. Subst Abus 2016;37(1):197-203

Harris SK, Knight JR. Putting the screen in screening: technology-based alcohol screening and brief interventions in medical settings. Alcohol Res 2014;36(1):63-79

Sawares AS, Shen N, Xue Y, Abi-Jaoude A, Wiljer D. The impact of mobile apps on alcohol use disorder: a systematic review protocol. JMIR Res Protoc 2017 Apr 04;6(4):e49

Illner A, Freisling H, Boeing H, Huybrechts I, Crispim SP, Slimani N. Review and evaluation of innovative technologies for measuring diet in nutritional epidemiology. Int J Epidemiol 2012 Aug;41(4):1187-1203

Amoutzopoulos B, Steer T, Roberts C, Cade JE, Boushey CJ, Collins CE, et al. Traditional methods new technologies - dilemmas for dietary assessment in large-scale nutrition surveys and studies: a report following an international panel discussion at the 9th International Conference on Diet and Activity Methods (ICDAM9), Brisbane, 3 September 2015. J Nutr Sci 2018;7:e11

Meredith SE, Alessi SM, Petry NM. Smartphone applications to reduce alcohol consumption and help patients with alcohol use disorder: a state-of-the-art review. Adv Health Care Technol 2015;1:47-54

Wieczorek L, Klingemann JI. Mobile applications used to limit alcohol consumption – a literature review. Alcohol Drug Addict 2020;33(1):43-64.

Attwood S, Parke H, Larsen J, Morton KL. Using a mobile health application to reduce alcohol consumption: a mixed-methods evaluation of the drinkaware track and calculate units application. BMC Public Health 2017 Dec 17;17(1):394

Garnett C, Oldham M, Angus C, Beard E, Burton R, Field M, et al. Evaluating the effectiveness of the smartphone app, Drink Less, compared with the NHS alcohol advice webpage, for the reduction of alcohol consumption among hazardous and harmful adult drinkers in the UK at 6-month follow-up: protocol for a randomised controlled trial. Addiction 2021 Feb;116(2):412-425

PROSPERO: International prospective register of systematic reviews. National Institute for Health Research. URL: https://www.crd.york.ac.uk/prospero/ [accessed 2021-12-13]

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Br Med J 2021 Mar 29;372:n71

Grainger R, Devan H, Sangelaji B, Hay-Smith J. Issues in reporting of systematic review methods in health app-focused reviews: a scoping review. Health Informatics J 2020 Dec 11;26(4):2930-2945

Winners & Losers in the Global App Economy. Farnham, Surrey, United Kingdom: Caribou Digital Publishing; 2016.

Huang Z, Soljak M, Boehm BO, Car J. Clinical relevance of smartphone apps for diabetes management: a global overview. Diabetes Metab Res Rev 2018 May;34(4):e2990.

Number of available Android applications. AppBrain. URL: https://www.appbrain.com/stats/free-and-paid-android-applications [accessed 2021-12-13]

Wang H, Li H, Guo Y. Understanding the evolution of mobile app ecosystems: a longitudinal measurement study of Google play. In: Proceedings of the The World Wide Web Conference.: Association for Computing Machinery, Inc; 2019 Presented at: WWW '19: The Web Conference; May 13 - 17, 2019; San Francisco CA USA p. 1988-1999.

Neufeld M, Ferreira-Borges C, Gil A, Manthey J, Rehm J. Alcohol policy has saved lives in the Russian Federation. Int J Drug Policy 2020 Jun;80:102636.

WHO ASSIST Working Group. The Alcohol, Smoking and Substance Involvement Screening Test (ASSIST): development, reliability and feasibility. Addiction 2002 Sep;97(9):1183-1194.

Sobell LC, Sobell MB. Timeline follow-back. In: Litten RZ, Allen JP, editors. Measuring Alcohol Consumption. Totowa, NJ: Humana Press; 1992:41-72.

ReduceYourDrinking - Reduce alcohol consumption successfully [Mobile App]. CorporateCom GmbH. 2015. URL: https://apps.apple.com/us/app/id943492616 [accessed 2021-07-08]

AlcoExpert: alcohol tracker [Mobile app]. Grey Media. 2019. URL: https://play.google.com/store/apps/details?id=com.alcoexpert [accessed 2021-07-08]

Wise Drinking: let's be smart [Mobile app]. Pernod Ricard. 2019. URL: https://play.google.com/store/apps/details?id=com.pernodricard.wisedrinking & hl=gsw & gl=US [accessed 2021-07-08]

Drynk alcohol tracker and insights [Mobile app]. Syzible. 2020. URL: https://play.google.com/store/apps/details?id=com.syzible.drynk & hl=de & gl=US [accessed 2021-07-08]

alcCalc [Mobile App]. Noth Co. Ltd. 2014. URL: https://apps.apple.com/us/app/alccalc-estimates-displays-alcohol-decomposition-time/id887557458 [accessed 2021-07-08]

Know Your Numbers (WessexAHSN) [Mobile app]. Network WAHS. 2017. URL: https://play.google.com/store/apps/details?id=com.wessexahsn.knowyournumbers & hl=de & gl=US [accessed 2021-07-08]

TRY DRY: the app for dry January and beyond [Mobile app]. Alcohol Change, UK. 2020. URL: https://play.google.com/store/apps/details?id=uk.org.alcoholconcern.dryjanuar [accessed 2021-07-08]

Know your units [Mobile app]. Alcohol And You, Northern Ireland. 2017. URL: https://apps.apple.com/bn/app/know-your-units/id1260567821 [accessed 2021-07-08]

Drinks meter [Mobile app]. Global Drug Survey. 2020. URL: https://play.google.com/store/apps/details?id=com.drinksmeter.alcohol [accessed 2021-07-08]

Dry days by AlcoChange [Mobile app]. AlcoChange. 2020. URL: https://apps.apple.com/us/app/dry-days-by-alcochange/id1062948603 [accessed 2021-07-08]

DrinkCoach [Mobile app]. Humankind. 2019. URL: https://play.google.com/store/apps/details?id=com.td.drinkcoach & hl=en_US & gl=US [accessed 2021-07-08]

Simple alcohol unit tracker [Mobile app]. DeveloperJam. App Store; 2020. URL: https://play.google.com/store/apps/details?id=com.developerjam.drinklogger [accessed 2021-07-08]

Saying when [Mobile App]. CAMH (Centre for Addiction and Mental Health). 2016. URL: https://play.google.com/store/apps/details?id=com.sayingwhencamh [accessed 2021-07-08]

MeSelfControl [Mobile App]. AppPeople. 2016. URL: https://apps.apple.com/ca/app/meselfcontrol/id1051751592 [accessed 2021-07-08]

Alcohol meter [Mobile App]. Alcaraz J (ZaraclaJ). 2019. URL: https://play.google.com/store/apps/details?id=zaraclaj.alcohol_counter [accessed 2021-07-08]

DrinkControl: alcohol tracker [Mobile app]. E-protect. 2020. URL: https://apps.apple.com/us/app/drinkcontrol-alcohol-tracker/id456207840 [accessed 2021-07-08]

Alcofy - Alcohol tracker and BAC drink calculator [Mobile app]. JMJ Dev. 2020. URL: https://play.google.com/store/apps/details?id=com.jmstudio.alctrack [accessed 2021-07-08]

Drink watch unit checker [Mobile app]. FASA - Henry McCrory. 2016. URL: https://apps.apple.com/us/app/drink-watch-unit-checker/id1186722413 [accessed 2021-08-07]

HSE Self assessment tool [Web-based tool]. Health Service Executive, Ireland. URL: https://www2.hse.ie/wellbeing/alcohol/self-assessment-tool/#Start [accessed 2021-12-21]

Drinkaware - Drinks calculator [Web-based tool]. Alcohol Awareness Foundation Ireland T/A. URL: https://www.drinkaware.ie/tools-resources/drinks-calculator [accessed 2021-12-21]

Alcohol diary [Mobile App]. James Porter. 2019. URL: https://apps.apple.com/us/app/alcohol-diary/id1437094610 [accessed 2021-07-08]

Luczak SE, Hawkins AL, Dai Z, Wichmann R, Wang C, Rosen IG. Obtaining continuous BrAC/BAC estimates in the field: a hybrid system integrating transdermal alcohol biosensor, Intellidrink smartphone app, and BrAC Estimator software tools. Addict Behav 2018 Aug;83:48-55

Alcohol Guidelines Review: summary of the proposed new guidelines. UK Chief Medical Officer. 2016. URL: https:/​/assets.​publishing.service.gov.uk/​government/​uploads/​system/​uploads/​attachment_data/​file/​489795/​summary.​pdf [accessed 2021-12-21]

Lee KK, Conigrave JH, Al Ansari M, Wilson S, Perry J, Zheng C, et al. Acceptability and feasibility of a computer-based application to help Aboriginal and Torres Strait Islander Australians describe their alcohol consumption. J Ethn Subst Abuse 2021 Mar 19;20(1):16-33.

Poulton A, Pan J, Bruns LR, Sinnott RO, Hester R. Assessment of alcohol intake: retrospective measures versus a smartphone application. Addict Behav 2018 Aug;83:35-41.

Lee KSK, Conigrave JH, Callinan S, Wilson S, Room R, Perry J, et al. Asking about the last four drinking occasions on a tablet computer as a way to record alcohol consumption in Aboriginal and Torres Strait Islander Australians: a validation. Addict Sci Clin Pract 2019 May 01;14(1):15

Vanderlee L, Reid JL, White CM, Hobin EP, Acton RB, Jones AC, et al. Evaluation of the online Beverage Frequency Questionnaire (BFQ). Nutr J 2018 Aug 01;17(1):73

Bertholet N, Daeppen J, McNeely J, Kushnir V, Cunningham JA. Smartphone application for unhealthy alcohol use: a pilot study. Subst Abus 2017;38(3):285-291.

Lee K, Conigrave J, Wilson S, Perry J, Callinan S, Room R, et al. Short screening tools for risky drinking in Aboriginal and Torres Strait Islander Australians: modified AUDIT-C and a new approach. Addict Sci Clin Pract 2019 Jul 01;14(1):22

Lee KK, Wilson S, Perry J, Room R, Callinan S, Assan R, et al. Developing a tablet computer-based application ('App') to measure self-reported alcohol consumption in Indigenous Australians. BMC Med Inform Decis Mak 2018 Jan 15;18(1):8

McKenna H, Treanor C, O'Reilly D, Donnelly M. Evaluation of the psychometric properties of self-reported measures of alcohol consumption: a COSMIN systematic review. Subst Abuse Treat Prev Policy 2018 Feb 02;13(1):6

Greenfield TK, Kerr WC. Alcohol measurement methodology in epidemiology: recent advances and opportunities. Addiction 2008 Jul;103(7):1082-1099

Bräker AB, Soellner R. Alcohol drinking cultures of European adolescents. Eur J Public Health 2016 Aug;26(4):581-586.

Weaver ER, Horyniak DR, Jenkinson R, Dietze P, Lim MS. "Let's get wasted!" and other apps: characteristics, acceptability, and use of alcohol-related smartphone applications. JMIR Mhealth Uhealth 2013;1(1):e9

GitHub: where the world builds software [Software repository]. GitHub, Inc. 2021. URL: https://github.com/ [accessed 2021-12-21]

Higgins K, Tutelman P, Chambers C, Witteman H, Barwick M, Corkum P, et al. Availability of researcher-led eHealth tools for pain assessment and management: barriers, facilitators, costs, and design. Pain Reports 2018 Sep;3(1):e686.

Grundy QH, Wang Z, Bero LA. Challenges in assessing mobile health app quality: a systematic review of prevalent and innovative methods. Am J Prev Med 2016 Dec;51(6):1051-1059.

The growing value of digital health. QVIA Institute. 2017. URL: https://www.iqvia [accessed 2021-12-21]

Anthes E. Pocket psychiatry - mobile mental-health apps have exploded onto the market, but few have been thoroughly tested. Nature 2016 Apr 7;532(7597):20-23

Ghassemlou S, Marini C, Chemi C, Ranjit YS, Tofighi B. Harmful smartphone applications promoting alcohol and illicit substance use: a review and content analysis in the United States. Transl Behav Med 2020 Oct 12;10(5):1233-1242

Chandrashekar P. Do mental health mobile apps work: evidence and recommendations for designing high-efficacy mental health mobile apps. Mhealth 2018 Mar;4:6

Terhorst Y, Philippi P, Sander LB, Schultchen D, Paganini S, Bardus M, et al. Validation of the Mobile Application Rating Scale (MARS). PLoS One 2020 Nov 2;15(11):e0241480

Brzan PP, Rotman E, Pajnkihar M, Klanjsek P. Mobile applications for control and self management of diabetes: a systematic review. J Med Syst 2016 Sep;40(9):210.

Marshall J, Dunstan D, Bartik W. The digital psychiatrist: in search of evidence-based apps for anxiety and depression. Front Psychiatry 2019;10:831

Tofighi B, Chemi C, Ruiz-Valcarcel J, Hein P, Hu L. Smartphone apps targeting alcohol and illicit substance use: systematic search in in commercial app stores and critical content analysis. JMIR Mhealth Uhealth 2019 Apr 22;7(4):e11831

Sardi L, Idri A, Fernández-Alemán JL. A systematic review of gamification in e-Health. J Biomed Inform 2017 Dec;71:31-48.

Edwards EA, Lumsden J, Rivas C, Steed L, Edwards LA, Thiyagarajan A, et al. Gamification for health promotion: systematic review of behaviour change techniques in smartphone apps. BMJ Open 2016 Oct 04;6(10):e012447

Schmidt-Kraepelin M, Thiebes S, Sunyaev A. Investigating the relationship between user ratings and gamification - A review of mHealth apps in the apple App Store and Google play store. In: Proceedings of the Hawaii International Conference on System Sciences. 2019 Presented at: Hawaii International Conference on System Sciences; 2019; Hawaii p. 1496-1505.

Foster E, Hawkins A, Simpson E, Adamson AJ. Developing an interactive portion size assessment system (IPSAS) for use with children. J Hum Nutr Diet 2014 Jan 19;27 Suppl 1:18-25.

Timon CM, Blain RJ, McNulty B, Kehoe L, Evans K, Walton J, et al. The development, validation, and user evaluation of foodbook24: a web-based dietary assessment tool developed for the Irish adult population. J Med Internet Res 2017 May 11;19(5):e158

Bucci S, Schwannauer M, Berry N. The digital revolution and its impact on mental health care. Psychol Psychother 2019 Jun 28;92(2):277-297.

Boumparis N, Schulte MH, Riper H. Digital mental health for alcohol and substance use disorders. Curr Treat Options Psych 2019 Nov 26;6(4):352-366.

Martin-Key N, Schei T, Barker E, Spadaro B, Funnell E, Benacek J, et al. The current state and diagnostic accuracy of digital mental health assessment tools for psychiatric disorders: protocol for a systematic review and meta-analysis. JMIR Res Protoc 2021 Jan 08;10(1):e25382