Driven by notifications – exploring the effects of badge notifications on user experience

Abstract

Notification badges are an unexplored category of visual feedback to which we are continuously exposed. This study aims to deepen knowledge on the topic by measuring the behavioral effects of notification badges on a large sample of smartphone users. More precisely, the goal of the study is to observe if the presence of notification badges increases the frequency of clicks on apps. More than 1000 participants were involved in a remote between-subjects experiment, allocated into fifteen equinumerous groups of comparison. Each participant was presented with a smartphone screen displaying fifteen app icons and just one badge notification. Participants were asked to perform a remote user test called First Impression Click Test: a methodology that indicates where they would click first to accomplish a given task (i.e., Where would you click first on this screen?). Our results show a large increase in the number of clicks on apps with notification badges compared to those without notification badges and suggest the important ability of these small affordances to attract attention and stimulate action. Based on the evidence provided, our findings have practical implications for user experience design.

Introduction

In recent years we have witnessed an exceptional growth in the spread of smartphones and in the use of applications. The number of smartphone subscriptions worldwide today surpasses six billion and is forecasted to further grow by several hundred million in the next few years (Statista, 2021; www.statista.com/topics/840/smartphones). On average, smartphone owners check their phone every 7 minutes, use 9 to 10 different apps per day and about 30 apps each month (App Annie–The state of mobile, 2021; www.appannie.com/en/go/state-of-mobile-2021). Applications like messaging, e-mail, and online social networks allow smartphone users to connect with their family, friends and even co-workers wherever they are. Most of these apps use proactive, push-based notifications (i.e., visual, auditory, and/or haptic alerts) to inform smartphone users about new, unattended messages or events, even when they are not actively using their device. Smartphones offer various methods for the user to receive notifications, including a status bar, LED indicator, pop-up screen, badge, vibration, sound, or a combination of these [1]. In this context, a badge is the red circle that appears on the upper right-hand corner of the app’s icon, capable of capturing and holding user attention. The white number inside the red circle represents the number of unread notifications waiting to be opened.

By visually showcasing notifications that a user hasn’t yet engaged with, badge notifications can trigger two main cognitive biases: salience and sense of urgency. Salience bias (also known as perceptual salience) is the cognitive bias that predisposes individuals to focus on items that are more visually prominent or emotionally striking and ignore those that are unremarkable, even though this difference is often irrelevant by objective standards. This creates a bias in favor of things that are striking and perceptible [2–4]. Sense of urgency is a cognitive bias or a systematic error of thinking, that puts people into reactive mode and pushes them to choose one option rather than another based on which they perceive most urgent, even if that choice is objectively worse or it distracts from long-term goals. In other words, people are psychologically inclined to put off meaningful work in favor of tasks that feel more urgent [5, 6]. Notification badges can therefore be considered as wordless but meaningful calls to action, triggering those cognitive biases with minimal effort.

While previous studies have focused on notifications in the broadest sense of the term, very little is known about smartphone badge notifications and their impact on a user’s behavior.

Several works have focused on the effects of desktop [7, 8] and mobile notifications [9–17]. Fallman & Yttergren [9] proposed a system that adapts smartphone notifications according to situational circumstances, Garzonis et al. [10] studied intuitiveness, learnability, memorability, and user preference of two types of audio notifications on the smartphone. Iqbal & Horvitz [7] and Mark, Voida and Cardello [8] investigated the role of desktop notifications in work environments and highlighted both their interruptive character and their stressful effect on those who receive them while carrying out a primary task. Fischer et al. [12], on the other hand, conducted a study on the timing of mobile notifications, finding that they are processed faster if they arrive at the end of an interaction with their device (call or SMS). Saket et al. [13] observed the urgency effect generated by ten different vibration modes as a notification system on smartphones. Pielot et al. [14] worked on finding predictors of the time it takes a user to interact with the notification of a mobile message.

One of the latest works published compared the effectiveness of two types of notification (push vs badge) in informing software developers about the availability of software updates and found that, on average, systems that use badge notifications are slightly less effective than those based on push notifications [18]. Morrison et al. [15] published an exploratory study on the potential impact of time (intelligent and non-intelligent) and the frequency (daily and occasional) of responding to push notifications on the use of a stress management tool on smartphones. The authors found that frequent notifications can encourage greater exposure to intervention content without discouraging engagement, but adaptive tailoring of notification timing does not always enhance their use. Fitz et al. [16] tested whether delivering notifications in predictable intervals throughout the day could improve psychological well-being and found that those who received notifications three times a day felt more attentive, productive, in a better mood, and more in control of their phones. with respect to those who received notifications in unpredictable ways. Loid, Täht and Rozgonjuk [17] found that notifications regarding excessive smartphone use did not lower self-reported scores about problematic smartphone use, nor participants’ screen time or the frequency of phone-checking behaviour.

With the aim of filling the literature gap and shedding light on smartphone badge notifications and their impact on user behavior, we conducted a remote and unmoderated test called First Impression Click Test, which allows to collect data on where, on an interface, people would click first to complete a given task [19, 20].

The goal of the study was to observe if the presence of badges would have increased the frequency of clicks on apps. To this end more than 1000 participants were involved in a remote between-subjects experiment, allocated into fifteen equinumerous groups of comparison. Each participant was presented with a smartphone screen displaying fifteen app icons and just one badge notification. Every screen was identical to the other, except for the app icon on which the badge notification was placed.

Materials and methods

Participants

We recruited 1095 participants (55% women, mean age = 38 years, SD = 10) through an online panel to take part in the experiment. All of them self-declared as smartphone users but were naïve as to the aims and the expected outcomes of the experiment. The experimental group was allocated to 15 equinumerous groups of comparison (73 participants each). We eventually rejected 86 participants out of 1095 from all analyses because of poor data quality (i.e. participants clicking on regions other than icons). A small financial compensation was offered to participants. The study was performed in keeping with the Declaration of Helsinki. All participants gave written informed consent before participation. The JAKALA Experience Lab Ethics Committee approved the study.

At the end of the experimental session, participants completed a short questionnaire regarding their app’s ordinary usage habits (Fig 1). The questionnaire investigated the frequency of use of the 15 apps shown during the test using a 5-point Likert scale (from 1 = "I never use it" to 5 = "I use it several times a day"). Participants declared that the apps they use the most were WhatsApp, Telephone Facebook, and Gmail while the ones they use the least were LinkedIn, Shazam, Teams and Just Eat.

[Figure omitted. See PDF.]

Results have been displayed according to the frequency of use (descending order).

Apparatus and stimuli

The First Impression Click Test is a user research methodology that allows one to collect first impressions regarding static interfaces and more specifically where people will click first to accomplish a given task [20]. This remote test is often used to validate templates and design choices, but additionally, it allows one to investigate the elements of an interface that most attract users’ attention and to make inferences on their information search patterns. In our study, the First Impression Click Test was carried out remotely through a dedicated online platform (Optimal Workshop-Chalkmark; www.optimalworkshop.com/chalkmark).

Each of the fifteen groups of participants (15 groups x 73 participants) carried out the remote test performing just one task on a single home screen of an Android smartphone with one notification badge placed on just one app out of the fifteen in total. The given task required participants to freely select the area that they would have clicked on first when viewing the screen. The task was thus formulated: “Where would you click first on this screen?”. The test ended when the participants clicked on the desired element of the interface. At the end of the test, a short questionnaire on apps use habits was administered.

The apps chosen for the home screen of an Android smartphone were selected according to their total number of downloads and frequency of use (App Annie–The State of Mobile 2021 Report; www.appannie.com/en/go/state-of-mobile-2021), aiming to reproduce a realistic and representative set of apps for Italian users. We preferred to employ Android in lieu of iOS since the former represents about the 75% of market share and is therefore the most common device among the Italian population (Statista, 2022; www.statista.com/statistics/623153/leading-mobile-operating-systems-ranked-by-market-share-in-italy). In addition to the apps available by default on a smartphone (Calendar, Telephone, SMS, Settings), nine further apps were selected:

* social media (Facebook, LinkedIn);

* messaging and email (WhatsApp, Messenger, Teams, Gmail and Outlook);

* shopping (Amazon, Just Eat);

* entertainment (YouTube, Shazam).

Experimental design and procedure

A between-subject design was employed. Participants were evenly distributed among 15 groups of the same size. Each of the fifteen groups of participants was exposed to a single experimental condition (one smartphone interface with the notification badge on a single app). Each experimental session lasted 90 seconds on average and consisted of:

1. reading a short instructions page for the first impression click test;

2. performing the first impression click test;

3. filling in a questionnaire on apps use habits and sociodemographic profile.

Dependent variable

To test the hypothesis, the percentage of clicks on the apps was employed as the only dependent variable. The given task consisted, for each of the participants, of just one single click on the chosen app. Every participant could therefore select only 1 app out of 15 for each experimental session. Therefore, the percentage of clicks was a dichotomous variable which contained two distinct values: 100% and 0%. The first value (100%) was attributed to 1 single app out of 15, the second (0%) to the rest of the apps (14 out of 15).

Results

The significance level α was set at .05 for all statistical analyses. The percentage of clicks was analyzed with a Kruskal–Wallis H test, a non-parametric method for comparing two or more independent samples of equal or different sample sizes. The adjusted Mann–Whitney U test [21, 22] was instead used for planned comparisons (Badge vs. No badge). Means, standard deviations and sample sizes for each of the dependent variables and experimental conditions are reported in Table 1.

[Figure omitted. See PDF.]

To test the hypothesis that the presence of a badge increases the percentage of clicks on that app, each of the apps with the badge was compared (one at a time) to the group constituted by the rest of the apps with no badge (n = 14). Results showed that the presence of the notification badge systematically captures more clicks with respect to the condition in which the badge is unavailable. As to the interactions with gender and age no results emerged between the two groups (i.e Badge vs. No badge). Main effects and planned contrasts for each of the apps are shown in Table 2. Results are reported in Fig 2.

[Figure omitted. See PDF.]

[Figure omitted. See PDF.]

Main effects were calculated using the Kruskal–Wallis H test (all ps < .001). Planned contrasts were calculated with the adjusted Mann–Whitney U test (all ps < .001).

Discussion

The goal of the present study is to investigate the impact of badge notifications on smartphone users’ behavior. Regardless of the app type, the presence of the notification badge systematically captured more clicks with respect to the condition in which the badge is unavailable. Based on the evidence provided, our findings highlight the strong impact of badge notifications and have practical implication for user experience design.

These results can be explained by to two main theories that refer to cognitive biases. Cognitive biases are systematic patterns of deviation from norm or rationality in judgment [23]. They arise from erroneous perceptions, from which judgments, prejudices and ideologies are inferred because of the brain’s attempt to simplify information processing [2, 3, 24]. In everyday life, cognitive biases can be beneficial because they do not require much mental effort and they allow people to make decisions quickly without subjecting each decision to careful criticism or judgment which takes time and energy. Cognitive biases are activated by what Kahneman defines as System 1, the most primordial component of our brain to which the automatic and fast processes belong (e.g., perception and intuition) and which prevails when faced with apparently simple problems or which suggest an immediate answer [25]. The solutions provided by System 1 are sometimes accepted beyond conscious and rational judgment and intuitions or perceptions, even if incorrect, can turn into behaviors: in this context, the slower one relating to logical processes such as rational thinking and reasoning (i.e., System 2), is not activated.

Among the various theories that have studied cognitive biases there are two that allow us to interpret the results of this study: the first refers to the saliency bias [4] and the second to the urgency bias [5, 6]. On the one hand, the salience bias describes the human tendency to focus on items or information that are more noteworthy while ignoring those that do not grab our attention. In our case, the notification badges, due to their physical characteristics (e.g., color, shape, positioning), create a salience bias in favor of those apps that, from a visual and perceptive level, instantly become more relevant than the others. On the other hand, the urgency bias describes the tendency for people to perform unimportant tasks over important tasks when the unimportant tasks are characterized merely by spurious urgency (and illusion of expiration). In this respect, the notification badges produce a reactive effect on a cognitive-emotional level and push people, due to the implicit meaning they convey, to choose an app rather than another based on the one they perceive as most urgent and important.

The results of our study can be considered pioneering in the field of notification badge design as there is no evidence in the literature regarding the topic of the study. Future studies might investigate the demographic characteristics of the participant sample or the physical aspects of the notification badges. As to the sample, participants from different nationalities and with different app usage habits might be involved. As to the manipulation of factors related to the notification badge, it might be useful to include factors such as the number shown in the badge (digit other than 1), the color and the position. Also, it might be interesting to compare badge notifications with other types of smartphone notifications such as status bar, LED indicator, pop-up screen, vibration, sound, or a combination of these. This will allow us to broaden the knowledge of the effect of these messages on people’s behavior and the awareness of the stimuli to which we are exposed every day.

Conclusions

Notification badges represent a category of warning to which we are continuously exposed, and which needs to be investigated since there is little knowledge in the literature regarding their effects on human behavior. This study aims to deepen the knowledge on the topic by measuring the effects of notification badges on the behavior employing a large sample of smartphone users. Our results show a large increase in the number of clicks on apps with notification badges compared to those without notification badges and suggest the important ability of these small affordances to attract attention and stimulate action.

Supporting information

S1 Dataset. Study data.

https://doi.org/10.1371/journal.pone.0270888.s001

(XLSX)

Acknowledgments

We would like to thank Prof. Paul Tokorcheck and Dr. Orazio Gallo for their precious help with manuscript editing.

Citation: Bartoli N, Benedetto S (2022) Driven by notifications – exploring the effects of badge notifications on user experience. PLoS ONE 17(6): e0270888. https://doi.org/10.1371/journal.pone.0270888

About the Authors:

Nicola Bartoli

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Writing – original draft, Writing – review & editing

Affiliation: Jakala Experience Lab, JAKALA S.p.A. S.B. s.u., Milan, Italy

Simone Benedetto

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Supervision, Writing – original draft, Writing – review & editing

E-mail: [email protected]

Affiliation: Jakala Experience Lab, JAKALA S.p.A. S.B. s.u., Milan, Italy

https://orcid.org/0000-0002-6053-1052

References

1. Yoon S, Lee K-P. A study on notification system design of smartphone messenger considering the user’s stress. Arch Des Res. 2015;28(2):75.

2. Kahneman D., Slovic S. P., Slovic P., & Tversky A. Judgment under uncertainty: Heuristics and biases. Cambridge university press. 1982.

3. Gilovich T, Griffin D, Kahneman D, Cambridge University Press. Heuristics and biases: The psychology of intuitive judgment. Gilovich T, Griffin D, Kahneman D, editors. Cambridge, England: Cambridge University Press; 2002.

4. Bordalo P, Gennaioli N, Shleifer A. Salience theory of choice under risk. Q J Econ. 2012;127(3):1243–85.

5. Kotter J. P., Kim W. C., & Mauborgne R. A. HBR’s 10 Must Reads on Change Management (including featured article" Leading Change," by John P. Kotter). Harvard Business Press; 2011.

6. Zhu M, Yang Y, Hsee CK. The mere urgency effect. J Consum Res [Internet]. 2018; Available from: https://doi.org/10.1093/jcr/ucy008

7. Iqbal ST, Horvitz E. Notifications and awareness: A field study of alert usage and preferences. In: Proceedings of the 2010 ACM conference on Computer supported cooperative work—CSCW ‘10. New York, New York, USA: ACM Press; 2010.

8. Mark G, Voida S, Cardello A. A pace not dictated by electrons" an empirical study of work without email. In 2012. p. 555–564.

9. Fallman D, Yttergren B. Meeting in quiet: choosing suitable notification modalities for mobile phones. In 2005. p. 55–es.

10. Garzonis S, Jones S, Jay T, O’Neill E. Auditory icon and earcon mobile service notifications: intuitiveness, learnability, memorability and preference. In 2009. p. 1513–1522.

11. Fischer JE, Yee N, Bellotti V, Good N, Benford S, Greenhalgh C. Effects of content and time of delivery on receptivity to mobile interruptions. In: Proceedings of the 12th international conference on Human computer interaction with mobile devices and services—MobileHCI ‘10. New York, New York, USA: ACM Press; 2010.

12. Fischer JE, Greenhalgh C, Benford S. Investigating episodes of mobile phone activity as indicators of opportune moments to deliver notifications. In: Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services—MobileHCI ‘11. New York, New York, USA: ACM Press; 2011.

13. Saket B, Prasojo C, Huang Y, Zhao S. Designing an effective vibration-based notification interface for mobile phones. In: Proceedings of the 2013 conference on Computer supported cooperative work—CSCW ‘13. New York, New York, USA: ACM Press; 2013.

14. Pielot M, Church K, de Oliveira R. An in-situ study of mobile phone notifications. In: Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services—MobileHCI ‘14. New York, New York, USA: ACM Press; 2014.

15. Morrison LG, Hargood C, Pejovic V, Geraghty AWA, Lloyd S, Goodman N, et al. Correction: The effect of timing and frequency of push notifications on usage of a smartphone-based stress management intervention: An exploratory trial. PLoS One. 2018;13(5): e0198008. pmid:29787600

16. Fitz N, Kushlev K, Jagannathan R, Lewis T, Paliwal D, Ariely D. Batching smartphone notifications can improve well-being. Comput Human Behav. 2019; 101:84–94.

17. Loid K, Täht K, Rozgonjuk D. Do pop-up notifications regarding smartphone use decrease screen time, phone checking behavior, and self-reported problematic smartphone use? Evidence from a two-month experimental study. Comput Human Behav. 2020; 102:22–30.

18. Mirhosseini S, Parnin C. Can automated pull requests encourage software developers to upgrade out-of-date dependencies? In: 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE; 2017.

19. Lindgaard G, Fernandes G, Dudek C, Brown J. Attention web designers: You have 50 milliseconds to make a good first impression! Behav Inf Technol. 2006;25(2):115–26.

20. Gronier G. Measuring the First Impression: Testing the Validity of the 5 Second Test. Journal of Usability Studies. 2016;12(1).

21. McKnight PE, Najab J. Mann‐Whitney U Test. In 2010. p. 1–1

22. Chen SX, Qin J, Tang CY. Mann-Whitney test with adjustments to pretreatment variables for missing values and observational study: Mann-Whitney Test with Adjustments to Pretreatment Variables. J R Stat Soc Series B Stat Methodol. 2013;75(1):81–102.

23. Haselton MG, Nettle D, Murray DR. The evolution of cognitive bias. In: The Handbook of Evolutionary Psychology. Hoboken, NJ, USA: John Wiley & Sons, Inc.; 2015. p. 1–20.

24. Orzan G, Zara IA, Purcarea VL. Neuromarketing techniques in pharmaceutical drugs advertising. A discussion and agenda for future research. J Med Life. 2012;5(4):428–32. pmid:23346245

25. Kahneman D. Thinking, fast and slow. Macmillan. 2011.