Defining destigmatizing design guidelines for use in sexual health-related digital technologies: A Delphi study

Abstract

Objective

Stigma has been recognized as a significant issue in sexual health, yet no specific guidelines exist to support digital health development teams in creating stigma-alleviating sexual health digital platforms. The purpose of this study was to develop a set of design guidelines that would serve as a reference point for addressing stigma during the design of sexual health-related digital platforms.

Materials and Methods

We conducted a 3-round Delphi study among 14 researchers in stigma and sexual health. A preliminary list of 28 design guidelines was generated from a literature review. Participants appraised and critiqued the clarity and usefulness of the preliminary list and provided comments for each item and for the overall group of items at each round. At each round, a content validity index and an interquartile range were calculated to determine the level of consensus regarding the clarity and usefulness of each guideline. Items were retained if there was high consensus or were dropped if there was no consensus after the three rounds.

Results

Nineteen design guidelines achieved consensus. Most of them were content-related guidelines and sought to address the emotional concerns of patients that could potentially aggravate stigma. The findings also reflected modern stigma management strategies of making stigma a societal attribute by challenging, exposing, and normalizing stigma attributes via web platforms.

Conclusion

To address stigma via digital platforms, developers should not just concentrate on technical solutions but seriously consider content-related and emotional design components that are likely to result in stigma.

Data Availability:Data for this study are available upon a reasonable request via the University of British Columbia Office of Research Ethics by e-mail at [email protected].

Funding:This study was funded by the UBC School of Nursing Lyle Creelman Endowment Fund and the Designing for People NSERC-CREATE program. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Sexual health-related conditions, herein described as a group of health issues or clinical syndromes that affects an individual’s sexuality or sexual functioning, have attracted widespread attention among healthcare providers and policymakers across the globe [1]. Stigma is a well-documented barrier to engagement in care, medication adherence, and health-seeking activities among people living with a variety of sexual health-related conditions [2]. Stigma manifests in two main forms: internalized stigma and public stigma [3]. Internalized stigma occurs when a person perceives their situation as a negative attribute with resulting decreases in self-esteem and self-worth, while public stigma manifests as overt discriminatory practices targeted at a group with attributes that are perceived to be negative [3]. Individuals encountering stigma report several challenges including poor decision-making, poor interaction with family and friends with its associated consequences of social withdrawal, disclosure-associated anxiety, and reduced sexual well-being [4,5]. The stigma faced by people with sexual health-related conditions is more pronounced in conventional health settings such as in-person visits at sexually transmitted infection (STI) clinics than in digital interventions [6]. For example, showing up at a sexual health clinic could reveal that a person has a sexual health condition, and having the sexual health condition may be considered a negative attribute which may contribute to both internal and public stigma. Despite global efforts in combating sexual health-related conditions, the stigma associated with such health issues continues to hamper progress [7]. Even though many biomedical and structural interventions have been developed for the management of sexual health-related conditions, the stigma associated with in-person visits and face-to-face interactions often poses a challenge to the effective utilization of these services [8].

Given the stigma associated with in-person clinical encounters and interpersonal interactions, various digital health interventions have been developed to complement, and in some cases replace, conventional health services [9–14]. Digital health interventions may be more useful for addressing sexual health-related stigma because people are more likely to use technology-based interventions for health problems that are perceived as embarrassing, stigmatizing, and difficult to discuss face-to-face [15]. However, the way in which digital health technologies are designed or deployed could inadvertently reinforce, (re)produce, or perpetuate stigma among users who may find the technology content or functions to be emotionally distressing [10].

Previous studies have revealed how some digital health technologies not only fail to address the stigma of sexual health issues but may also contain interface components that could inadvertently aggravate stigma among end-users [13,16]. For instance, people living with sexual health-related conditions can link the appearance of sexual health-related content to an existing public stigma, thereby resulting in a digital health technology that elicits stigma. The inability of digital health technologies to alleviate stigma or the possibility of provoking stigma via digital platforms suggests a possible lack of a structured set of design guidelines to help address stigmatizing components during the creation of such digital platforms. The absence of structured destigmatizing design guidelines may make stigma a lesser consideration in digital health design despite being an outcome of interest for healthcare providers, patients, and web users alike. In an early project, we started to develop destigmatizing design guidelines but recognized that the field would benefit from a rigorous expert review [17]. Thus, the purpose of this study was to develop a set of destigmatizing design guidelines that would serve as a reference point for addressing stigma during the design of sexual health-related digital platforms. Design guidelines are generally described as “rules of thumb” or “universally applicable laws, rules, or considerations” that are used as a foundation for a pleasant design [18]. The guidelines produced from this study might help digital health development teams (e.g., software designers and content creators) to determine which digital health content or features could be perceived to be stigmatizing by end-users [19]. We also expect the findings to help move the emphasis of digital health development from largely employing technical solutions to considering other non-technical factors that might alleviate or aggravate stigma.

Materials and methods

This study adopted a Delphi technique to develop a set of destigmatizing design guidelines. A Delphi study was chosen because it supports the development of items through consensus-building among a group of experts on a given topic. A panel of experts in stigma and sexual health research were consulted via an online survey and asked to appraise and critique a preliminary set of design guidelines identified from the literature and from prior studies [9–17,19,20]. The panel of experts were also asked to comment on each guideline as well as identify additional design guidelines that they thought could help address stigma via digital platforms. There were three rounds of feedback and additional items that were generated from participants’ comments were integrated into the subsequent round. Details of each step are explained below.

Ethics statement

This study was approved by the University of British Columbia Behavioral Research Ethics Board (Approval number = H21-00760). All participants provided written consent before participating in this study.

Pre-Delphi stage

Consistent with prior Delphi studies, a candidate list of 28 items was generated through a literature review before the start of the Delphi study [21,22]. Using the concept of trauma-informed care as the guiding framework, we generated design guidelines that could help to alleviate stigma related to sexual health [23]. Trauma-informed care is a holistic approach to healthcare that seeks to offer an understanding of and a thoughtful response to individuals who have experienced an emotionally traumatic event(s), aimed at enhancing their resilience and self-efficacy [24]. Indeed, trauma-informed care is becoming an important area of interest in digital health research [25]. A trauma-informed care framework was considered useful for addressing stigma via digital platforms because of the intertwined relationship between people who experience stigma and trauma [26], and also because of how a trauma-informed framework has been successfully used in HIV stigma-reduction activities [27]. Given that some digital health technologies inadvertently perpetuate stigma [13,16], a different approach using a trauma-informed care framework was thought to have the potential to address stigma concerns among people who use digital health technologies. We specifically used the trauma-informed care framework proposed by Harris and Fallot [23]. Unlike traditional user-interface design guidelines that mostly concentrate on addressing technical problems [28], the use of this framework helped identify preliminary design guidelines from the literature review that were largely content-related and non-technical in nature. By adopting this framework, we were able to generate non-technical design solutions that cover a broad range of stigma management strategies [29–32].

Participant recruitment

A list of 158 researchers in sexual health and stigma-related research across North America, Europe and Africa was purposefully compiled via publications, conference presentations, and through the authors’ professional networks. These researchers were subsequently recruited via email invitation. To be eligible, potential participants were required to have at least five years of experience in sexual health or stigma-related research, be fluent in English, and have a minimum of an undergraduate degree. The target sample size was 30 researchers, which is consistent with prior Delphi studies [22,33].

Data collection

Data collection occurred in 3 rounds of Delphi and ended after the third round. Data from each round were analyzed before the next round began. In Round 1, the preliminary list of 28 design guidelines were circulated to the participants via an online survey platform (QualtricsTM). The participants rated the clarity and usefulness of each guideline in addressing sexual health-related stigma via digital platforms on a scale of 1–7, where 1 = strongly disagree, 2 = disagree, 3 = mildly disagree, 4 = neutral, 5 = mildly agree, 6 = agree, and 7 = strongly agree. Participants also justified their ratings by commenting on the usefulness and clarity of each item. At the end of the survey, participants were asked to provide additional ideas that they thought could help to address stigma. Items that achieved consensus in Round 1 were retained as part of the final set of design guidelines while those not achieving consensus were revised and fed back in Round 2. Additional design guidelines were also generated from participants comments and suggestions from Round 1. These additional guidelines were presented to participants in Round 2. In Round 2, participants appraised the clarity and usefulness of the revised and newly generated guidelines. Items that achieved consensus were retained as part of the final set of guidelines. Additional guidelines were also generated from Round 2 and the new items as well as items not meeting consensus were presented in Round 3. The same appraisal process occurred in Round 3. However, because this was the final round, no additional comments or suggestions were solicited from the participants. Instead, participants were asked to state their concluding remarks regarding the study. Consensus items were retained and those not achieving consensus were dropped. An item was considered as having been agreed upon if it obtained a mean value ≥ 5 on the Likert scale (i.e., overall value greater than “mildly agree”).

Data analysis

At the end of each round, a content validity index (CVI) and interquartile range (IQR) were calculated for each guideline across the two dimensions of clarity and usefulness. Consensus in this study was defined as a CVI of ≥ 80% and an IQR of ≤ 1 on both clarity and usefulness dimensions. A CVI measures the percentage of people who agree with the clarity and usefulness of the guidelines in addressing stigma while an IQR measures the variation among participants’ agreement or disagreement. That is, for an item to reach consensus, at least 80% of the panel should agree on its clarity and usefulness and the IQR for such an agreement should not be more than 1. An IQR was chosen because it is less affected by outliers and is also considered a better measure of dispersion for smaller sample sizes than a standard deviation [34].

In the second round, stability was calculated for both clarity and usefulness dimensions to determine the consistency of answers between Round 1 and Round 2. Stability refers to the consistency of answers between two successive rounds of a Delphi study [35]. It is calculated on items not meeting consensus to determine which items to drop. For Likert scale items, stability is often calculated by following a five-step process including: i) adding up the number of responses for each response option for each item in Rounds 1 and 2; ii) calculating the difference in the response option between Rounds 1 and 2 (for instance, if 5 people responded between “strongly disagree to neutral” for an item in Round 1 and 3 people choose the same response in Round 2, then the absolute difference would be 2); iii) adding all the absolute differences across all items; iv) dividing the total sum by the number of rounds (2 rounds); and v) dividing the results from step 4 by the number of participants (14 participants between Round 1 and Round 2 in our study). The final absolute score could range from 0–1 where a higher score (typically above 0.2, or 20%) indicates wide variations in response options between the two rounds and a lower score (i.e., typically lower than 20%) suggests that responses for an item between two rounds will not change in another round. Stable items had responses between “strongly disagree” to “neutral” while unstable items had responses from “strongly disagree” to “strongly agree” on the Likert scale. The purpose of calculating stability was to establish which items were unlikely to change if sent to Round 3. Items were considered stable (i.e., if participants’ responses ranged from strongly disagree to neutral) and were subsequently dropped if they achieved a stability value of ≤ 0.2 (or 20%) on both clarity and usefulness dimensions. Items with stability levels >0.2 (or 20%) in either 1 or both dimensions were considered unstable (i.e., if responses ranged from strongly disagree to strongly agree) and were revised for Round 3. Since Round 3 was the final round, stability was not calculated on items not achieving consensus.

Results

Characteristics of the Delphi panel

Of the 158 participants who were contacted, 22 (13.9%) expressed interest and met the eligibility criteria. Of these 15 participated in Round 1, 14 participated in Round 2, and 13 participated in Round 3. Of the 15 initial participants, 5 (33.3%) were from Canada, 5 (33.3%) were from Europe, 4 (26.6%) were from the United States, and 1 (6.6%) was from Rwanda. Ten (66.6%) participants self-identified as a woman, 4 (26.6%) identified as a man, and 1 (6.6%) did not report on gender identity. Thirteen (86.7%) were either assistant, associate, or full professors at research-intensive universities while 2 (13.3%) were working at public health agencies. The mean number of years of work experience in sexual health and stigma-related research was 21.6 years (SD = 10.26, Range = 6–46 years). 13 participants had a PhD, 1 had a Masters, and 1 had both an MD and a Masters.

Delphi results

In Round 1, twelve (12) design guidelines achieved consensus. However, participants provided useful comments on 8 of these items that warranted further revisions (see tables below and S1 Table). Therefore, only 4 items were retained as part of the final set of design guidelines in Round 1. Of the 16 guidelines that did not meet consensus, 9 were revised, 2 were merged, and 5 were dropped (i.e., DG3, DG4, DG6, DG14, DG20, and DG24)–bringing the number of design guidelines for Round 2 to 26. In Round 2, 12 of the 26 guidelines achieved consensus and were added as part of the final guidelines. Two (2) items were dropped because they were either stable with “strongly disagree—neutral” scores on both clarity and usefulness dimensions while another 2 were dropped because the content was present in other newly generated guidelines. Ten items were thus revised, and two additional items were generated, making a total of 12 items for review in Round 3. In Round 3, only three of the 12 items achieved consensus. Since this was the final round, no further revisions were conducted on the guidelines that did not achieve consensus. At the end of Round 3, 19 design guidelines were found to have achieved consensus. Fig 1 shows a summary of the Delphi process, together with the number of items that achieved consensus, were revised, dropped, or retained for each round of the Delphi study. The figure also shows the number of respondents for each round of the Delphi. Table 1 shows the detailed results for each round of the Delphi study for the original 28 guidelines.

Fig 1. Flowchart of the Delphi Technique Process.

Table 1. Results for the newly generated design guidelines for Rounds 2 and 3.

Revision of design guidelines not achieving consensus

The design guidelines that did not meet consensus were revised because they were perceived as either not clear, double-barrelled, or not useful/irrelevant to stigma reduction. Therefore, the revisions were mainly done to improve on items’ clarity, usefulness, and relevance to stigma reduction via digital platforms. Some items were dropped or merged through the revision process because they were either thought to be redundant or could inadvertently aggravate rather than alleviate stigma. For instance, design guidelines that were meant to hide stigmatizing attributes on digital platforms (i.e., Ensure appropriate flexibility of web features [DG13]; Make logos, icons, and terminologies subtle [DG5]) were seen as likely to perpetuate or worsen stigma because such features might suppress people’s feelings and their ability for expression. Similarly, design guidelines that centered on integrating links and hyperlinks on digital platforms [DG12] were considered as having an inherent security and privacy risk. Furthermore, design guidelines that centered on having website users narrate their experiences [DG21 & DG7.1] were considered inappropriate and not secure if there were no trained operational staff to moderate such platforms. See S1 Table for the original design guidelines and their corresponding revisions in Rounds 1 and 2. Additional guidelines were also generated by reframing participants’ comments that reflected antistigma-informed approaches to digital technology design. As seen in Table 2 below, 8 additional guidelines were obtained from the revision process in Round 1, and 2 additional guidelines were obtained in Round 2. Because the additional guidelines were generated after the Delphi study had begun, they were appraised twice in Round 2 and only once in Round 3

Table 2. Results for the newly generated design guidelines for Rounds 2 and 3.

The final set of design guidelines

The study started with 28 items generated from the pre-Delphi stage and an additional 10 items generated from Rounds 1 and 2 (Total = 38). At the end of Round 3, 19 items representing 50% of the original (n = 28) and newly generated (n = 10) items were judged by the panel of experts as clear and useful in addressing stigma via digital platforms. Table 3 shows the final set of consensus guidelines mapped onto the Harris and Fallot five principles of trauma-informed care framework. Design guidelines on this table without a decimal point are items that achieved consensus following a single appraisal process (both original and revised items). Item IDs with “.1” suffix are guidelines that achieved consensus in Round 2 following Round 1 revisions, while item IDs with “.2” suffix indicate guidelines that achieved consensus in Round 3 following Round 2 revisions.

Table 3. Final set of design guidelines mapped onto the 5 principles of trauma-informed care framework.

Discussion

To the best of our knowledge, this study is the first to develop a set of technology design guidelines from a health-related domain by employing a trauma-informed care framework. Even though the usefulness of the trauma-informed care framework was not empirically tested in this study, the results align with previous studies that suggested that a trauma-informed care framework could help address stigma [27]. The findings of this study are expected to make a unique contribution to the body of literature on design guidelines as well as stigma management via digital platforms. Unlike traditional design guidelines that tend to focus on the technical solutions and the functionality/usability of digital technologies, this study demonstrates a shift from a predominantly technically focused approach to designing digital platforms to include emotional and content-related guidelines.

The development of content-related and largely emotional design guidelines in this study reinforces the need for and importance of incorporating emotional design requirements in developing digital health technologies [36]. Thus, the guidelines developed in this study are expected to help technology developers think beyond systems’ usability and functionality features to include essential elements that can safeguard patients’ emotional safety, including not contributing to stigma. The incorporation of content-related guidelines in designing digital platforms is important because design guidelines that fall outside the usual technical guidelines are generally less obvious, more difficult to specify, and most often not considered by digital health developers [28]. By situating this study at the intersection of sexual health and digital health design, it can also be argued that the final design guidelines reflected human-centred values including human dignity, privacy, anonymity, autonomy and choice, sympathy, human relationships, and strength-based approaches, which are considered cardinal hallmarks of good healthcare [37].

Unlike prior stigma research that emphasizes hiding or avoiding strategies as appropriate mechanisms for alleviating stigma [29–32], this study suggested otherwise. Instead of hiding stigmatizing attributes as a way of addressing sexual health-related stigma, the design guidelines produced in this study rather seek to challenge, expose, and normalize ostensibly stigmatizing attributes via digital platforms. This open approach to addressing stigma reflects modern stigma management strategies that seek to shift the focus from the individual to include other people and larger societal factors that simultaneously operate to shape and perpetuate stigma [38–40]. A reason that was advanced against hiding or avoiding strategies is the fact that hiding stigmatizing content may rather lead people to accept public perceptions or accept the status quo regarding the stigmatizing attribute [30]. In other words, hiding or avoiding strategies may perpetuate internalized stigma by forcing people to accept the negative attributes associated with their condition. The findings of this study thus suggest that earlier stigma management strategies that center on hiding, avoiding, evading, or covering the stigmatizing attributes might have outlived their usefulness in addressing stigma in our current context.

The design guidelines produced in this study also reflected modern stigma management strategies that adopt education as a tool to increase peoples’ knowledge about a specific condition. Education strategies involve interventions that aim to inform the public, community groups, and individuals by increasing their knowledge about a specific condition by providing facts that counterbalance the false assumptions upon which stigma is based [31]. Participants in this study had a positive consensus on two of the four principles that adopted education strategies to alleviate stigma. These principles include DG16.1 (i.e., Include information that corrects myths about the condition to enable users to get accurate information), and DG17.1 (i.e., Provide factual and plain language information that normalizes and de-stigmatizes sexual health-related conditions). Participants’ agreement on the usefulness of education strategies in addressing stigma confirms the predominant use of educational messages in stigma reduction campaigns [31]. Participants not only agreed on the usefulness of these educational principles in addressing stigma but also agreed with principles that would make educational messages accurate, reliable, and non-judgmental. This reinforces earlier studies that suggest that educational messages that are accurate, reliable, credible, and non-judgmental are crucial for addressing stigma [41,42].

While education strategies were considered useful in addressing sexual health-related stigma, participants’ comments suggested that the content of educational messages is very important for its effects on stigma and stigma reduction. Indeed, the effect of educational content on stigma reduction seems to be more pronounced in the information displayed on sexual health-related websites [13,43]. The manner in which educational messages are crafted and displayed on digital platforms could determine their success in alleviating stigma [9–16]. Even though the panel of experts argued against hiding or avoiding stigmatizing content, adopting the opposite approach by displaying content in a superficial and unfavourable manner could inadvertently perpetuate stigma in the end. To determine the right approach, careful consideration of the context in which the digital health technology will be developed would determine whether openly displaying sexual health-related content or being discrete with content would help in addressing stigma. The effect of context-specific content display on stigma alleviation was not explored in this study. Further studies are needed to examine how, for example, the same web-based content impacts stigma in different cultural and linguistic settings.

The study also produced design guidelines that reflect the principles of inclusive design [44,45]. Unlike traditional inclusive design guidelines that are largely focused on designing products for older people and people with physical disabilities [44], this study extends the notion of inclusive design to include people with a range of ages, ethnicities, and gender identities [DG 19.1]. At a time when the digital divide is negatively impacting people from marginalized sex and gender identities, [46] consideration of the design guidelines produced in this study as belonging to the inclusive design sphere may help digital health teams to understand how to design digital health technologies that are inclusive of marginalized groups most often affected by stigma [46].

Limitations

This study has some limitations that should be considered when interpreting or applying the findings. Unlike the preliminary design guidelines identified in the literature review, the 10 additional guidelines generated during the Delphi procedure did not benefit from a 3-round appraisal process. Also, the data collection method did not allow participants to discuss the topic amongst each other. Instead of recruiting only people with research expertise on the topic, the Delphi process could have benefited from a review by experts in the field of sexual health practice such as nurses, physicians, and social workers. Further studies should convene a focus group among researchers and field experts as well as people who use digital health interventions to further understand how the design guidelines can be applied. Finally, out of 158 participants contacted, only 22 agreed to participate, and 13 participated in all three rounds. It is unknown if the demographics of participants who agreed to participate or those who participated were truly representative of the 158 people who were originally contacted.

Conclusion

The findings of the Delphi study extended the knowledge on design guidelines from a predominant focus on technical and usability considerations of digital health technologies to include emotional and content-related guidelines. Design guidelines that transcend the usual functionalist considerations are often not considered by development teams yet considered critical for ensuring emotional comfort of the end-users. Unlike the earlier stigma management strategies, the design guidelines also reflect a modern approach to managing stigma by suggesting that the stigmatizing attribute be made public via digital platforms. This public approach has the potential to ultimately make stigma more of a community or societal concern rather than an individual attribute. For the design guidelines to be successful in addressing stigma, project managers or organizations that wish to embark of digital health projects for sensitive health topics like sexual health should consider making this reference guide an essential part of the project contract when engaging with the development team. This way, digital health developers can ensure that the final product will strive to prevent stigma.

Supporting information

S1 Table. Design guideline revisions at Rounds 1 and 2.

https://doi.org/10.1371/journal.pdig.0000223.s001

(DOCX)

Citation: Abdulai A-F, Howard AF, Yong PJ, Currie LM (2023) Defining destigmatizing design guidelines for use in sexual health-related digital technologies: A Delphi study. PLOS Digit Health 2(7): e0000223. https://doi.org/10.1371/journal.pdig.0000223

References

1. WHO. Sexual health, human rights and the law. Reprod Health Matters. 2015;23(46):193–5.

2. Stangl AL, Lloyd JK, Brady LM, Holland CE, Baral S. A systematic review of interventions to reduce HIV-related stigma and discrimination from 2002 to 2013: how far have we come? J Int AIDS Soc. 2013; pmid:24242268

3. Dudina V, Tsareva A. Studying Stigmatization and Status Disclosure Among People Living with HIV/AIDS in Russia Through Online Health Communities | SpringerLink. 2018. p. 15–24.

4. Mill J, Edwards N, Jackson R, Austin W, MacLean L, Reintjes F. Accessing health services while living with HIV: Intersections of stigma. Can J Nurs Res. 2009; pmid:19831060

5. Rueda S, Mitra S, Chen S, Gogolishvili D, Globerman J, Chambers L, et al. Examining the associations between HIV-related stigma and health outcomes in people living with HIV/AIDS: a series of meta-analyses. BMJ Open. 2016; pmid:27412106

6. Nyblade L, Stockton MA, Giger K, Bond V, Ekstrand ML, Lean RM, et al. Stigma in health facilities: Why it matters and how we can change it. BMC Med. 2019; pmid:30764806

7. Hogben M, Ford J, Becasen JS, Brown KF. A Systematic Review of Sexual Health Interventions for Adults: Narrative Evidence. J Sex Res. 2015;52(4):444–69. pmid:25406027

8. Lunny C, Taylor D, Memetovic J, Wärje O, Lester R, Wong T, et al. Short message service (SMS) interventions for the prevention and treatment of sexually transmitted infections: A systematic review protocol. Syst Rev. 2014; pmid:24433348

9. Bauermeister JA, Muessig KE, LeGrand S, Flores DD, Choi SK, Dong W, et al. HIV and Sexuality Stigma Reduction Through Engagement in Online Forums: Results from the HealthMPowerment Intervention. AIDS Behav. 2019; pmid:30121727

10. Davis WM, Shoveller JA, Oliffe JL, Gilbert M. Young people’s perspectives on the use of reverse discourse in web-based sexual-health interventions. Cult Health Sex. 2012; pmid:22943601

11. Flickinger TE, DeBolt C, Xie A, Kosmacki A, Grabowski M, Waldman AL, et al. Addressing Stigma Through a Virtual Community for People Living with HIV: A Mixed Methods Study of the PositiveLinks Mobile Health Intervention. AIDS Behav. 2018; pmid:29882048

12. Israel T, Choi AY, Goodman JA, Matsuno E, Lin YJ, Kary KG, et al. Reducing internalized binegativity: Development and efficacy of an online intervention. Psychol Sex Orientat Gend Divers. 2019;

13. Shearer LS, Simmons L, Mindel A, Stanberry LR, Rosenthal SL. Reducing the stigma of herpes simplex virus infection: Lessons from an online video contest. Sex Health. 2012;9(5):438–44. pmid:23036138

14. Witzel TC, Guise A, Nutland W, Bourne A. It Starts with Me: Privacy concerns and stigma in the evaluation of a Facebook health promotion intervention. Sex Health. 2016; pmid:27075940

15. Redston S, de Botte S, Smith C. Resolving embarrassing medical conditions with online health information. Int J Med Inf. 2018; pmid:29605386

16. Abdulai AF, Howard AF, Currie LM. Stigmatizing and de-Stigmatizing Properties of Web Apps for Sexual Health-Related Conditions: A Scoping Review. Int J Sex Health. 2021;

17. Abdulai AF, Howard AF, Noga H, Yong PJ, Currie LM. Application of Anti-Stigma Design Heuristics for Usability Inspection. Stud Health Technol Inform. 2021;284:239–43. pmid:34920518

18. Interaction Design Foundation. What are Design Principles? | Interaction Design Foundation [Internet]. 2020 [cited 2020 Aug 12]. Available from: https://www.interaction-design.org/literature/topics/design-principles

19. Vaes K. Design for Empowerment, the Stigma-Free Design Toolkit. In: Advances in Intelligent Systems and Computing. 2019.

20. Abdulai AF, Howard AF, Yong PJ, Noga H, Parmar G, Currie LM. Developing an Educational Website for Women With Endometriosis-Associated Dyspareunia: Usability and Stigma Analysis. JMIR Hum Factors. 2022;9(1).

21. Humphrey-Murto S, Varpio L, Gonsalves C, Wood TJ. Using consensus group methods such as Delphi and Nominal Group in medical education research*. Med Teach. 2017;

22. Strudwick G, Nagle LM, Morgan A, Kennedy MA, Currie LM, Lo B, et al. Adapting and validating informatics competencies for senior nurse leaders in the Canadian context: Results of a Delphi study. Int J Med Inf. 2019; pmid:31445258

23. Harris M, Fallot RD. Envisioning a trauma-informed service system: a vital paradigm shift. New Dir Ment Health Serv. 2001; pmid:11291260

24. Hopper EK, Bassuk EL, Olivet J. Shelter from the Storm: Trauma-Informed Care in Homelessness Services Settings. Open Health Serv Policy J. 2009;

25. Chen JX, McDonald A, Zou Y, Tseng E, Roundy KA, Tamersoy A, et al. Trauma-Informed Computing: Towards Safer Technology Experiences for All. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems [Internet]. New York, NY, USA: Association for Computing Machinery; 2022 [cited 2022 Dec 30]. p. 1–20. (CHI ‘22). Available from: https://doi.org/10.1145/3491102.3517475

26. Frieh EC. Stigma, trauma and sexuality: the experiences of women hospitalised with serious mental illness. Sociol Health Illn. 2020; pmid:31758579

27. Sockolow P, Joppa M, Zhu J. Assessment of evaluation frameworks for design of a sexual risk prevention game for black adolescent girls. In: Studies in Health Technology and Informatics. 2018. pmid:29857398

28. Shehadeh K, Arman N, Khamayseh F. Semi-Automated Classification of Arabic User Requirements into Functional and Non-Functional Requirements using NLP Tools. In: 2021 International Conference on Information Technology, ICIT 2021—Proceedings. 2021.

29. Goffman E. Stigma; Notes on the management of spoiled identity. JASON ARONSON N Y N. 1974;

30. Meisenbach RJ. Stigma management communication: A theory and agenda for applied research on how individuals manage moments of stigmatized identity. J Appl Commun Res. 2010;

31. Heijnders M, Van Der Meij S. The fight against stigma: an overview of stigma-reduction strategies and interventions. Psychol Health Med. 2006; pmid:17130071

32. Link BG, Phelan JC. Conceptualizing Stigma. Annu Rev Sociol. 2001;

33. Chen Y. Exploring Design Guidelines of Using User-Centered Design in Gamification Development: A Delphi Study. Int J Hum-Comput Interact. 2019;

34. Carmody KW. Exploring serious game design heuristics: A delphi study. Diss Abstr Int Sect Humanit Soc Sci. 2013;74(6-A(E)):No-Specified.

35. Dajani JS, Sincoff MZ, Talley WK. Stability and agreement criteria for the termination of Delphi studies. Technol Forecast Soc Change. 1979;

36. Vermeeren APOS, Roto V, Väänänen K. Design-inclusive UX research: Design as a part of doing user experience research. Behav Inf Technol. 2016;

37. Schmidt BJ, McArthur EC. Professional nursing values: A concept analysis. Nurs Forum (Auckl). 2018;53(1). pmid:29419942

38. Friedman AL, Bozniak A, Ford J, Hill A, Olson K, Ledsky R, et al. Reaching youth with sexually transmitted disease testing: Building on successes, challenges, and lessons learned from local get yourself tested campaigns. Soc Mark Q. 2014;

39. Hood JE, Friedman AL. Unveiling the hidden epidemic: A review of stigma associated with sexually transmissible infections. Sex Health. 2011; pmid:21592429

40. Rocha-Jiménez T, Morales-Miranda S, Fernández-Casanueva C, Brouwer KC, Goldenberg SM. Stigma and unmet sexual and reproductive health needs among international migrant sex workers at the Mexico–Guatemala border. Int J Gynecol Obstet. 2018; pmid:29328511

41. Cisu TI, Mingin GC, Baskin LS. An evaluation of the readability, quality, and accuracy of online health information regarding the treatment of hypospadias. J Pediatr Urol. 2019;15(1). pmid:30449679

42. Killip SC, Kwong NKR, Macdermid JC, Fletcher AJ, Carleton NR. The quality, readability, completeness, and accuracy of ptsd websites for firefighters. Int J Environ Res Public Health. 2020;17(20). pmid:33086772

43. Hightow-Weidman LB, Muessig KE, Pike EC, LeGrand S, Baltierra N, Rucker AJ, et al. HealthMpowerment.org: Building Community Through a Mobile-Optimized, Online Health Promotion Intervention. Health Educ Behav. 2015; pmid:25588932

44. Jordan PW. Inclusive design: An holistic approach. In: Proceedings of the XIVth Triennial Congress of the International Ergonomics Association and 44th Annual Meeting of the Human Factors and Ergonomics Association, “Ergonomics for the New Millennium.” 2000.

45. Jordan PW. Designing pleasurable products- An induction to new human factors.pdf. Des Issues. 2002;18(1).

46. Brewer LPC, Fortuna KL, Jones C, Walker R, Hayes SN, Patten CA, et al. Back to the future: Achieving health equity through health informatics and digital health. JMIR MHealth UHealth. 2020;8(1).