Usability Evaluation of a Web-Based Support System for People With a Schizophrenia Diagnosis

Abstract

Background: Routine Outcome Monitoring (ROM) is a systematic way of assessing service users’ health conditions for the purpose of better aiding their care. ROM consists of various measures used to assess a service user’s physical, psychological, and social condition. While ROM is becoming increasingly important in the mental health care sector, one of its weaknesses is that ROM is not always sufficiently service user-oriented. First, clinicians tend to concentrate on those ROM results that provide information about clinical symptoms and functioning, whereas it has been suggested that a service user-oriented approach needs to focus on personal recovery. Second, service users have limited access to ROM results and they are often not equipped to interpret them. These problems need to be addressed, as access to resources and the opportunity to share decision making has been indicated as a prerequisite for service users to become a more equal partner in communication with their clinicians. Furthermore, shared decision making has been shown to improve the therapeutic alliance and to lead to better care.

Objective: Our aim is to build a web-based support system which makes ROM results more accessible to service users and to provide them with more concrete and personalized information about their functioning (ie, symptoms, housing, social contacts) that they can use to discuss treatment options with their clinician. In this study, we will report on the usability of the web-based support system for service users with schizophrenia.

Methods: First, we developed a prototype of a web-based support system in a multidisciplinary project team, including end-users. We then conducted a usability study of the support system consisting of (1) a heuristic evaluation, (2) a qualitative evaluation and (3) a quantitative evaluation.

Results: Fifteen service users with a schizophrenia diagnosis and four information and communication technology (ICT) experts participated in the study. The results show that people with a schizophrenia diagnosis were able to use the support system easily. Furthermore, the content of the advice generated by the support system was considered meaningful and supportive.

Conclusions: This study shows that the support system prototype has valuable potential to improve the ROM practice and it is worthwhile to further develop it into a more mature system. Furthermore, the results add to prior research into web applications for people with psychotic disorders, in that it shows that this group of end users can work with web-based and computer-based systems, despite the cognitive problems they experience.

Introduction

Routine Outcome Monitoring (ROM) has become increasingly important in the mental health care sector. Although there is no universal definition, ROM can be described as the use of standard instruments to systematically and continuously assess aspects of mental health service users’ health for the purpose of better aiding their care [1]. The format of ROM varies between countries, but it usually consists of several quantitative measures used to assess a service user’s physical, psychological, and social condition. ROM is carried out for service users with a single diagnosis and short-term problems, as well as people with a severe mental illness. This latter group includes service users diagnosed with schizophrenia.

Schizophrenia is a mental disorder characterized by cognitive dysfunctions and abnormalities in perception of reality. People diagnosed with schizophrenia often experience hallucinations, delusions, and disorganized speech and thinking, accompanied by significant social and occupational problems [2]. Due to the complexity of this disorder and the diversity of care needed for service users diagnosed with schizophrenia, proper and frequent evaluation of treatment is particularly vital. That is why ROM offers much potential for better care of these people [3].

However, the effects of ROM on mental health care have been mixed. On the one hand, research shows that the use of outcome measures, combined with adequate feedback, helps clinicians to recognize and anticipate problems in individual treatment processes and to provide better care as a result [4-6]. On the other hand, ROM is not always used to in a way that empowers service users and improves shared decision making between service user and clinician [7,8]. One problem is that clinicians tend to concentrate on those ROM results that provide information about clinical symptoms and functioning. However, service user-oriented approaches promote a focus on personal recovery, which reflects the importance of finding meaning and giving value to personal experiences [7]. A second problem is that service users have limited access to ROM results and they are often not equipped to interpret them [8,9]. These problems need to be addressed, as research has shown that access to resources and the opportunity to share decision making has been indicated as a prerequisite for service users to become a more equal partner in communication with their clinicians [10,11]. Furthermore, shared decision making has been shown to improve the therapeutic alliance, and to lead to better care and treatment [12,13].

Since 2007, ROM assessments have been a regular element in care for people with psychotic disorders in the northern provinces of the Netherlands. The ROM protocol (called PHAMOUS), which is specifically developed for psychotic disorders, consists of a physical investigation (eg, weight, height, waist measurement, and glucose levels), multiple interviews and questionnaires concerning psychiatric and psychosocial issues, and service user satisfaction [14]. All service users with schizophrenia who receive care from any mental health care organization involved take part in ROM assessment at least once a year. After completion of the assessment, the parameters of the ROM assessment are uploaded into a central database by clinicians and research nurses via a link in the patient’s service user’s electronic file. Currently, the ROM-results are only reported to clinicians. Clinicians are supposed to discuss the results with their patients so that they can mutually decide whether the course of treatment needs readjustment [15]. However, a large percentage of service users do not receive adequate feedback concerning their ROM-results, as clinicians are not yet accustomed to discussing ROM results with service users [16].

In an attempt to improve ROM practice and to increase potential for service user empowerment, we developed a prototype of a web-based support system that provides service users diagnosed with schizophrenia with personalised advice, based on their ROM results. By means of this support system, the current problems with ROM practice may be partly tackled. The personalized advice provides users with accessible information about their ROM results, which may enable them to participate in shared decision making, and pave the way to better care. Prior research has shown that people with psychotic disorders can work with web-based and computer-based systems, despite the severity of their symptoms [eg, 17-21]. Findings are, however, inconsistent as to the amount of support service users need in working with computers (eg, [18] versus [21]).

In the present study, we extended the existing research by investigating the usability of a web-based support system for ROM. We examined whether our support system can make ROM-results more accessible to service users and provide them with more concrete information that they can use to discuss their personal goals with their clinician. The aim of this paper is to provide a brief overview of the web-based system and to report on its usability from the perspective of service users with schizophrenia.

Methods

Content and Technology of the Web-Based Support System Prototype

The prototype of the web-based support system is called WEGWEIS, which is a Dutch abbreviation that stands for web environment for empowerment and individual advice. The WEGWEIS support system offers users advice about various topics related to psychiatric treatment, rehabilitation, and personal recovery. This advice is based on the service user’s ROM assessment results, as conducted in the northern provinces of the Netherlands. The support system is a website, which can be accessed by entering a username and a password (see Multimedia Appendix 1). The system is to be used by service users at home or in a clinical setting (eg, a community hospital).

When building the prototype, we focussed on two important and widely used ROM measures, namely the clinician-rated Health of the Nation Outcome Scales [22], which measures health and social functioning, and the service user-rated Manchester Short Assessment of Quality of Life [23], which measures quality of life. Based on item scores of these measures and using innovative algorithms combined with ontological reasoning, the system identifies specific health care problems for each individual service user and provides relevant and tailored advice [24]. The algorithms are innovative because they break with conventional case-based reasoning approaches in that they decouple symptoms from outcomes, allowing the outcomes to be dynamic [24]. The content of the advice consists of information derived from evidence-based research (eg, the Dutch Multidisciplinary Guideline for Schizophrenia), clinical expertise, and service user experiences.

When, for example, the ROM results indicate that a service user is experiencing physical problems, the system offers advice indicating that physical problems can be a side effect of medication, referring to the Dutch Multidisciplinary guideline for schizophrenia. Furthermore, the advice suggests that side effects may be resolved by adjustment of the medication. Service users are also referred to their psychiatrist – by name – for more information (see Multimedia Appendix 2). When service users appear to experience problems with personal safety, they are provided information about and linked to the local patient counsellor. They also have the opportunity to read about experiences of other service users (see Multimedia Appendix 3). In another example, service users who are troubled by hearing voices are provided a video showing someone suffering from the same condition and offering information about treatment options (see Multimedia Appendix 4). More information about the advice can be found elsewhere [25]. The algorithm for advice selection, as well as a brief overview of system design and architecture are presented elsewhere [24].

The prototype is created with open source software, using the Ruby on Rails Web-framework (

References

Trauer T. Introduction. In: Trauer T, editor. Outcome measurement in mental health: theory and practice. Cambridge: Cambridge University Press; 2010:1-14.

American Psychiatric Association. Diagnostic and statistical manual of mental disorders: DSM-IV. Washington, DC: American Psychiatric Association; 2000.

Opler LA, Ramirez PM, Mougios VM. Outcome measurement in serious mental illness. In: Ishak WW, Burt T, Sederer LI, editors. Outcome measurement in psychiatry: a critical review. Washington, DC: American Psychiatric Pub.; 2002:139-154.

Lambert MJ, Hansen NB, Finch AE. Patient-focused research: using patient outcome data to enhance treatment effects. J Consult Clin Psychol 2001 Apr;69(2):159-172.

Lambert MJ, Harmon C, Slade K, Whipple JL, Hawkins EJ. Providing feedback to psychotherapists on their patients' progress: clinical results and practice suggestions. J Clin Psychol 2005 Feb;61(2):165-174.

Whipple JL, Lambert MJ. Outcome measures for practice. Annu Rev Clin Psychol 2011 Apr;7:87-111.

Lakeman R. Standardized routine outcome measurement: pot holes in the road to recovery. Int J Ment Health Nurs 2004 Dec;13(4):210-215.

Guthrie D, McIntosh M, Callaly T, Trauer T, Coombs T. Consumer attitudes towards the use of routine outcome measures in a public mental health service: a consumer-driven study. Int J Ment Health Nurs 2008 Apr;17(2):92-97.

Happell B. Meaningful information or a bureaucratic exercise? Exploring the value of routine outcome measurement in mental health. Issues Ment Health Nurs 2008 Oct;29(10):1098-1114.

GGZ Nederland. Naar herstel en gelijkwaardig burgerschap. Visie op de (langdurende) zorg aan mensen met ernstige psychische aandoeningen. URL: http://www.ggznederland.nl/scrivo/asset.php?id=305955 [accessed 2011-12-09] [WebCite Cache]

Deegan PE. Recovery and empowerment for people with psychiatric disabilities. Soc Work Health Care 1997;25(3):11-24.

Mahone IH, Farrell S, Hinton I, Johnson R, Moody D, Rifkin K, Region Ten CSB/UVA School of Nursing Partnership. Shared decision making in mental health treatment: qualitative findings from stakeholder focus groups. Arch Psychiatr Nurs 2011 Dec;25(6):e27-e36.

Frank AF, Gunderson JG. The role of the therapeutic alliance in the treatment of schizophrenia. Relationship to course and outcome. Arch Gen Psychiatry 1990 Mar;47(3):228-236.

PHAMOUS. Pharmacotherapy monitoring and outcome survey URL: http://www.phamous.eu/home.html [accessed 2011-12-08] [WebCite Cache]

Makkink S, Kits L. Herstellen doe je zelf. In: Van Hees S, Van der Vlist P, Mulder N, editors. ROM. Van weten naar meten, ROM in de GGZ. Amsterdam: Boom; 2011:97-108.

Schaefer B, Nijssen Y, Van Weeghel J. Van meten naar oplossingsgericht werken. In: Van Hees S, Van der Vlist P, Mulder N, editors. ROM. Van weten naar meten, ROM in de GGZ. Amsterdam: Boom; 2011:89-96.

Schrank B, Sibitz I, Unger A, Amering M. How patients with schizophrenia use the internet: qualitative study. J Med Internet Res 2010 Dec;12(5):e70

Kuosmanen L, Jakobsson T, Hyttinen J, Koivunen M, Välimäki M. Usability evaluation of a web-based patient information system for individuals with severe mental health problems. J Adv Nurs 2010 Dec;66(12):2701-2710.

Jones RB, Atkinson JM, Coia DA, Paterson L, Morton AR, McKenna K, et al. Randomised trial of personalised computer based information for patients with schizophrenia. BMJ 2001 Apr 7;322(7290):835-840.

Rotondi AJ, Anderson CM, Haas GL, Eack SM, Spring MB, Ganguli R, et al. Web-based psychoeducational intervention for persons with schizophrenia and their supporters: one-year outcomes. Psychiatr Serv 2010 Nov;61(11):1099-1105.

Bickmore TW, Puskar K, Schlenk EA, Pfeifer LM, Sereika SM. Maintaining reality: relational agents for antipsychotic medication adherence. Interacting with Computers 2010;22(4):276-288.

Wing JK, Beevor AS, Curtis RH, Park SB, Hadden S, Burns A. Health of the Nation Outcome Scales (HoNOS). Research and development. Br J Psychiatry 1998 Jan;172:11-18.

Priebe S, Huxley P, Knight S, Evans S. Application and results of the Manchester Short Assessment of Quality of Life (MANSA). Int J Soc Psychiatry 1999;45(1):7-12.

Emerencia AC, Krieke van der L, Petkov N, Aiello M. Assessing schizophrenia with an interoperable architecture. 2011 Oct 28 Presented at: Assessing schizophrenia with an interoperable architecture. 20th ACM International Conference on Information and Knowledge Management MIXHS'11; 2011 Oct 28; Glasgow, Scotland, UK. p.82. In press.

van der Krieke L, Emerencia AC, Sytema S. An online portal on outcomes for Dutch service users. Psychiatr Serv 2011 Jul;62(7):803.

Rotondi AJ, Sinkule J, Haas GL, Spring MB, Litsche CM, Newhill CE, et al. Designing websites for persons with cognitive deficits: design and usability of a psychoeducational intervention for persons with severe mental illness. Psychological Services 2007;4(3):202.

Cooper A, Reimann R. Implementation models and mental models. In: Cooper A, Reimann R, editors. About Face 2.0: the essentials of user interface design. Indianapolis: Wiley Publishing; 2003:21-32.

Valimaki M, Anttila M, Hatonen H, Koivunen M, Jakobsson T, Pitkanen A, et al. Design and development process of patient-centered computer-based support system for patients with schizophrenia spectrum psychosis. Inform Health Soc Care 2008 Jun;33(2):113-123.

Stephanidis C. editor. User interfaces for all. Concepts, methods, and tools. Mahwah: Lawrence Erlbaum Associates; 2001.

Hennink M, Hutter I, Bailey A. Participant recruitment. In: Hennink M, Hutter I, Bailey A, editors. Qualitative Research Methods. London: Sage Publications Ltd; 2011:81-107.

Nielsen J. What is usability? In: Usability engineering. San Francisco, Calif.: Morgan Kaufmann Publishers; 1993:23-48.

Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. In: Proceedings of the INTERACT'93 and CHI'93 Conference on human factors in computing systems. 1993 Presented at: INTERACT'93 and CHI'93 conference; 24-29 April 1993; Amsterdam, The Netherlands;.

Nielsen J. Heuristic evaluation. In: Nielsen J, Mack RL, editors. Usability inspection methods. New York: Wiley; 1994:25.

Hennink M, Hutter I, Bailey A. Observation. In: Qualitative Research Methods. London: Sage Publications Ltd; 2011:169-200.

Van De Belt TH, Engelen LJ, Berben SA, Schoonhoven L. Definition of Health 2.0 and Medicine 2.0: a systematic review. J Med Internet Res 2010;12(2):e18

Internet World Stats. Top 58 countries with highest penetration rates URL: http://www.internetworldstats.com/stats9.htm [accessed 2011-10-20] [WebCite Cache]