APPLICABILITY OF AN ASSESSMENT MODEL FOR HEALTHCARE INFORMATION SYSTEMS IN A PUBLIC HOSPITAL

Abstract

Assessment processes are essential to guarantee quality and continuous improvement of software in healthcare, as they measure software attributes in their lifecycle, verify the degree of alignment between the software and its objectives and identify unpredicted events. This article analyses the use of an assessment model based on software metrics for three healthcare information systems from a public hospital that provides secondary and tertiary care in the region of Ribeirao Preto. Compliance with the metrics was investigated using questionnaires in guided interviews of the system analysts responsible for the applications. The outcomes indicate that most of the procedures specified in the model can be adopted to assess the systems that serves the organization, particularly in the attributes of compatibility, reliability, safety, portability and usability.

Headnote

ABSTRACT

Assessment processes are essential to guarantee quality and continuous improvement of software in healthcare, as they measure software attributes in their lifecycle, verify the degree of alignment between the software and its objectives and identify unpredicted events. This article analyses the use of an assessment model based on software metrics for three healthcare information systems from a public hospital that provides secondary and tertiary care in the region of Ribeirao Preto. Compliance with the metrics was investigated using questionnaires in guided interviews of the system analysts responsible for the applications. The outcomes indicate that most of the procedures specified in the model can be adopted to assess the systems that serves the organization, particularly in the attributes of compatibility, reliability, safety, portability and usability.

Keywords: Health information systems; Quality assessment; Software metrics.

1.INTRODUCTION

The global health observatory data repository, from the WHO, with information about healthcare investments in over 190 countries, shows a rising curve of expenses per capita in health (WHO,2013). According to Newell (2011), the costs for funding healthcare have grown globally due to factors such as increased life expectancy, advances in healthcare technology and policies for universal access to healthcare, in spite of government actions to mitigate budgetary impacts, with public budget constraints, above all after the economic crises that have occurred on a global scale, since 2008. In countries where healthcare systems are private or mixed, they also try to minimize these costs, to make insurance and health plan operators economically feasible.

In healthcare organizations, healthcare information systems (HIS) add information technology and communication to address their processes (Ammenwerth et al., 2004) and integrate people, procedures and technologies to collect, store, manipulate and recover information (Wager, Lee & Glaser, 2009). They are characterized by complex and multidisciplinary deployment, produce impacts in learning and in the adaptation to organizational routine and involve several groups of stakeholders - patients, service providers, regulating agents and professionals (Fichman, Kohli& Krishnan, 2011).

Investments in information systems can constitute part of the healthcare organization policies to reduce the tension between costs and budgets, in order to improve efficiency and quality in the processes that occur in this sector. The information systems improve healthcare organization efficiency, reduce medical prescription error rates, help professionals and managers in decision-making and in preventive medicine (Hillestad et al., 2006; Ammenwerth et al., 2003) and have great potential to reduce costs and improve healthcare outcomes (Fichman, Cohli&Krisnan, 2011; Oliveira et al., 2011). Research outcomes on the assessment of information systems for the healthcare sector also show how administrators recognize the importance of information systems as critical resources and that there is great demand to align information systems to the management process (D'Souza &Sequeira, 2011).

The development and maintenance of healthcare information systems are complex activities, due to: (a) lack of standardization and interoperability difficulties between applications (Hillestad et al.,2006), (b) the interdisciplinary characteristic of healthcare that demands added knowledge from several user professionals in the construction of information systems (Fichman, Kohli& Krishnan, 2011; Carvalho&Eduardo, 1998) and (c) the fragmented nature of the healthcare sector and the difficulties to systematize processes in applications (Abouzahr&Boerma, 2005), besides the actual change in paradigm, of a reactive model, centered on the disease, to a preventive model, that makes communication flow difficult among the three levels of attention and in continuous attention (OPAS, 2011).

Within this context of complexity, assessment is an essential activity to guarantee healthcare software quality as well as its continuous enhancement. Software assessment activities measure the attributes of a system in several phases of its lifecycle, help in the optimization of outcomes, identify unpredicted events (Ammenwerth et al.,2004) and allow us to analyze the degree to which information systems address their objectives (Yusofe al., 2008).

This article specifies 42 software metrics as a technique to measure quality attributes established in a structured model guided towards healthcare information system assessment and verifies the feasibility of these metrics in applications that offer support to clinical, outpatient and administrative processes in a public hospital that serves the macro region of Ribeiräo Preto.

2.THE QUALITY ASSESSMENT MODEL

The model proposed by Morais & Costa (2013), presented in Figure 1 was used as a theoretical reference. The model uses attributes from the product's quality dimension in Rule ISO/IEC 25010 (ISO, 2011a), which includes eight quality characteristics: functional supportability, performance efficiency, compatibility, usability, reliability, safety, maintainability and portability. Each characteristic is made up of a set of sub-characteristics, which are described in Attachment I.

Due to the degree of subjectivity found in the sub-characteristics - inherent to the model, as it can be applied to any software product - each attribute was associated to a set of indicators, obtained in a systematic research process in databases, which selected 32 indicators from seven relevant papers: Paiand Huang (2011), Viitanen et al.(2011), HubnerBloderandAmmenwerth (2009), Ribière et al.(1999), Otieno et al.(2008), Anderson and Aydin(2005) and Lima et al.(2009).

Using a semantic analysis, each indicator was classified into a characteristic and a subcharacteristic of the product's quality dimension from Rule ISO/IEC 25010 (ISO, 2011a). Another 10 indicators were added by the authors based on the reading of documents ISO/IEC 25023 (ISO, 2011b) and ISO/IEC 9123-3 (ISO, 2003). The inclusion of these indicators covered all the quality dimension characteristics of the product, thus establishing the framework.

For each indicator, the model proposes assessment questions and/or software inspection procedures, applicable to different stakeholders, to objectively assess the indicator. The model includes the following as stakeholders:

-Managers: users of the system at its strategic level;

-Health Professionals: users of the system at tactical and operational levels;

- Patients/users: access the system only for queries;

- IT Professionals: compose the technical staff for the application, such as developers and/or maintainers of the information systems.

The assessment questions were elaborated and grouped into questionnaires guided towards user profiles (managers, health professionals and patients) and are specified in Morais & Costa (2013). The elaboration of questions was guided in the writing of clearlyexpressed texts without ambiguities and in language that was easy to understand. Measuring scales were also developed observing clear and appropriate reading and coverage of the universe of possible answers with uniform distribution. Ordinal and interval scales were used, according to the definition from Malhotra (2006), with five response options organized in increasing order of adequacy.

The inspection procedures proposed in the model must be directed towards IT professionals who maintain applications, stakeholders who have access to information about the requirements of the systems, track record of changes, track records of defects and system failures and other relevant information to obtain the measurements predicted. These procedures must use the documental analysis, tests/simulations of software use or performance measurements/essays as ways of obtaining measurements.

Attachment II describes the detailing and specifications of the model's assessment questions and inspection procedures. The first column of Attachment II indicates the stakeholder users of the application for which the assessment questions were guided towards. The metrics specified in the last column are directed towards the application maintainers.

When including questionnaires directed to the application users and inspection procedures guided towards maintainers, one distinguishing characteristic of this model is the fact that it "listens to both sides". Many times, conflicts are established due to lack of communication (Dallavalle, 2000), the distance between the organization and the service provider (Albertin& Moura, 1995) or due to inadequate implantation processes identified by Caldas & Wood (2000), as acquisition of software without clear-cut criteria, by imposition or low involvement of the user.

3.METHOD

The work done was qualitative and it was divided into a first conceptual phase, with the specification of inspection procedures for the model used and a second empirical phase, with the investigation of compliance to the metrics obtained in the first phase for three applications in a public hospital offering regional medium and high complexity services.

The first phase of this work is classified as methodological research as it refers to an instrument that takes in the reality studied: according to Vergara (2005, p. 47), the methodological research "is associated to paths, forms, ways, procedures to reach a certain end," characterized in this study.

The specifications described in this first phase used the following documents ISO/IEC 25023 (ISO, 2011b) and ISO/IEC 9126-3 (ISO, 2003) as a basis with context adaptations for healthcare information systems. The scales for the metrics specified in the inspection procedures are valued between zero and one: the closer they are to one, the greater the compliance of the system with the specified metric.

A survey was made in the second phase of the research using interviews guided towards system analysts responsible for the applications. A questionnaire was answered in the interviews, where the interviewer asked about the feasibility of each specified metric for the application maintained by the system analyst, with the response options "feasible/unfeasible/not applicable to the context". The option "not applicable to the context" refers to cases where the interviewee assesses that the metrics proposed were incongruous with the target system. Remarks from professionals were noted, regarding feasibility or nonfeasibility of the inspection procedures.

The interviews with the application maintainers were carried out in July 2013, in sessions of 90 minutes' maximum and included part of the time for suggestions given by the professionals. As a limitation, it must be observed that these studies were restricted to the analysis of procedures only for the stakeholders included in the assessment model, to verify and describe the applicability of the metrics proposed in the model. An assessment and application of instruments guided towards other stakeholders is a proposal for future research work.

4.OUTCOMES

4.1. The specifications of the software inspection procedures

Chart 1 describes the indicators and software metrics inspection procedures specified for the assessment model. The first column numbers its 38 indicators, classified in the model's characteristics and sub-characteristics. Inspection procedures were not specified for four indicators: "Extensive training is not needed to learn about the system", "The interface (screens, forms, data entry, reports or graphs) as well as all the terms and concepts used in the system are clear and have no ambiguities", "The system is easy to use, intuitive" and "The system presents a uniform and standardized interface", all of them have the usability characteristic. Psychometric assessment questions guided towards users were elaborated for these indicators, to answer how many system functions comply with the indicators, using an ordinal scale of five points: 0% (none), up to 25% (few), between 25 and 75% (about half), more than 75% (most) and 100% (all).

The second column of Chart 1 describes the metric specifications for the indicators assessed using inspection procedures. 37 procedures are based on the use of metrics from the document ISO/IEC 25023(ISO, 2011b), four are based on rule ISO/IEC 9126-3 (ISO, 2003) and one was proposed by the author. Every metric is labeled with an abbreviation of the characteristic it refers to and numbered sequentially.

4.2. Feasibility study of inspection procedures used in assessment model

In this phase, a survey was carried out with information technology professionals from Hospital das Clínicas in the Medical School of Ribeiräo Preto (HCFMRP-USP), to investigate the applicability of inspection procedures specified for the proposed assessment model, according to that specified in the method section.

The survey was performed at the Information and Analysis Center (CIA - Centro de Informaçoes e Análises), the department responsible for information technology management at the hospital. The CIA provides the hospital with its own IT development and infrastructure that was gradually organized as from 1995, when the applications maintained by PRODESP were migrated. PRODESP previously offered information technology support for the hospital.

Currently, the CIA maintains 65 information systems in deployment or operation, 55 developed internally and 10 contracted from third-parties. As collaborators, it has 20 business and system analysts, 2 software quality engineers, 3 network administrators and 1 project manager, as well as a group of information technicians to provide support and service to the users.

The applications are developed in the Microsoft.Netplatform and the Oracle database manager, they use a UML (Unified Modelling Language) for analysis specifications and prototype techniques and tools for survey and specification of requirements. Some legacy Delphi systems are still maintained gradually being submitted to reengineering for technological update and review of functionalities.

The Senior Management of HCFMRP has sponsored policies to promote improvement of the software development process quality, with SBIS/CFM (SBIS, 2013) and MPS-BR (SOFTEX, 2013) certifications, to improve IT infrastructure development enhancement.

For the survey on applicability of inspection procedures proposed, three information systems were investigated. They support clinical and hospital processes at HCFMRP:

* Laboratory Information System (LIS - Sistema de Informaçoes Laboratoriais): offers support for management of scheduled examinations and performed in the different clinical analysis laboratories at the hospital.

* Clinical Service Elaboration (EAC -Elaboraçao de Atendimento Clínico): shows the observations and evolutions in patient care pointed out by doctors, paramedics and nurses, through electronic forms.

* Surgical Procedure Management (CIRÚRGICO-3 - Gerenciamento de Procedimentos Cirúrgicos): for surgeries performed at the hospital, this system manages waiting lists, scheduling, notes on surgical procedures, issuing of surgical records, consumption of material as well as analytical and managerial reports.

Chart 2 describes the outcomes obtained in the survey performed using questionnaires, where the system analysts responsible for the applications were interviewed, according to that described in the method section. The chart enumerates the metrics for each quality attribute. Note that the same names attributed to the metrics in Chart 1 were used. The affirmative answers are indicated with the symbol "?", while the negative ones with "?". The symbol "-" indicates that the metric does not apply to the system context and "?"indicates that the question was not answered by the interviewee. The last column summarizes the explanations from the respondents about compliance with procedures.

In absolute numbers, 69 feasibility indications were computed for the metrics proposed against 40 indications of non-feasibility (there were also 7 comments about absence of context of use and 10 questions were not answered). Graph 1 shows the distribution of answers per quality characteristic.

As shown in graph 1, there was a positive consensus for all the metrics of Compatibility, Safety and Portability characteristics (fully compliant) and for most of the Usability sub-characteristics (partially compliant). It was observed that the applicability of metrics for Functional Supportability were conditioned to the existence of functional requirement documentation for the applications - not always available for the applications, which resulted in partial compliance with the procedures, as well as with two Usability metrics.

For the Reliability metric group, there was also a positive consensus, with the exception of the metric that assesses component redundancy, as there was no specification on the non-functional requirements of the infrastructure. For the metric MTBF (Media Time Between Failures), it was correctly observed that the procedure does not consider the size of the system and that the number of application functions must consider the measurement for calculation.

The absence of non-functional requirement specifications for response time, authentication time and simultaneous accesses made it unfeasible to use all the Performance Efficiency metrics. For the Maintainability characteristic, there was little feasibility: the outcomes show that the Testability depends on the availability of text cases for each system and indicate that some metric specifications in these groups must be reviewed, for better clarity, as this group concentrated the largest number of non-answered questions.

As suggested by this outcome, the analysts interviewed agree that many of the specified procedures can be adopted for assessment of the systems that address HCFMRP, also that they are aligned with the specifications recommended by certification SBIS/CFM (SBIS, 2013), recently implanted in the IT development and infrastructure areas.

5.FINAL REMARKS

For an assessment model for healthcare information systems based on the quality dimension of the product in rule ISO/IEC 25010 and in quality indicators researched in literature, this article describes the specification of a set of software inspection procedures associated to the indicators proposed and assesses the feasibility of these procedures at HCFMRP, a public hospital with regional coverage.

The architecture of the assessment model has as a distinguishing factor the application of assessment instruments to stakeholders with conflicting interests: its architecture predicts questionnaires directed towards different user profiles of the system and inspection procedures, with collection of software metrics with the system maintainers, using documental analysis, tests and software simulation use. This work includes as a study object the second group of assessment instruments for the model, directed towards application maintainers.

The feasibility study developed for systems that address HCFMRP suggests that most of the specified procedures are applicable to the hospital context, particularly the metrics for the attributes Compatibility, Reliability, Safety, Portability and Usability. A longitudinal measurement strategy, with a view of historical assessment data can guide improvement processes and reengineering of systems.

Future work includes assessment together with other stakeholders not included in the assessment, other assessment studies of applications and the submission of the model and outcomes of assessments to specialist panels in healthcare information, organized in focal groups, for validation and adjustments of the framework developed.

When covering the main quality aspects of the healthcare information system domain, with a holistic range of indicators, questionnaires and software metrics, this work can contribute as another reference of studies that involve assessment processes of the technical quality of healthcare software and other areas of application, with adaptations.

Many managers have as a motto that everything that is managed must be measured - after all, measuring allows for quantification and, consequently, more effective management. When identifying a demand to discuss the quality of healthcare information systems, it is also expected that the outcome of this work can add content and/or provide subsidies for projects that deal with standardization of assessment plans and monitoring of system quality and in enhancement projects of these software assets.

References

REFERENCES

Abouzahr, C., &Boerma, T. (2005).Health information systems: the foundations of public health".Bulletin of the World Health Organization, 83, 578-583. Retrieved from http://www.who.int/bulletin/volumes/83/8/en/index.html

Albertin, A.L.,& Moura, R.M. (1995). Administraçâo de informática e seus fatores críticos de sucesso no setor bancário privado nacional.Revista de Administraçâo de Empresas, 35(5), 2229.

Ammenwerth, E., Gräber, S., Herrmann, G., Bürkle, T., &König, J. (2003). Evaluation of health information systems-problems and challenges. International journal of medical informatics, 71(2), 125-135.

Ammenwerth, E., Brender, J., Nykänen, P., Prokosch, H. U., Rigby, M., &Talmon, J. (2004). Visions and strategies to improve evaluation of health information systems: Reflections and lessons based on the HIS-EVAL workshop in Innsbruck. International journal of medical informatics, 73(6), 479-491.

Anderson J.,&Aidyn, C. (2005).Evaluating the organizational impact of healthcare information systems .New York:Springer.

Caldas, M. P., &Wood, T. J. (2000). Fads and fashions in management: the case of ERP. Revista de Administraçâo de Empresas, 40(3), 8-17.

Carvalho, A. O., &Eduardo, M. B. P. (1998).Sistemas de Informaçâo em Saúde para Municipios. Série Saúde & Cidadania. v. 6. Säo Paulo:Faculdade de Saúde Pública da Universidade de Säo Paulo.

D'Souza, S. C., &Sequeira, A. H. (2011).Information Systems and Quality Management in Healthcare Organization: An Empirical Study. Journal of Technology Management for Growing Economies, 2(1), 47-60.

Dallavalle, S. I. (2000).Regras de Negocio, um fator chave de sucesso noprocesso de desenvolvimento de sistemas de informaçâo .Anais do XX ENEGEP - Encontro Nacional de Engenharia de Produçâo. Säo Paulo.

Fichman, R. G.,Kohli, R., & Krishnan, R. (2011). The role of information systems in healthcare: current research and future trends.Information Systems Research, 22(3), 419-428.

Hillestad, R., Bigelow, J., Bower, A., Girosi, F., Meili, R., Scoville, R., & Taylor, R. (2005). Can electronic medical record systems transform health care? Potential health benefits, savings, and costs. Health Affairs, 24(5), 1103-1117.

Hübner-Bloder, G., &Ammenwerth, E. (2009). Key Performance Indicators to Benchmark Hospital Information Systems-A Delphi Study. Methods of Information in Medicine, 48(6), 508-518.

ISO (2011a).ISO/IEC 25010. Systems and software engineering -systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models. Geneva:International Organization for Standardization.

ISO (2003). ISO/IEC TR 9126-3. Software engineering - Product Quality - part 3: Internal metrics.Geneva:International Organization for Standardization.

ISO (2011b). ISO/IEC 25023. Systems and software engineering -systems and software Quality Requirements and Evaluation (SQuaRE) - Measurement of system and software product quality (in development). International Organization for Standardization.

Lima, C. R. A., Schramm, J. M. A., Coeli, C. M., & Silva, M. E. M. (2009). Revisäo das dimensöes de qualidade dos dados e métodos aplicados na avaliaçâo dos sistemas de informaçâo em saúde. Cadernos de saúde pública, 25(10), 2095-2109.

Malhotra, N. K. (2006). Pesquisa em Marketing: uma orientaçâo aplicada. Porto Alegre : Bookman.

Morais, R. M., & Costa, A.L. (2013). Avaliaçâo orientada a múltiplos stakeholders: um modelo para os sistemas de informaçâo do Sistema Único de Saúde. In Anais do XXXVII Encontro da ANPAD - EnANPAD 2013. Rio de Janeiro.

Newell, S. (2011). Special section on healthcare information systems. The Journal of Strategic Information Systems, 20(2), 158-160.

Oliveira, S. V. W. B., Arroyo, C. S., Oliveira, M. M. B., & Ferreira, C. H. (2011). Use and Development of Health Information Systems: The Experience of an Organizational Unit Responsible for the Technological Services atPublic Hospitals. Journal of Information Systems and Technology Management, 8(1) , 155-178.

OMS (2013). Global Health Observatory Data Repository. Retrieved fromhttp://apps.who.int/ghodata/?vid=1901

OPAS (2011). A atençâo à saúde coordenada pela APS: construindo as redes de atençâo no SUS.Brasília: Organizaçâo Pan-Americana da Saúde.

Otieno, G. O., Hinako, T., Motohiro, A., Daisuke, K., &Keiko, N. (2008). Measuring effectiveness of electronic medical records systems: towards building a composite index for benchmarking hospitals. International journal of medical informatics, 77(10), 657-669.

Pai, F. Y., & Huang, K. I. (2011). Applying the Technology Acceptance Model to the introduction of healthcare information systems. Technological Forecasting and Social Change, 78(4), 650-660.

Ribière, V., LaSalle, A. J., Khorramshahgol, R., & Gousty, Y. (1999). Hospital Information Systems Quality: A customer satisfaction assessment tool. In Proceedings of the 32nd Annual Hawaii International Conference on System Sciences, 1999. HICSS32. (pp. 7-pp). IEEE.

SBIS (2013). Manual de Certificaçâo para Sistemas de Registro Eletrônico em Saúde (SRES). Retrieved from http://www.sbis.org.br/certificacao/Manual_Certificacao_SBISCFM_2009_v3-3.pdf

SOFTEX (2013). MPS.BR - Melhoria de Processo do Software Brasileiro. Guia Geral. Retrieved from http://www.softex.br/mpsbr/_guias/guias/MPS.BR_Guia_Geral_2009.pdf

Vergara, S. C. (2005). Projetos e Relatórios de Pesquisa em Administraçâo. Säo Paulo:Atlas.

Viitanen, J., Hyppönen, H., Lääveri, T., Vänskä, J., Reponen, J., &Winblad, I. (2011). National questionnaire study on clinical ICT systems proofs: Physicians suffer from poor usability. International journal of medical informatics, 50(10), 708-725.

Wager, K. A., Lee, F. W., & Glaser, J. P. (2009). Health care information systems: a practical approach for health care management. Wiley. com.

Yusof, M. M., Papazafeiropoulou, A., Paul, R. J., &Stergioulas, L. K. (2008). Investigating evaluation frameworks for health information systems. international journal of medical informatics, 77(6), 377-385.