Quantitative analysis in a heuristic evaluation of web-based encyclopedia for children

Abstract

Anapedia is a web-based encyclopedia designed specifically for children. The web was built because there are still very few Indonesian-language encyclopedias for children. This paper presents an evaluation of the usability of the Anapedia website developed for children in Indonesia to retrieve information available in a wide range of areas. In the context of usability evaluation of children encyclopedia, this is the first study to analyze usability problems identified by experts during the heuristic inspection on their overall evaluation score. This evaluation, performed by seven usability experts, involved the identification and analysis of usability problems in the selected website. The assessment process was mainly based on the Sirius framework evaluation mechanism. The web considered as having high usability level, but they identified many usability problems for further improvement of the site. The most violated heuristic item was found to be help whereas the least violated heuristic item was labelling aspect.

Headnote

ABSTRACT

Anapedia is a web-based encyclopedia designed specifically for children. The web was built because there are still very few Indonesian-language encyclopedias for children. This paper presents an evaluation of the usability of the Anapedia website developed for children in Indonesia to retrieve information available in a wide range of areas. In the context of usability evaluation of children encyclopedia, this is the first study to analyze usability problems identified by experts during the heuristic inspection on their overall evaluation score. This evaluation, performed by seven usability experts, involved the identification and analysis of usability problems in the selected website. The assessment process was mainly based on the Sirius framework evaluation mechanism. The web considered as having high usability level, but they identified many usability problems for further improvement of the site. The most violated heuristic item was found to be help whereas the least violated heuristic item was labelling aspect.

Keywords:

Children encyclopedia Heuristic evaluation Sirius framework Web usability

(ProQuest: ... denotes formulae omitted.)

1. INTRODUCTION

Usability is one of the key success factors of a website that determines the ease or difficulty of users operating a website [1]-[3]. Besides the performance and reliability factors, usability is included as an important attribute of software quality [2]. According to International Organization for Standardization (ISO) 9241-11, usability is defined as the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use [4], [5]. Usability is related to user experience when interacting with the system including the aspect of learnability, understandability, and operability [6], [7]. The usability level of the software can be measured using an automated or manual approach using tools or evaluations from end users or experts. These approaches can identify usability issues in terms of efficiency, effectiveness, and user satisfaction. The choice of evaluation method greatly influences the accuracy of the results which will inform the quality of the measured software. Usability evaluation is not only needed as a reference for the quality of the website (the lower value will affect the efficiency of use and user satisfaction), but also contribute for decreasing the risk of the result [8].

The applications interface is usually designed based on the perspective of the developer, so it is often not according to user needs [9]. Therefore, an evaluation of the user interface needs to be done to ensure that the interface is in accordance with user requirements. Based on the assessment mechanism, there are three categories of usability evaluation, namely: 1) User-based method, where the system user is involved in the process of identifying usability issues. The purpose of this method is to measure the user performance or satisfaction when interacting with the system [10]; 2) Evaluator based method, where usability issues are identified by a group of evaluators. One is a heuristic evaluation that involves a group of experts to assess the user interface; and 3) Device-based method, where identification of problems related to the user interface is done using software [6]. The way this device works is to ensure that the system meets the specified usability standards.

Heuristic evaluation is the most popular method used in software usability measurement because it is fast and inexpensive. The method proposed by Nielsen and Molich involves a group of evaluators (generally numbering 3 to 5 people) who understand some of the principles of usability (known as 'heuristics') or who are usability experts [11]. These ten principles are still relevant and are widely used in evaluating software interfaces. Heuristic evaluation gives effective results when applied to software development iteratively even though it does not involve many evaluators, it is able to detect problems related to usability at the software interface [11], [12]. Heuristic evaluation methods can be applied to web-based software [13], [14] and mobile [15] both applying qualitative and quantitative approaches. Website evaluation using heuristic evaluation has been carried out by Rangraz, et al. [16] where heuristic evaluation is applied to evaluate the national health information system with the number of evaluators as many as five experts who judge based on the ten principles of Nielsen's usability; Inal [14] used a heuristic method to inspect the national library website where the evaluators involved were 57 students then combined with the System Usability Scale (SUS) and Net Promoter Score (NPS) methods. The heuristic evaluation used is able to detect most of the crucial usability issues in the system. However, there are not many studies that discuss the application of heuristic evaluation on children's encyclopedia web.

One of the heuristic evaluation methods that is comprehensive, empirical, and can be applied to websites with various types is the Sirius framework [17]. This framework covers ten aspects of assessment which includes general aspects, website identity and information, accomplishment of user interface, and search features. Furthermore, all aspects are then break down into 83 criteria [13], [17]. Compared with other quantitative heuristic evaluation methods [18]-[20], this framework can be used in evaluating several types of websites including educational websites, e-commerce, e-learning, institutions, government, and social media. In addition, the criteria are more comprehensive (representing all the principles of Nielsen's heuristic evaluation) and the results of the evaluation are quantitative values that state the level of website usability. Several studies have used this framework for evaluating usability including Aparna and Baseer [7] and Rodríguez et al. [21].

Anapedia is a web encyclopedia designed specifically for children [22]. The app was built because there are still very few Indonesian-language encyclopedias for children. Given the large number of schoolage children in Indonesia, this website will be able to support the learning process of children. Anapedia Web has been tested using black-box testing to measure the performance aspects of the application. Based on the previous research, users still do not feel comfortable when adopting Anapedia because of the user interface. Even though, usability aspects have a strong connection with user readiness [23]. Therefore, in this study, we analyze the user interface of Anapedia from a perspective of an education website using the Sirius framework. The research questions in this study are: 1) What is the usability level of Anapedia's website according to Sirius framework evaluation; and 2) What are the usability problems of Anapedia.

2. RESEARCH METHOD

This section explains research methodology to answer research questions. The overall process is shown in Figure 1. The methodology is divided into three phases. First stage, a literature review process was carried out to obtain an overview of usability evaluation both manually and automatically in the usability evaluation of the children's encyclopedia website. Sirius has 10 assessment aspects (as shown in Table 1) and each aspect has sub-criteria so that the total criteria are 83 items [13]. Framework on Figure 2 is able to produce global usability values based on expert judgment while providing information about usability errors from websites based on their type. This information is useful for the website developer to improve the interface.

The evaluators provide an assessment of all criteria covering 10 aspects as shown in Table 1. There are 2 rating scales, i.e: 1) using the numbers 0 to 10 where a value of 10 is given if the criterion is fully achieved and a value of 0 if it is not met or a value between 0 and 10 according to the level of achievement of the website; 2) using labelling as shown in Table 2, where the value of YES if a criterion is achieved by the web or not conform in the whole site (NWS) if the web does not meet the criteria or values other than both, namely not conform in the main links (NML), not sustained in the home page (NHP), and not conform in one or more subpages (NSP). Not applicable criteria (NA) if the evaluator decides that an assessment criteria cannot be applied to the web. Each criterion is given a weight (critical/major/moderate/minor) where the weight of each criterion is determined by the type of website to be evaluated. Based on the value given by the evaluator, global usability can be calculated using (1), where PU: usability procentage (%), nec: number of criteria evaluated (maximum number is 83, but not all of them are used depending on the type of website), sv: Sirius value (value of evaluator), wc: weighting coefficient (weighting for criteria calculated using (2). For (2), rv is the weight value of each criterion. If the evaluator is more than one person, then the global usability value is obtained from the average calculation.

... (1)

... (2)

In the second stage, the usability evaluation process uses two approaches that begin with developing an evaluation scenario. For automatic assessment, it is done by entering the Anapedia web URL (http://anapedia.org) into the Google PageSpeed web [24]. The manual assessment begins by determining the evaluator who will provide an assessment based on the criteria in the Sirius framework. Based on research from [11], [13], the number of expert evaluators involved in heuristic assessment is as many as five people. However, the evaluation in this study will be carried out by seven experts as evaluators. Furthermore, the manual evaluation uses the following scenario [12]:

- Review: Evaluators are given time to use the Anapedia website at http://anapedia.org/. For the search feature, the evaluator includes several keywords namely "photosynthesis", "plants", "snake", "sun", and "rainbow" in the text field provided.

- Evaluation: Evaluator observes search results. The evaluator evaluates the overall website interface based on 10 aspects in the Sirius framework.

- Rate severity: Evaluators give ratings for each criterion using a rating scale of 0 ... 10 or NWS ... YES.

The Anapedia web page is shown in Figure 3. The evaluator will enter the requested keyword in the column provided and press the "Search on Anapedia" button. The evaluator repeats the search process on Anapedia with different keywords. The evaluator then provides an assessment based on his experience interacting with the Anapedia website for a number of 83 Sirius framework criteria.

3. RESULTS AND ANALYSIS

3.1. Calculating global usability

In the third phase, a calculation is performed to determine the global usability value of the Anapedia web. Based on the content and objectives, the Anapedia website is included in the education web. Examples of global usability calculations from one evaluator for GA and II aspects can be seen in Table 3 (see in Appendix). The ev is the evaluation value given by the evaluators, sv is the conversion of the evaluator value, rv is the relevant criterion value for the educational web type, and wc is the weight coefficient of each criterion. Then the final value for each criterion is calculated, these values are then added to the fav value. Based on the table, the global usability value of evaluator one is 94.66% (shown in Table 3 in the% usability column). The same steps are applied for the other six evaluators. Based on the results of the evaluations of all evaluators, an average global usability value of 81.4% was obtained which means that the usability of Anapedia's website is included in the high category [21]. Automatic evaluations for the Anapedia web are conducted using Google PageSpeed. Google PageSpeed can be used to measure the performance and loading pace of the web [17], [25]. Based on an analysis using Google PageSpeed, Anapedia's web loading speed is 80% which is included in the HIGH category [26].

3.2. Analysis

A comparison of the usability level of all aspects is shown in Figure 4. The results show that the best usability component on the Anapedia website is the labelling (LB) aspect of 84% where all the criteria in this aspect are considered GOOD. Labelling aspects include evaluating labels on the web, page title, and page uniform resource locator (URL). Conversely, the help (HE) aspect perform poorly with a value of 17%. This result is related to the absence of the Help feature on the Anapedia web. In addition to the two components, the usability criteria that has not been met by the Anapedia website and their weights are shown in Table 4. Table 5 presents a comparison of aspects of the Sirius framework with several quantitative evaluation methods.

4. CONCLUSION

This research provides information to the developer team regarding web quality in the context of usability for the improvement of design both in terms of interface and content. Evaluation by experts is one way to measure website usability. In this study we evaluated Anapedia Web, the first open web-based encyclopedia for children built in Bahasa, that is still considered low in terms of comfort. This aspect is related to the usability of the website. The automated and manual tools were conducted to measure the usability aspect of the app. Google PageSpeed app were used for the automatic evaluation and heuristic (with the involvement of a group of evaluators) for measuring the usability level manually. Based on the evaluations of all evaluators, the average value of global usability is considered high. To find out which aspects have the highest and lowest values, measurements are made based on evaluator evaluations and the best and worst aspects of the results are related to labeling and assistance. To be more thorough, it is necessary to do an assessment by the user using existing methods.

Sidebar

Article Info

Article history:

Received Jul 21, 2020

Revised Jul 14, 2021

Accepted Jul 22, 2021

References

REFERENCES

[1] K. Benmoussa, M. Laaziri, S. Khoulji, M. L. Kerkeb, and A. El Yamami, "Evaluating the the Usability Usability of Moroccan University Management Web Platform," Procedía Manuf., vol. 32, pp. 1008-1016, 2019, doi: 10.1016/j.promfg.2019.02.315.

[2] A. Anand and G. Bansal, "Interpretive structural modelling for attributes of software quality," J. Adv. Manag. Res., vol. 14, no. 3, pp. 256-269, 2017, doi: 10.1108/JAMR-11-2016-0097.

[3] I. Dianat, P. Adeli, M. Asgari Jafarabadi, and M. A. Karimi, "User-centred web design, usability and user satisfaction: The case of online banking websites in Iran," Appl. Ergon., vol. 81, 2019, doi: 10.1016/j.apergo.2019.102892.

[4] S. Kaur, K. Kaur, E. T. Campus, and P. Kaur, "Analysis of Website Usability Evaluation Methods," in 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom), 2016, no. December, pp. 1043-1046.

[5] N. Bevan, J. Carter, and S. Harker, "ISO 9241-11 Revised : What Have We Learnt About Usability Since 1998 ?," Human-Computer Interact., vol. 1, pp. 143-151, 2015, doi: 10.1007/978-3-319-20901-2.

[6] A. Alshehri, M. Rutter, and S. Smith, "Assessing the relative importance of an e-learning system's usability design characteristics based on students' preferences," Eur. J. Educ. Res., vol. 8, no. 3, pp. 839-855, 2019, doi: 10.12973/eu-jer.8.3.839.

[7] S. S. Aparna and K. K. Baseer, "SIRIUS-WUEP: A Heuristic-Based Framework for Measuring and Evaluating Web Usability in Model-Driven Web Development," in Emerging ICTfor Bridging the Future-Proceedings of the 49th Annual Convention ofthe Computer Society ofIndia (CSI), 2015, vol. 1, pp. 303-310, doi: 10.1007/978-3-319-13728-5.

[8] A. Bin Deraman and F. A. Salman, "Managing usability evaluation practices in agile development environments," Int. J. Electr. Comput. Eng., vol. 9, no. 2, pp. 1288-1297, 2019, doi: 10.11591/ijece.v9i2.pp1288-1297.

[9] H. M. Az-zahra, W. Parwaningsuci, and M. C. Saputra, "Usability Evaluation of User Interface in Badan Narkotika Nasional East Java Province Website," 2018 Int. Conf. Sustain. Inf. Eng. Technol., pp. 262-265, 2018, doi: https://doi.org/10.1109/SIET.2018.8693144.

[10] K. Benmoussa, M. Laaziri, S. Khoulji, M. L. Kerkeb, and A. El Yamami, "Enhanced model for ergonomic evaluation of information systems: application to scientific research information system," Int. J. Electr. Comput. Eng., vol. 9, no. 1, pp. 683-694, 2019, doi: 10.11591/ijece.v9i1.pp683-694.

[11] J. Nielsen, "Finding Usability Problems through Heuristic Evaluation," in Proceedings of the SIGCHI conference on human factors in computing systems, 1992, pp. 373-380, doi: https://doi.org/10.1145/142750.142834.

[12] M. González, L. Masip, A. Granollers, and M. Oliva, "Quantitative analysis in a heuristic evaluation experiment," Adv. Eng. Softw., vol. 40, no. 12, pp. 1271-1278, 2009, doi: 10.1016/j.advengsoft.2009.01.027.

[13] M. C. S. Torrente, A. B. M. Prieto, D. A. Gutiérrez, and M. E. A. De Sagastegui, "Sirius: A heuristic-based framework for measuring web usability adapted to the type of website," J. Syst. Softw., vol. 86, no. 3, pp. 649-663, 2013, doi: 10.1016/j.jss.2012.10.049.

[14] Y. Inal, "University students' heuristic usability Inspection of the national library of Turkey website," Aslib J. Inf. Manag., vol. 70, no. 1, pp. 66-77, 2018, doi: https://doi.org/10.1108/AJIM-09-2017-0216.

[15] R. Hoi, Y. Fung, D. K. W. Chiu, E. H. T. Ko, K. K. W. Ho, and P. Lo, "Heuristic Usability Evaluation of University of Hong Kong Libraries ' Mobile Website," J. Acad. Librariansh., vol. 42, no. 5, pp. 581-594, 2016, doi: 10.1016/j.acalib.2016.06.004.

[16] F. Rangraz, E. Nabovati, R. Bigham, and R. Farrahi, "Informatics in Medicine Unlocked Usability evaluation of a comprehensive national health information system: A heuristic evaluation," Informatics Med. Unlocked, vol. 19, p. 100332, 2020, doi: 10.1016/j.imu.2020.100332.

[17] K. Sagar, "A systematic review of software usability studies," Int. J. Inf. Technol., 2017, doi: 10.1007/s41870-017-0048-1.

[18] A. K. Nazir, I. Zafar, A. Shaheen, and B. Maqbool, "GGSE-Website Usability Evaluation Framework," in Intelligent Computing: Proceedings of the 2018 Computing Conference, 2018, vol. 2, no. January, doi: 10.1007/978-3-030-01177-2.

[19] H. Mohammed, R. Yusoff, and A. Jaafar, "Quantitive analysis in a heuristic evaluation for Usability of Educational Computer Game (UsaECG)," in 2012 International Conference on Information Retrieval & Knowledge Management, 2012, pp. 187-192, doi: 10.1109/InfRKM.2012.6205033.

[20] L. Hasan, "Evaluating the Usability of Educational Websites Based on Students' Preferences of Design Characteristics," Int. Arab J. e-Technology, vol. 3, no. January 2014, pp. 179-193, 2016.

[21] G. Rodríguez, J. Pérez, S. Cueva, and R. Torres, "A framework for improving web accessibility and usability of Open Course Ware," Comput. Educ., vol. 109, no. February, pp. 197-215, 2017, doi: 10.1016/j.compedu.2017.02.013.

[22] N. Hayatin et al., "Anapedia-An Open Web-based Encyclopedia for Indonesian Children," J. Inf. Syst. Eng. Bus. Intell., vol. 5, no. 1, p. 48, 2019, doi: 10.20473/jisebi.5.1.48-56.

[23] D. Yuniarto, A. Subiyakto, and A. A. Rahman, "Technology readiness and usability of office automation system in suburban areas," TELKOMNIKA Telecommunication, Computing, Electronics and Control, vol. 18, no. 2, pp. 676-684, 2020, doi: 10.12928/TELKOMNIKA.v18i2.

[24] J. Perez, S. Cueva, and R. Torres, "Data in Brief Accessibility and usability OCW data: The UTPL OCW," Data Br, vol. 13, pp. 582-586, 2017, doi: 10.1016/j.dib.2017.06.007.

[25] E. Budiman, N. Puspitasari, M. Wati, and J. A. Widians, "Web Performance Optimization Techniques for Biodiversity Resource Portal Web Performance Optimization Techniques for Biodiversity Resource Portal," J. Phys. Conf. Ser., vol. 1230, no. 1,2019, doi: 10.1088/1742-6596/1230/1/012011.

[26] S. Gopinath, V. Senthooran, and N. Lojenaa, "Usability and Accessibility Analysis of Selected Government Websites in Sri Lanka," in 2016 IEEE Region 10 Symposium (TENSYMP), 2016, no. May, pp. 394-398, doi: 10.1109/TENCONSpring.2016.7519439.

[27] H. Omar Mohamed, R. Yusoff, and A. Jaafar, "Quantitive analysis in a heuristic evaluation for usability of educational computer game (UsaECG)," Proc. - 2012 Int. Conf. Inf. Retr. Knowl. Manag. CAMP'12, pp. 187-192, 2012, doi: 10.1109/InfRKM.2012.6205033.

(ProQuest: Appendix omitted.)