Content area
Differences in cellular phones' complexity and their impact on children's performance are under study in this experiment. Twenty children (age 9-14 years) solved tasks on two phones that were simulated according to existing models on a PC with a touch screen, holding constant display size, fonts, and colors. Actions were logged and analyzed regarding execution time, detour steps, and specific errors. Results show that children using the Siemens C35i with 25% higher complexity (with regard to number of required production rules) spent double the time solving tasks and undertook three times as many detour steps as children using the less complex Nokia 3210. A detailed analysis of user actions revealed that the number of production rules to be learned fails to account for most difficulties. Instead, ambiguous naming, poor categorization of functions, and unclear functionality of keys undermined performance. Actual or potential applications of this research include guidelines to improve the usability of all devices with small displays and hierarchical menu structures, such as cellular phones. [PUBLICATION ABSTRACT]
Full text
Differences in cellular phones' complexity and their impact on children's performance are under study in this experiment. Twenty children (age 9-14 years) solved tasks on two phones that were simulated according to existing models on a PC with a touch screen, holding constant display size, fonts, and colors. Actions were logged and analyzed regarding execution time, detour steps, and specific errors. Results show that children using the Siemens C35i with 25% higher complexity (with regard to number of required production rules) spent double the time solving tasks and undertook three times as many detour steps as children using the less complex Nokia 3210. A detailed analysis of user actions revealed that the number of production rules to be learned fails to account for most difficulties. Instead, ambiguous naming, poor categorization of functions, and unclear functionality of keys undermined performance. Actual or potential applications of this research include guidelines to improve the usability of all devices with small displays and hierarchical menu structures, such as cellular phones.
INTRODUCTION ]
The number of children possessing cellular phones has increased remarkably within recent years. At least 45% of Germany's 12- to 13-year-olds own such a technical device (CHIP Online, 2003).
The numerous cellular phone models of various brands on the market differ in menu structure as well as number and functionality of keys for navigating through the hierarchical menu. Commonly agreed rules as to which arrangement of functions and keys lead to the best usability have not been defined. Perhaps that is why manufacturers generally allocate few resources to investigating usability issues. Some emphasize "ease and joy of use for complex systems" (e.g., Siemens, as reported in Stanney & Salvendy, 2000), but this may be more of a marketing tool than a focus of research. Even researchers argue that modern phones do not differ much, apart from their shape and color (e.g., Harper, 2001), and the industry may assume that the prevailing small differences in usability of the phones will become less important in the future because of today's children's ability to handle technological Systems easily. Compared with adults, who grew up in a less technological environment, kids are supposed to understand the mode of operation of these devices much faster by virtue of their early contact with interactive technology (e.g., computers, video games). Additionally, children's fascination for explorative and inquisitive activities is well known. However, little to no research has been conducted to evaluate children's abilities using technical products, and hence these assumptions have no empirical foundation, even if they seem to be common sense.
Good usability of cellular phones is crucial: Telecommunication providers are investing huge sums of money in new services and technologies (e.g., UMTS, or Universal Mobile Telecommunications System, and WAP, or Wireless Application Protocol), which will be used only if the phone enables quick and easy access to the desired information. Children are certainly a customer group focused on by the industry.
Research Issues and Experimental Logic
The current level of knowledge about children's purposeful interaction with technical devices is very low. Thus it is important to explore how differences in complexity affect the usability of cellular phones for young novice users as well as which specific difficulties they experience.
The factors that influence the complexity of a cellular phone, and thus the difficulties users have in learning how to efficiently use it, have not been systematically analyzed. In the literature, approaches can be found that define the complexity of devices by decomposing users' behavior on the keystroke level, quantifying the number of operations the user has to carry out. In Kieras and Poison (1985), for example, the number of production rules (condition-action pairs in the form "if" [condition] "then" [action]) the user has to learn to operate a system is of central importance. Applying this approach to cellular phones, it may be assumed that the more production rules that have to be learned and memorized by the user to carry out certain tasks, the higher the system's complexity. In order to solve tasks on a cellular phone, one has to know which menu entries have to be selected and which keys need to be pressed to select them. Accordingly, for each event in the display a production rule of the form "if-then" needs be learned; what is seen in the display by the user corresponds to the "if" (condition), and the keystroke required (e.g., to select a function or to scroll to the next item) corresponds to the "then" (action).
To fulfill the demand for ecological validity, two existing models of different brands, the Nokia 3210 and the Siemens C35i, were selected for this study. These phones have comparable functionality but differ in the number of production rules required to carry out frequently used functions.
Table 1 shows the number of production rules that have to be learned to solve four common tasks (calling someone using the phone directory, sending a short text message, hiding their own number, and editing an entry in the phone book) on these two mobile phones.
To carry out these four functions using the Siemens C35i a total of 45 production rules need to be learned, whereas for the Nokia 3210 only 36 production rules are required. As the difficulty of each rule cannot be predicted with the applied theoretical approach, all rules are assumed to be of the same complexity and therefore equally easy to learn. Taking the "easier to use" phone always as baseline, and assuming a linear relationship between number of production rules and time needed to complete tasks, leads to the conclusion that it should take in total approximately 25% longer to solve the tasks with the Siemens phone than with the Nokia. Regarding the single tasks, it is expected (according to the number of production rules) that users in the Nokia group will clearly outperform Siemens users in sending a short message (10 vs. 14 production rules) and hiding their own number (12 vs. 19 production rules), whereas Siemens users should show slightly superior performance when calling somebody (3 vs. 4 production rules) and when editing a number in the phone directory (9 vs. 10 production rules). As the adequacy of a user's task representation presumably varies from task to task (the notion of how to call somebody should be more adequate than how to hide their own number when calling, given that the first is frequently used on fixed-line phones), comparisons between only the two phones regarding each task will be undertaken, not between different tasks.
It is unclear whether the described formalism is adequate to predict the difficulties users will face, because it may be assumed that some "if-then" rules are more difficult to learn; for example, the specific name and allocation of functions in certain submenus or a key's functionality can be more or less intuitive. The experiment reported in this paper explores the impact of differences in cognitive complexity on child users, as well as the more specific difficulties children experience when interacting with the tested mobile phones, in order to derive recommendations for improved usability.
Usability of cellular phones can be evaluated using the standard EN ISO 9241-11 (1997), which states that usable systems have to be effective, be efficient, and satisfy the subjective needs of the user. Easy learnability of the systems' functionalities is another crucial usability criterion mentioned by many authors, such as Bennett (1984), Dix, Finley, Abowd, and Beale (1998), and Norman (1991). Therefore the effect of retention will be examined by applying the four tasks a second time in a slightly modified way.
METHOD
The experiment was designed to explore the usability of two existing cellular phones, which differ in cognitive complexity, for young novice users (children 9-14 years of age).
Variables
The independent variable under study is the cognitive complexity of the two cellular phones.
As dependent variables the standard measures (EN ISO 9241-11, 1997) for usability were used, such as effectiveness and efficiency as well as user ratings regarding the ease of use. For effectiveness, the number of tasks solved was measured. For efficiency, four parameters were analyzed: solution time (in seconds); number of detour steps (difference between the number of keystrokes actually done and the steps that were necessary to solve the task in the shortest way possible); incorrect paths taken; and keys used.
Tasks
All participants were asked to solve four different tasks on the phone: Task 1 was calling a number from the phone directory; Task 2 was sending a short text message (short message service, or SMS; in order to control for differences in typing speed, we provided the message to participants, who were required to send it only when they reached the adequate point in the menu); Task 3 was hiding or sending their own number when calling someone; and Task 4 was editing a number in the phone directory.
To ensure that the children understood semantically what they had to do in the different tasks, the instructions were carefully tailored with respect to a child-friendly setting, with examples taken from children's common experiences. The four tasks were presented twice in slightly modified way to measure retention.
Apparatus and Materials
In order to record users' actions while solving the tasks without disturbing the children, we simulated the cellular phones as a software solution that was run on a PC, which was connected to a 15-inch (38-cm) touch screen (TFT-LCD Iiyama TXA 3841, twisted nematic, with a display resolution of 1024 × 768 pixels; the touch logic was developed by ELO RS232C). The software prototypes mirrored exactly the menu structures and navigation keys given in the real phones. However, both simulated phones were identical in physical dimensions of the prototype, font type, and letter size, and three menu items were presented at a time on the display. Thus we ruled out possible confounding effects regarding visual or lexical factors. Also, the participants did not know which brand they were working with, avoiding biases.
We assessed acceptance and judged ease of use after the children completed the tasks by asking them a number of questions, which they answered by referring to smiling, neutral, or sad faces. The questions concerned the children's estimated difficulties in understanding the different tasks, problems understanding the functionalities of the navigation keys and the naming of the functions, as well as their general ease or difficulty in using the cell phones.
Participants
Twenty participants 9 to 14 years of age took part in the study. Ten participants were assigned to the Nokia group and 10 to the Siemens group. No participants had cellular phones themselves, and all had very little or no experience using cellular phones. The mean age was 12.9 years in the Nokia group and 12.7 years in the Siemens group. Seven boys and 3 girls were assigned to each of the two groups.
To ensure that differences in performance between the two conditions would be attributable to the usability of the cellular phones, we surveyed the children's experience using technology. Results from analyses of variance show no significant differences between the two groups regarding frequency of use and estimated difficulty using various technical devices (PC, wireless phone, videocassette recorder, fax machine). Furthermore, the participants' reported interest in new technologies was "rather high" in both conditions, not differing meaningfully.
Procedure
Before beginning the tasks on the simulated cellular phone, the children were asked questions regarding their age and previous experiences with various technological products. The children completed the questionnaire shown on the monitor by touching the respective fields; their answers were entered by the experimenter with the keyboard. This procedure enabled the children to get used to the functioning of the touch screen.
To avoid any feeling of performance pressure for the children, no time limit for solving the tasks was set. At the beginning of the experiment, the children were instructed that the study aimed to evaluate how well cellular phone manufacturers have done their job in providing easily usable phones. It was emphasized that it was not them, the kids, who were being tested. In addition, the children were told that they should work as thoroughly and quickly as possible. However, after a child had spent more than 10 min trying to solve one single task and seemed to run out of ideas as to how to carry on in a constructive way, the experimenter suggested going on to the next task because this task was actually very difficult.
When participants succeeded in solving the task, a "congratulations" message was shown on the screen. User manuals were not provided. The order of task presentation was the same (1-4) in both trials. After the tasks were completed, the experimenter assisted the participant in completing the questionnaire assessing the subjective ease of use of the cellular phone. The whole experiment lasted 40 min to 1 hr, depending on the working speed of the child.
The children volunteered to take part in the study. Because cellular phones are a highly prestigious and valuable topic at that age, the children had much fun and were quite proud of participating. After the experiment, all participants were compensated with a voucher for a visit to the movies.
RESULTS
Performance Using the Phones
First we analyze the overall performance using the two phones by comparing the average number of tasks solved using the Nokia and Siemens phones, as well as the time and the average number of detour steps a participant needed for the eight tasks using each phone. Then we evaluate performance in the two groups for each task separately.
Overall performance. Considering the process of solving the 8 tasks in total, effectivity was comparably good in both conditions, with an average of 7.0 of the 8 tasks solved by the children in the Siemens group and 7.4 in the Nokia condition. However, immense differences in efficiency measures could be revealed: Participants using the Siemens C35i needed on average 32 min 9 s to process the 8 tasks, undertaking 1315 detour steps, whereas Nokia 3210 users executed the tasks in half the time (M = 14 min 47 s), F(1, 19) = 12.93, p < .01, with a third of the number of detour steps (M = 434.2), F(1, 19) = 11.79, p < .01. Both measures indicating the difference in usability of the two phones are highly significant. Regarding the number of production rules to be learned, it was expected that the Siemens group would show 25% longer time on task and 25% more detour steps as compared with the Nokia group; however, t tests revealed that the performance of the Siemens group was significantly worse than predicted regarding detour steps, t(9) = 3.18, p < .05, and time on task, t(9) = 3.22, p < .05.
Task level. From analysis of the number of production rules to be learned in each task, it was expected that the Nokia 3210 users would show superior performance in sending a short text message and hiding their own number, whereas Siemens users would be slightly better when making a call and editing an entry in the phone book. In the following performance comparisons for the two groups, results of analyses of variance will be shown for the four tasks, averaging the two trials, and t tests will show whether the outcomes differ meaningfully from what was expected according to cognitive complexity theory. Additionally, we will report specific difficulties the children experienced with the two phones, which may explain unexpected performance differences.
Making a call. According to the number of production rules, a 33% disadvantage for the Nokia phone was predicted. However, making a call with the Nokia 3210 using the internal phone directory was nearly five times faster (M = 47 s), F(1, 19) = 6.13, p <.05, and needed 1/6 of the detour steps (M = 25), F(1, 19) = 6.37, p < 0.05, as compared with the Siemens phone (means of 3 min 46 s and 154 detour steps). Figure 1 illustrates the expected results (left) and the real outcomes (middle and right). The t tests show that the Nokia group's performance differed significantly from the expected time on task, t(9) = -22.32, p < 0.01, and detour steps, t(9) = -38.42, p < .01. Difficulties experienced by the Siemens C35i users were attributable mainly to their inability to identify the external phone book key and to understand that the phone directory is accessed only when being on the topmost level and then pressing this key. Only half of the Siemens participants managed to access the phone directory within the first minute; the others spent most of the time searching in the main menu for the phone book. Two children using the Siemens phone did not even solve this task within 10 min. In the Siemens group 50% of the children searched within the main menu in the "call register," mistaking this function for the phone directory, whereas in the Nokia group only 20% mistook the call register for the phone book.
Figure 1. Expected performance difference between the phones (left) and performance outcomes regarding time on task (middle) and detour steps (right) when calling someone (Task 1).
Sending an SMS. With means of 1 min 58 s and 52 detour steps, sending a SMS with the Nokia tended to be faster, F(1, 19) = 3.74, p < .1, and needed fewer detour steps, F(1, 19) = 4.32, p < .1, as compared with the Siemens phone. The Siemens group needed 3 min 54 s and 143 detour steps to solve this task, thus taking 98% longer and needing 175% more steps (Figure 2, middle and right). According to cognitive complexity analysis, Siemens users should have taken only about 40% longer to solve this task (Figure 2, left). The difference between expected time on task and outcome in the Siemens group is marginally significant, t(9) = 1.86, p < .1. The Siemens users' great difficulties may be attributable to the function's location in the "notifications" submenu: 50% searched for the function of sending a SMS in the "Internet" submenu, 40% in "office and fun," and 50% in the external phone directory. These misleading categories may have resulted in many of the detour steps.
Figure 2. Expected performance difference between the phones (left) and performance outcomes regarding time on task (middle) and detour steps (right) when sending a short message (Task 2).
Hiding their own number. Showing or hiding their own number was 162% to 208% more efficient in the Nokia group (means of 2 min 13 s and 84 detour steps) than in the Siemens condition (means of 5 min 49 s and 259 detour steps; Figure 3). These differences regarding execution time, F(1, 19) = 11.52, p < .01, and detour steps undertaken when changing the setting of the cellular phone, F(1, 19) = 7.62, p < .05, are statistically significant. Furthermore, performance in the Siemens group differed significantly from the 58% longer time on task, t(9) = 2.51, p < .5, and 58% more detour steps, t(9) = 2.23, p = .05, that were expected because of the additional production rules (Figure 3, left).
Figure 3. Expected performance difference between the phones (left) and performance outcomes regarding time on task (middle) and detour steps (right) when users were hiding or sending their own number (Task 3).
What may have been the reason for this huge performance difference? The correct function for hiding the phone number is named "incognito" in the Siemens C35i and is located under "settings" and then "during calls" (original term). However, 80% searched the function under "call divert," 60% under "notifications," and 50% under "office and fun" or "profiles" (all of which are located on the first menu level), and they often did so several times while trying to solve the task. When the users chose "settings," all first tried the menu entry "security" and 90% tried "phone" before entering the correct category, "during calls." Even when children had chosen the correct function, "incognito," 70% did not switch the setting from "off" to "on" at first but did so only after returning to this point (see Figure 5, right). Using the Nokia phone, 60% of the users got lost in the "phone book" and 50% in the "call register." Within the correct submenu, "settings," all searched for the correct function in "phone settings" and 70% in "security settings," whereas it is located in "call settings" (see Figure 5, left).
Editing an entry in the phone book. Contrary to what was expected, Siemens users needed more time (M = 152.7 s) and more detour steps (M = 102.3) than did Nokia users (119.1 s and 57.2 detour steps) to edit an entry in the phone directory, but these differences between the two phones did not reach statistical significance. However, performance in the Nokia group, differed meaningfully from the time on task, t(9) = -2.39, p < .05, and detour steps, t(9) = -6.50, p < .01, that were expected according to the 11 % more production rules to be learned, as compared with the Siemens phone (Figure 4). As with the first task, children's difficulty recognizing that the phone book cannot be reached through the menu but has to be accessed with the external phone book key was the main reason for the large number of detour steps.
Figure 4. Expected performance difference between the phones (left) and performance outcomes regarding time on task (middle) and detour steps (right) when editing an entry in the phone directory (Task 4).
General Errors
For all tasks, between 25% and 70% of the children searched for the solution in the submenu called "call register," which both phones have. This seems to be a very generic term for them, under which they expect to find all different kinds of functions, and it is often misleading. Using the Siemens C35i phone, the centrally positioned big green receiver key (see Figure 5, right) was used on average 40.9 times by each participant in the eight tasks, even though its use was required only twice (in the two tasks in which the child had to make a call). This indicates that the design of the key was misleading, which resulted in it being interpreted by participants as a selection button.
Figure 5. Snapshots of display and navigation keys of the Nokia 3210 (left) and the Siemens C35i (right).
Retention
The children were able to remember what they had learned in the first trial: Overall execution time was significantly shorter in the second trial than in the first, F(1, 18) = 76.12, p < .01, and the users also made significantly fewer detour steps, F(1, 18) = 37.73, p < .01. This result indicates that participants were able to understand some rules of the functioning, thus improving their performance when solving a task for the second time. Highly significant interactions between the trials and prototype indicate that regarding both measures, time needed, F(1, 18) = 19.81, p < .01, and detour steps, F(1, 18) = 13.64, p < .01, the group using the Siemens C35i showed higher improvement than did the Nokia 3210 group. The Siemens users needed 22 min 28 s and 861 detour steps for the first four tasks and 9 min 41 s and 445 detour steps for the second trial, thus improving by 57% and 48%, respectively, whereas Nokia users spent 9 min 2 s and 269 detour steps solving the first four tasks and 4 min 53 s and 165 detour steps when the tasks were repeated, which means an improvement of 46% and 39%, respectively. However, even though the Siemens users showed a somewhat greater enhancement in performance, they still needed more than twice the amount of time and detour steps to process the tasks in the second trial compared with the Nokia users.
Acceptance of the Two Phones
After completing the tasks, the children were asked questions regarding their experience with the phone they used. On one hand, the two groups did not differ with regard to their understanding of what they had to do in the tasks, F(1, 19) = 1, which was high in both groups. The subjective general difficulties solving the tasks on the phones, indicated by smiling (=1), neutral (=2), or frowning (=3) faces, tended to be bigger in the Siemens group (M = 2.25) than in the Nokia group (M = 1.85), F(1, 19) = 3.1, p < .1, but the difficulties in understanding the terms and categorization of the functions did not differ between the two groups (being rated neutral). On the other hand, Siemens users had significantly more difficulties understanding the functionalities of the navigation keys (M = 2.5 for Siemens and M = 1.45 for Nokia users), F(1, 19) = 12.85, p < .01. When asked the reason for their difficulties understanding the keys of the Siemens C35i, the children reported being confused by different key features (e.g., they had mistaken the green receiver key for a selection key; they did not know that the red receiver sign is used for hierarchical steps back; and they had difficulty recognizing the phone book key; Figure 5, right). The children using the Nokia 3210 reported problems only in understanding that the same key that is used for hierarchical steps back (the "c" button) is also used to correct entries (Figure 5, left).
DISCUSSION
The present study focused on the usability of cellular phones when used by children between 9 and 14 years of age. The cellular phones examined, the Nokia 3210 and the Siemens C35i, differed regarding their cognitive complexity. For the Siemens phone the user must learn 25% more production rules to operate four common functions, as compared with the Nokia phone. In order to measure retention we had the children solve these four tasks on the phone twice. While the users were solving the tasks, their actions were logged, enabling a detailed insight into their behaviors. Effectiveness (number of tasks solved) and efficiency (execution time and detour steps) as well as user acceptance and specific errors were analyzed as usability measures. The findings are now critically discussed from different perspectives: cognitive complexity and its limitation in predicting novice children's performance, as well as specific suboptimal aspects of the cellular phones' interfaces, which lead to design recommendations. Furthermore, retention, children as mobile phone users, and the methodological approach are considered.
Cognitive Complexity
Without doubt, a testimony to the importance of ergonomic issues in the human-machine interface has been demonstrated here. We found enormous performance differences between the tested phones concerning children's efficiency: The Nokia 3210 convincingly outperformed the Siemens C35i. However, performance differences between the two phones cannot be attributed primarily to cognitive complexity defined as the number of production rules to be learned by the user to operate the device. We have assumed a linear relationship between the number of production rules and time on task or detour steps, and one could argue that other forms of equations would better predict the ease of use. However, the fact that children using the Nokia 3210 were even faster and needed fewer detour steps for tasks that required more production rules to be learned, as compared with the Siemens C35i, shows that mathematical models both linear and nonlinear ones - fail to account for the most influential factors affecting the usability of a device. The data demonstrate that the number of production rules that have to be learned in order to solve tasks is not the main reason behind users' difficulties; rather, ambiguous naming and allocation of functions in the menu, as well as the functioning and design of keys, derogate usability. These shortcomings of the user interface, which account for a substantial part of the users' detours, could not be quantified a priori with the known formalisms. Nevertheless, they can lead to specific design recommendations.
Design Recommendations
Naming. The experiment described here gives evidence that the naming of menus, submenus, and functions is of crucial importance for good usability of a cellular phone. Four recommendations can be derived from this study:
1. Abstract terms should be avoided. Participants in the study did not take the abstract term "incognito" for the function of hiding the phone number when calling, even though most knew what the term meant, according to postexperimental interviews.
2. Very generic terms are misleading. The children associated the term "call register" with all kinds of functions and thus selected it often during navigation.
3. Very similar terms should be avoided. We found impressive data regarding the difficulties children experienced with similar submenus, such as "call settings," "phone settings," and "security settings" (see Figure 5, left). Categorizing the functions in intuitively understandable menus and submenus is accordingly of great importance. Using card-sorting tasks, it was shown in a related study with children (Bay & Ziefle, 2003) as well as young and older adults (Ziefle & Bay, 2004) that the ability to reconstruct the categorization of functions to submenus is closely related to performance using a cellular phone. This means that only those users who have a correct mental representation of the allocation of functions to the right superordinate term within the menu can efficiently solve tasks on a mobile phone.
4. Double negations, as "incognito on," are to be generally avoided because an affirmation here means not showing the number. This confuses the users, as confirmed by the data.
Keys. Unambiguous key design is of crucial importance for usability of a cellular phone. In the Siemens C35i, four shortcomings were detected (see Figure 5, right).
1. Implementation of a key for a function, which is expected to be found within the main menu, is problematic. Half of the participants in this study spent several minutes searching for the phone directory within the main menu.
2. Using ambiguous icons to define the keys is often problematic because users have difficulties understanding their meaning. This problem arose here with the stylized book on the key that activated the phone book.
3. Key size, color, and position are other highly crucial issues and must be designed according to the key's function and importance. The big green centrally positioned key of the Siemens C35i was, because of its features, mistaken for a selection button and used repeatedly (on average 40 times by each user), even though users were required to press it only twice, in order to make calls.
4. Keys that change their functionality at different points of the menu are misleading. This was the case for the red receiver key on the Siemens C35i and the "c" button of the Nokia 3210. As the children reported, and according to the experimenters' observation, this led to confusion. The specific effects of key functionalities are addressed in more depth in a related study (Ziefle, Bay, & Schwade, 2004), which isolated these factors from other confounding variables.
Retention
The findings demonstrated that when the participants were solving tasks for the second time, the number of production rules that had to be learned did not explain differences in performance between the two phones. Taking all tasks together, in the second trial the children still needed double the time and made twice as many detour steps to process the tasks when using the Siemens phone.
Children as Users of Mobile Phones
It is a widespread assumption that children are quickly able to handle all kinds of technical devices because they have become accustomed to them from an early age. Therefore it might be assumed that the ergonomie interface is less important for younger users and, accordingly, that children should have minor difficulties handling complex menus and need less time to find out how a device works, given their playful, curious, and rather flexible working style. As we found out, this is not true. Even for young users, who were educated in a technologically prone world and who are experienced with using various types of technical interfaces, the ergonomie human-machine interface is of great importance. A comparison with data from adults (Ziefle, 2002a, 2002b) shows that the critical points on a phone at which users experience difficulties are the same across different age groups. For example, adults also mistook the green receiver key for a confirmation key and did not recognize the phone book key or the term "incognito"; children showed only more persistence than older adults by actively exploring the menu and undertaking many detour steps. Consequently, from the present data it may be concluded that good design should lead to devices that are easily usable by a wide range of users, adults as well as children. Of course, for very young children - that is, children younger than 9 years, who may have difficulties reading - forms of interaction principles other than a menu need be considered.
Methodology
This study focused on the analysis of user protocols as methodological access, collecting hints from users' behavior to derive crucial ergonomie aspects of the interface. In contrast, most testing institutes and manufacturers of technical devices usually rely mainly on user preference data because users' opinions can be easily obtained. However, user acceptance measures are highly likely to reflect issues other than "pure" usability ratings: Factors such as social desirability, misestimation of one's own performance, and habituation to a specific brand may contaminate acceptance judgments and make it necessary to prove that preference ratings conform to performance outcomes. This approach was undertaken here. F,ven if the children's ratings reflect performance measures to a certain degree, they appear to be insufficient to extract specific weaknesses of an interface. In addition, attempts to simulate user's behavior using formalisms, such as the one applied here, cannot properly predict the usability of a device for novice users. The actual difficulty in learning specific "if-then" rules still has to be detected through user testing.
Admittedly, the analysis of user protocols is a highly time-consuming procedure, but it provides a detailed insight into users' behavior. It makes it possible to reconstruct, precisely and individually, how participants tried to solve the task, which paths were chosen, and consequently which might be the crucial problems of the technical device in misleading the user. Thus, from a methodological point of view, log files are useful - in fact, indispensable - for the evaluation of the human-machine interface.
Final Conclusions
The results of the present study are not restricted to the area of cellular phones but, rather, seem to be useful for many user interfaces. It was shown that formal theories explaining the complexity of a device as a function of the number of production rules (in the form of "if-then") required to operate a device cannot predict novice users' performance. They do not account for many important aspects of a device, such as appropriate naming and organization of functions as well as intuitive functionalities of the keys. However, the study also shows, rather vividly, that a lot of experimental research still has to be done to create easily usable human-machine interfaces for consumers with little previous experience with a specific device.
ACKNOWLEDGMENTS
The authors acknowledge the unbreakable enthusiasm of the participants of this study and their parents for entrusting us with their beloved children. Additionally, Philipp Brauner, who implemented the software for the simulated cellular phones, as well as Preethy Pappachan and Judith Strenk, who supported us in creating the figures for this article, deserve our gratitude. Final thanks go to two anonymous reviewers for their helpful comments on earlier versions of this manuscript.
REFERENCES
Bay, S., & Ziefle, M. (2003). Performance on mobile phones: Does it depend on proper cognitive mapping? In D. Harris, V. Duffy, M. Smith, & C. Stephanidis (Eds.), Human-centred computing: Cognitive, social and ergonomic aspects (pp. 170-174). Mahwah, NJ: Erlbaum.
Bennett, J. L. (1984). Managing to meet usability requirements: Establishing and meeting software development goals. In J. Bennett & D. Care (Eds.), Visual display terminals: Usability issues and health concerns (pp. 161-184). Englewood Cliffs, NJ: Prentice Hall.
CHIP Online. (2003). Studie: Jedes fünfte Kind telefoniert mobil [Survey: Every fifth child uses mobile phone]. Retrieved November 18, 2003, from http://www.chip.de/news/c_news_ 9670561.html
Dix, A., Finley, J., Abowd, G., & Beal, R. (1998). Human-computer interaction. London: Prentice Hall.
EN ISO 9241-11. (1997). Ergonomiste Anforderungen für Bürotàtigkeiten mit Bildschirmgeràten: Teil 11. Anforderungen an die Gebrauchstauglichkeit [Ergonomie requirements for office work with visual display terminals: Pari 11. Guidance on usability]. Berlin: Beuth.
Harper, R. (2001). The mobile interface: Old technologies and new arguments. In B. Brown, N. Green, & R. Harper (Eds.), Wireless world: Social and interactional aspects of the mobile age (pp. 207-226). London: Springer.
Kieras, D., & Poison, P. G. (1985). An approach to the formal analysis of user complexity. International journal of Man-Machine Studies, 22, 365-394.
Norman, K. L. (1991). The psychology of menu selection. Norwood, NJ: Ablex.
Stanney, K. M., & Salvendy, G. (Eds.). (2000). Special issue: ease and joy of use for complex systems at Siemens. International Journal of Human-Computer Interaction, 12, 279-468.
Ziefle, M. (2002a). The influence of user expertise and phone complexity on performance, ease of use and learnability of different mobile phones. Behaviour and Information Technology, 21, 303-311.
Ziefle, M. (2002b). Usability of menu structures and navigation keys in mobile phones: A comparison of the ease of use in three different brands. In H. Luczak, A. E. Cakir, & G. Çakir (Eds.), Proceedings of the 6th International Scientific Conference on Work with Display Units (pp. 359-361). Berlin: Ergonomic.
Ziefle, M., & Bay, S. (2004). Mental models of a cellular phone menu: Comparing older and younger novice users. In S. Brewster & M. Dunlop (Eds.), Mobile human-computer interaction - Mobile HCI 2004 (pp. 25-37). Berlin: Springer.
Ziefle, M., Bay, S., & Schwade, A. (2004). On keys' meanings and modes: The impact of different key solutions on children's efficiency using a mobile phone. Manuscript submitted for publication.
Susanne Bay and Martina Ziefle, RWTH Aachen University, Aachen, Germany
Address correspondence to Martina Ziefle, Department of Psychology, RWTH Aachen University, Jaegerstrasse 17-19, 52056 Aachen, Germany; [email protected]. HUMAN FACTORS, Vol. 47, No. 1, Spring 2005, pp. 158-168. Copyright © 2005, Human Factors and Ergonomics Society. All rights reserved.
Susanne Bay received her M.S. in psychology in 2002 from RWTH Aachen University, where she is a doctoral candidate in the Department of Industrial Psychology.
Martina Ziefle is an associate professor in the Department of Psychology at RWTH Aachen University. She received her Ph.D. in psychology from the University of Fribourg, Switzerland, in 1991.
Date received: July 3, 2002
Date accepted: May 13, 2004
Copyright Human Factors and Ergonomics Society Spring 2005