Epistemic action: A measure for cognitive support in tangible user interfaces?

Abstract

The quality of user interfaces is often measured in terms of efficiency, effectiveness, and satisfaction. In the area of tangible user interfaces, epistemic-or exploratory-action has been suggested as a fourth measure of quality. In computer game studies (Kirsh & Maglio, 1992, 1994), players used epistemic actions to modify the environment, which helped them determine the correct position of blocks with less mental effort. There, the researchers found that it might be easier to physically modify the external world and then interpret it than to compute and interpret a new state mentally. Specifically, epistemic action may be a relevant concept when researching tangible user interfaces incorporating physical handles. This article examines the potential relations between the three traditional measures of usability and epistemic actions using three spatial planning tools with different degrees of physicality. The results indicate that epistemic action is a measure that is independent of the three traditional usability measures: efficiency, effectiveness, and satisfaction. However, epistemic action does not increase linearly with the physicality of a user interface, and it probably is a more complex measure that is also related to the reusability of the interface. Further research is needed to fully understand the potential of this measure. [PUBLICATION ABSTRACT]

Headnote

The quality of user interfaces is often measured in terms of efficiency, effectiveness, and satisfaction. In the area of tangible user interfaces, epistemic-or exploratory-action has been suggested as a fourth measure of quality. In computer game studies (Kirsh & Maglio, 1992, 1994), players used epistemic actions to modify the environment, which helped them determine the correct position of blocks with less mental effort. There, the researchers found that it might be easier to physically modify the external world and then interpret it than to compute and interpret a new state mentally. Specifically, epistemic action may be a relevant concept when researching tangible user interfaces incorporating physical handles. This article examines the potential relations between the three traditional measures of usability and epistemic actions using three spatial planning tools with different degrees of physicality. The results indicate that epistemic action is a measure that is independent of the three traditional usability measures: efficiency, effectiveness, and satisfaction. However, epistemic action does not increase linearly with the physicality of a user interface, and it probably is a more complex measure that is also related to the reusability of the interface. Further research is needed to fully understand the potential of this measure.

In the field of human-computer interaction, the usability of a program or tool is often measured in terms of efficiency, effectiveness, and satisfaction (ISO/IEC, 1998).1 Recently, in the field of tangible user interfaces (TUIs), an additional measure for the quality of tangible tools was introduced: epistemic action. Kirsh and Maglio (1992, 1994) distinguished between observable user actions as being either "epistemic" or "pragmatic." An epistemic action is an action whereby users change their environment to search for a solution or strategy to perform a certain task rather than to move closer to an external goal state. A pragmatic action is strictly the action needed to perform this task. Epistemic action is sometimes also called "trialand- error" (Sharlin, Watson, Kitamura, Kishino, & Itoh, 2004) or "exploratory motor activity." Kirsh and Maglio (1992) illustrated epistemic action with the example of how players of the computer game Tetris rapidly rotate falling bricks instead of mentally determining the correct position for a brick and then rotating it to that position. Players use epistemic actions to modify the environment, which helps them to determine the correct position faster than they could do the corresponding mental rotations. It might be easier to physically modify the external world and then interpret it rather than compute and interpret a new state mentally. It has been suggested that epistemic action is a relevant concept when researching computer interfaces that involve physical objects such as TUIs (Patten & Ishii, 2007). Sharlin et al. suggested that support for epistemic actions is a decisive factor in the success of a TUI. Furthermore, in an unpublished conference paper, Kim and Maher (2006) showed that TUIs, as compared with graphical user interfaces (GUIs), "lead to a design process comprised of epistemic actions, thereby changing designers' spatial cognition." This in turn affects design processes by increasing designers' problem-solving behaviors and leading to a creative design. In this article, we examine potential relations between the three traditional measures of usability (efficiency, effectiveness, and satisfaction) and the number of epistemic actions. We do this by looking at three different spatial planning tools, each offering different levels of physicality. The aim is to determine whether the latter measure can be a useful additional measure of quality for TUIs.

TUIs

Fitzmaurice, Ishii, and Buxton (1995) were the first to distinguish TUIs from other interfaces, calling them "graspable" user interfaces. In his thesis, Fitzmaurice (1996) defined a graspable user interface as "a physical handle to a virtual function where the physical handle serves as a dedicated functional manipulator." Ullmer and Ishii (2000) later suggested the term tangible user interface and defined its most widely used meaning: "[TUIs] give physical form to digital information, employing physical artifacts both as 'representations' and 'controls' for computational media. Tangible user interfaces . . . couple physical representations (e.g., spatially manipulable physical objects) with digital representations (e.g., graphics and audio), yielding user interfaces that are computationally mediated but generally not identifiable as 'computers' per se" (pp. 916-917).

The most important characteristic of TUIs is the seamless integration of representation and control, which means that physical objects are both the representation of information and the physical controls for the direct manipulation of their underlying associations. Although traditional interface elements such as the mouse and keyboard are also physical in form, they are not TUIs, since the physical form of the mouse does not have any representational significance for what is being manipulated.

Epistemic Action and TUIs

According to Sharlin et al. (2004), "a good physical tool enables users to perform pragmatic, goal-oriented activity as well as trial-and-error activity [or epistemic actions] and ensures that the cost of speculative exploration of the task space is low" (p. 6). According to Fitzmaurice (1996), epistemic 6 can support a user's cognition by (1) reducing the memory load involved in mental computation, (2) reducing the number of steps in mental computation, and (3) reducing the probability of error of mental computation. Combining these findings, one can conjecture that a user interface that offers the best possibilities for epistemic actions is also the interface that offers the highest cognitive support (Kim & Maher, 2006). We therefore conclude that observable epistemic action could be an indication of the quality of a TUI. More specifically, we hypothesize that epistemic action is related to the level of physicality in a TUI, since physical objects offer good possibilities for epistemic actions. To test this hypothesis, we performed an experiment using three spatial planning tools offering clearly different levels of physicality (from no to complete physicality).

METHOD

In a previous between-subjects experiment (Wirz, 2005), a member of our lab measured efficiency (trial time), effectiveness (percentage of correct trials), satisfaction (questionnaire), and epistemic action (average number of tested blocks in a trial) for three spatial planning tools. Here we returned to these tools to explore the relation between epistemic action and the other three measures.

Tasks

Tools like BUILD-IT and Modeller are typically used for production plant layout, in which a typical task is to bring 3-D models into a special relation with each other. We designed a spatial planning task (see Fjeld, Guttormsen Schär, Signorello, & Krueger, 2002) that could be represented by all of the tools we had chosen to test and that represents a basic production plant layout task. This task utilized nine square blocks with different heights arranged in a three-by-three matrix, a fixed target, and a handheld laser source. The task was to place the laser source on the unique one of the nine alternative blocks where its beam could hit the nearby target as close to the center as possible. The laser source could be positioned, rotated, and slightly adjusted on any of the nine blocks.

All participants performed the same 12 tasks with the tool that was assigned to them. First, they were shown a demonstration task (always the same), and then they performed a task (also always the same) in which they could get help from the experimenter. Finally, they performed all of the 10 experimental tasks. The ordering of the experimental tasks was varied from participant to participant by using a Latin square design. Before each experimental task, the blocks were rearranged by the experimental leader without the participant being present.

Tools

The spatial planning task was to be performed using one of three tools, each of which could be used for the same tasks but with different degrees of potential physical interaction. The tools are presented here in increasing order of physical representation, or tangibility.

No physical interaction: Modeller. This computeraided design (CAD) tool offers virtual tools and views on a computer screen (Gähwiler & Fjeld, 2002; see Figure 1). It presents three different views of the situation: from the top, from the side, and in a 3-D perspective. The user manipulates the tool by using a mouse and keyboard.

Some physical interaction: BUILD-IT. This tool (Fjeld, Lauche, et al., 2002; Fjeld, Voorhorst, Bichsel, Krueger, & Rauterberg, 2000) also employs virtual models of the blocks, the target, and the laser source. The participants manipulate the virtual laser source, however, by using one physical brick (see Figure 2). The resulting situation is displayed on a screen.

Only physical interaction: PhysicalBlocks. This tool consists of nine metal blocks, a standard laser source, and a target consisting of a metal pin attached to a metal flag. The participant can adjust the placement of the laser source on the blocks and rotate it (Figure 3).

The tools chosen for our comparison can be placed at different positions in the reality-virtuality continuum described by Milgram, Takemura, Utsumi, and Kishina (1994). What those researchers describe as a real environment is the same as what we call a physical environment here. PhysicalBlocks and Modeller are located at the two outer bounds of the continuum, and TUIs such as BUILDIT are typically located somewhere in-between.

Furthermore, we can also place the three tools at different positions on a reusability scale. Modeller is a tool that is reusable for almost all spatial tasks, since it is a general CAD tool. BUILD-IT is designed for spatial planning tasks and has successfully been reused for architectural planning (Höger, 2001), simulation of layout processes (Fjeld, Jourdan, Bichsel, & Rauterberg, 1998), production planning (Masurat, 2001), and even software development planning (Neuner, 2001). These alternative uses of BUILD-IT have been systematically examined and compared by Fjeld, Morf, and Krueger (2004). Physical- Blocks, on the other hand, is a tool constructed solely for the planning task described in this article. In contrast to Modeller and BUILD-IT, PhysicalBlocks cannot easily be reused for any other task.

Procedure

The user study was performed with participants individually during the daytime. Each participant solved 1 demo task, 1 aided task, and 10 unaided experimental tasks. The study was carried out in an office with the investigator sitting next to the participant. It consisted of the following procedure.

First, we gave the participant a description of the general task (see Figure 4) and, more specifically, a description of performing the task with the given tool. A description of when to consider a task completed followed. This took about 5-10 min, depending on the kind and number of questions that a participant had.

This was followed by one demo trial (three false blocks followed by the correct block were tested), which took 1-2 min.

The participant then performed one aided trial, in which questions were answered and help was offered at need. This also took 1-2 min.

For all unaided experimental trials, we recorded the blocks tested (epistemic action), the trial time (efficiency), and whether the indicated block was the correct one (effectiveness). Blocks tested more than once were counted each time. Trial time was registered when the participant rang a bell. The next task was set up without the participant watching. For BUILD-IT and Modeller, this meant that the experiment leader pushed a "next task" button in the program. For PhysicalBlocks, this meant that the test participant went behind a curtain while the experiment leader reconfigured the blocks for the next task. The experiment leader told the participant when he or she could come back again.

Finally, the participant filled out a questionnaire (satisfaction) described below. This took about 5 min.

Participants

The experiment was carried out with 20 test participants for each tool. The participants consisted of 60 undergraduate and graduate students. There were 6 female and 54 male participants, who ranged in age from 20 to 34 years. No importance was given to whether participants were right- or left-handed, and they were paid 10 Swiss francs each.

Questionnaire

User satisfaction was measured by three questions. All choices were ordered from low to high satisfaction using a 5-point Likert scale ranging from low (-2) to high (+2).

* How clear was the task formulation with the tool? (The exclusive choices were not at all, not, neutral, good, and very good.)

* How did you perceive the difficulty of the task? (The exclusive choices were very difficult, difficult, neutral, easy, and very easy.)

* How suitable was the tool you used to solve the task? (The exclusive choices were of no help at all, of no help, well suited, of help, and of high help.)

RESULTS

The average results for each of the measures are given in Table 1. The first six columns show the three usability measures and, for trial time and satisfaction, the standard deviations for each of the tools. The last two columns show the epistemic action measure with its standard deviation.

ANOVA tests were performed for trial time, satisfaction, and number of blocks tested, but not for the percentage of correct trials, because the necessary preconditions for such analysis were not fulfilled for that measure. The tests showed significant differences for all three measures. However, since the ANOVA tests do not indicate the tools between which there was a significant difference, Tukey HSD post hoc tests were performed for all tool pairs. Table 2 shows the results of these tests.

DISCUSSION

Combining the trial time, percentage of correct trials, and average satisfaction measures, we can conclude that Modeller had the lowest usability, PhysicalBlocks had the highest, and BUILD-IT lay in-between those two. These results can be directly related to how much physical interaction each tool offers: The tool offering the most physical interaction was also the one with the highest usability.

However, for epistemic action, we see a result that is not related to any of the traditional usability measures. The number of tested blocks in a trial was lowest for Physical- Blocks but highest for BUILD-IT, and in the middle for Modeller. Epistemic action, as measured by the number of blocks tested, was thus not directly related to the level of physical interaction offered by the tool or to its usability. The tool that offered the most physical interaction was, indeed, the one with the lowest number of blocks tested in a trial.

An explanation for the low number of epistemic actions observed in the Modeller is that the costs of taking physical action in an external task environment are weighed against the benefit of the information gained by taking the action (Maglio, Wenger, & Copeland, 2008). Performing any actions in the Modeller was difficult, which probably prohibited users from performing trial-and- error actions, whereas this was much easier with BUILD-IT and PhysicalBlocks. This is in line with Kim and Maher's (2006) finding that a design process using a TUI can involve many more epistemic actions than when using a GUI, such as Modeller. Kim and Maher compared the design processes for designers using a TUI and a GUI to perform renovation tasks in which they configured an arrangement of furniture within a 3-D space. This research showed that designers using a TUI displayed more epistemic actions. Furthermore, "the designers using the TUI exhibited more gestures that are beneficial for designers' perceptual actions as a complementary strategy to the 3-D modelling actions" (Kim & Maher, 2006).

A surprising result, however, was the fact that the number of epistemic actions was lower for PhysicalBlocks than for BUILD-IT. There are several reasons why a tool that offers more possibilities for physical interaction would not yield the highest number of epistemic actions. In our study, the physical nature of PhysicalBlocks offered users the possibility to change the position of their head to determine the correct block. Although such head movements represent observable user behavior, we did not consider such movements as countable epistemic user actions when testing a block. In future studies, we may consider counting changes in head position as another type of epistemic action that is made possible with this tool. Another explanation is that it is also possible that PhysicalBlocks offered such rich visible support in the physical world that epistemic actions to modify the world were not even necessary, because users could see the correct solution immediately. In light of the usability measures alone, this observation might suggest that successful TUIs should be as close to reality as possible, just like PhysicalBlocks, and that designers should strive for the lowest epistemic action. However, we think that this is too simple a reasoning. As mentioned earlier, there is a drawback to user interfaces such as PhysicalBlocks: PhysicalBlocks is a highly specialized tangible representation of a given positioning task, whereas BUILD-IT and Modeller are reusable user interfaces for a range of spatial tasks.

Conclusion and Future Work

Considering epistemic or trial-and-error action in the context of TUIs is relatively new, so not much research has yet been done on the exact applicability of this concept. With this study, we have tried to explore in more detail how epistemic action could be used as a quality measure. Our findings show that epistemic action is not a simple linear measure of the cognitive support of a TUI-first, because the definition of an epistemic action requires more clarification, and second, because it is possible that increased possibilities for physical interaction decrease the need for environmental modifications in order to find a solution or strategy. This might result in a lower number of epistemic actions for tools that do provide more cognitive support. However, our findings also suggest that these tools may also be highly specialized for a certain task, and therefore not reusable. In future work, we will first need to determine more specifically what should be considered an epistemic action. Furthermore, we need to determine whether there is an optimum combination of physicality and reusability that one could strive for when designing a TUI. To do this, it will be necessary to evaluate more tools and compare the numbers of epistemic actions with the levels of physicality and reusability of the particular tools. By doing so, we may be better able to evaluate the quality of different versions of tangible tools to create reusable tools that support cognition, thereby assisting in creative design processes. Our research here is a first step in that direction.

Footnote

NOTE

1. The latest ISO definition (ISO/IEC, 2006) considers effectiveness the most important among these three measures.

(Manuscript received October 30, 2008; revision accepted for publication May 7, 2009.)

References

REFERENCES

FITZMAURICE, G. W. (1996). Graspable user interfaces. PhD thesis, 1996, University of Toronto. Available at www.dgp.toronto.edu/~gf/ papers/PhD%20-%20Graspable%20UIs/Thesis.gf.html.

FITZMAURICE, G. W., ISHII, H., & BUXTON, W. (1995, May). Bricks: Laying the foundations for graspable user interfaces. Paper presented at the 1995 Conference on Human Factors in Computing Systems (CHI 1995), Denver. doi:10.1145/223904.223964

FJELD, M., GUTTORMSEN SCHÄR, S., SIGNORELLO, D., & KRUEGER, H. (2002). Alternative tools for tangible interaction: A usability evaluation. In Proceedings of the IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR 2002) (pp. 157-166). Piscataway, NJ: IEEE Press.

FJELD, M., JOURDAN, F., BICHSEL, M., & RAUTERBERG, M. (1998). BUILD-IT: An intuitive simulation tool for multi-expert layout processes. In M. Engeli & V. Hrdliczka (Eds.), Fortschritte in der Simulationstechnik (ASIM) [Advances in simulation] (pp. 411-418). Zurich: v/d/f Hochschuleverlag.

FJELD, M., LAUCHE, K., BICHSEL, M., VOORHORST, F., KRUEGER, H., & RAUTERBERG, M. (2002). Physical and virtual tools: Activity theory applied to the design of groupware. Computer Supported Cooperative Work, 11, 153-180. doi:10.1023/A:1015269228596

FJELD, M., VOORHORST, F., BICHSEL, M., KRUEGER, H., & RAUTERBERG, M. (2000, April). Navigation methods for an augmented reality system. Paper presented at the Conference on Human Factors in Computing Systems (CHI 2000), The Hague. doi:10.1145/ 633292.633298

GÄHWILER, M., & FJELD, M. (2002). Evaluation of CAD-systems as alternative tools to a tangible user interface (TUI). Unpublished bachelor's thesis. Available at http://e-collection.ethbib.ethz.ch/view/eth: 26850.

HÖGER, K. (2001). Build-it competition: Collaborative design with new interface technologies. In M. Engeli (Ed.), Bits and spaces: CAAD for physical, virtual, hybrid realms (pp. 26-31). Basel: Birkhäuser.

ISO/IEC (1998). Ergonomic requirements for office work with visual display terminals (VDTs)-Part 11: Guidance on usability (Tech. Rep. ISO 9241-11). Geneva: International Organization for Standardization.

ISO/IEC (2006). Ease of operation of everyday products-Part 1: Design requirements for context of use and user characteristics (Tech. Rep. ISO 20282-1). Geneva: International Organization for Standardization.

KIM, M. J., & MAHER, M. L. (2006, September). The effects of a tangible user interface on designers' spatial cognition. Paper presented at the International Conference on Spatial Cognition 2006, Bremen, Germany.

KIRSH, D., & MAGLIO, P. (1992, July). Some epistemic benefits of action: Tetris, a case study. Paper presented at the Fourteenth Annual Conference of the Cognitive Science Society, Bloomington, IN.

KIRSH, D., & MAGLIO, P. (1994). On distinguishing epistemic from pragmatic action. Cognitive Science, 18, 513-549.

MAGLIO, P. P., WENGER, M. J., & COPELAND, A. M. (2008). Evidence for the role of self-priming in epistemic action: Expertise and the effective use of memory. Acta Psychologica, 127, 72-88.

MASURAT, T. (2001, May). Partizipatives Planen von Fabrikanlagen am Planungstisch BUILD-IT. Poster presented at the Tag der Forschung Informationstechnologie, TU Clausthal, Germany. Available at http://www2.imab.tu-clausthal.de/f iles/anlagenprojektierung/veroeffentlichungen/ partplanen.htm.

MILGRAM, P., TAKEMURA, H., UTSUMI, A., & KISHINA, F. (1994, October). Augmented reality: A class of displays on the reality-virtuality continuum. Paper presented at the SPIE Telemanipulator and Telepresence Technologies Conference, Boston.

NEUNER, M. (2001). Team-Plattform für die integrierte Software- Entwicklung [Press release]. Available at http://idw-online.de/pages/ de/news42795.

PATTEN, J., & ISHII, H. (2007, April). Mechanical constraints as computational constraints in tabletop tangible interfaces. Paper presented at the SIGCHI Conference on Human Factors in Computing Systems (CHI 07), San Jose, CA. doi:10.1145/1240624.1240746

SHARLIN, E., WATSON, B. A., KITAMURA, Y., KISHINO, F., & ITOH, Y. (2004). On humans, spatiality and tangible user interfaces. Pervasive & Ubiquitous Computing, 8, 338-346. doi:10.1007/s00779-004 -0296-5

ULLMER, B., & ISHII, H. (2000). Emerging frameworks for tangible user interfaces. IBM Systems Journal, 39, 915-931.

WIRZ, R. (2005). Evaluation of a comparative study of tangible user interfaces. Unpublished bachelor's thesis. Available at www.t2i.se/ pub/papers/Raphael_Wirz_2005.pdf.