Improving the Visualization of Call Trajectories, in Call Center Monitoring

Abstract

This research evaluates the usability of an interactive tool that provides real-time data on the status of human-machine dialogues in a call center and displays the callers' paths through the dialog system. A usability study was conducted to evaluate the use of the actual interface and two new designs for the display of call trajectories. An empirical study with 18 subjects was conducted in which traditional tasks using three types of information displays were executed, while task time and accuracy were recorded. Significant reductions in task time and error rate were observed with the proposed designs. [PUBLICATION ABSTRACT]

Headnote

Abstract

This research evaluates the usability of an interactive tool that provides real-time data on the status of human-machine dialogues in a call center and displays the callers' paths through the dialog system. A usability study was conducted to evaluate the use of the actual interface and two new designs for the display of call trajectories. An empirical study with 18 subjects was conducted in which traditional tasks using three types of information displays were executed, while task time and accuracy were recorded. Significant reductions in task time and error rate were observed with the proposed designs.

Keywords

Call center monitoring, Call trajectories, Human-machine dialogues, Usability testing

1. Introduction

Call centers are "a set of resources - typically personnel, computers, and telecommunication equipment - which enable the delivery of services via the telephone" [2]. To keep their position in the market and satisfy the needs of the modern customers, the companies have been enforced to provide services by the telephone to their customers, making call centers an important part of their organizations. Traditional ways of monitoring dialog systems such as call monitoring and summary reports are either unrepresentative samples of the callers' population, subjective, or too coarse-grained to be of diagnosis value [1]. It is important for the organizations to gather real-time performance information to support timely decision making. The use of call trajectories for call center monitoring represent an advantage to traditional call monitoring and summary reports as the information is not limited to records of callers, enabling the visualization of the conditions of the call center. Call trajectory analysis may assist in the identification of crowded areas of the call center where customer accumulation occurs, incidences of repetition of instructions or customer tasks, most and least repeated paths, customer trends, behavior, and preferences.

The concept of call trajectory monitoring was applied by a telecommunication company in the development of a monitoring system that allows real-time system monitoring and provides graphical views of the callers' paths through the dialog system. Although the development of this tool represented an innovative approach for call center monitoring it introduced new challenges for the system analysts as it compromised the usability of the monitoring system.

The following problems related to the usability of the tool were observed: (1) the display of call trajectories was very crowded, which makes it very difficult to follow the paths of the calls, (2) the dialogues and sub-dialogues were randomly organized in the layout, which makes it difficult to compare with the call-flow specification, (3) the expressions of the dialogue system (prompts) were identified with a number, (4) all the types of prompts were in the same color, which makes it difficult to distinguish them, (5) the arcs representing the paths of the callers contained too many statistics which makes it harder to distinguish the critical measures and their values. These problems affected the usability of the tool and produced frustration in the users.

A usability study was conducted to test the efficiency and effectiveness of the monitoring system using three different types of interfaces to display the call trajectories: the current design, a "detail-hidden" approach which separated the call flow dialogues in different windows, and a "mouse-over" approach which consolidated the prompts that were associated to the same sub-dialogue and provided zoom features by using the mouse. Both designs provide alternatives to mitigate these problems. Three different scenarios of dialogues were recreated on each interface design, representing low, medium, and high complexity dialogues. The subjects executed typical dialogue-system monitoring tasks while the task time and error rate were recorded. The task time, error rate, and user preference were assessed for each interface design.

The objective of this research is to determine which aspects of the graphical design would make the dialog trajectory analysis more effective, efficient and pleasant. The goal is to make valid suggestions to avoid the problems reported by the users and provide a good visualization of the call trajectories. The results obtained from this experiment will assist in the development of a functional interface. In addition the results represent a contribution to the human factors field by providing insights of users' preferences in the visualization of data, as they may be extended to other types of interfaces. While the content of the article and the recommendations might be specific to the context of the application studied, the Industrial Engineering community may benefit from: knowing an innovative way for call center monitoring, understanding the methodology of usability testing after development, applying the use of design of experiments to usability testing, and understanding the importance of usability testing and the incorporation of the voice of the customer in the design of artifacts.

2. Literature Review

A call center is used for the purpose of receiving and transmitting customers' requests by using the telephone. The customer inquiries may be handled either by an operator, a machine, or by a combination of both. Call centers are an integral part of many businesses with a significant and growing economic role [2]. In call centers the use of effective performance measures and efficient monitoring is a prerequisite for effective decision making [8]. Regardless the type of dialog system, a method for monitoring the user-system interaction is required [1].

The use of call trajectories for call center monitoring was proposed by a telecommunication company as an alternative to summary reports and call monitoring. The call trajectories are the routes taken by the calls in the dialogue system while the callers are served. Such trajectories provide a visual representation of the call center status that can be used by call center analysts and managers to detect problems related to the advancement of the callers in the system and to assess the capability of the call flow design to fulfill the customer effectively and efficiently. This approach represents an advantage to traditional call monitoring as it visualizes the length of the calls, the steps required for multiple types of services, and the multiple ways in which a customer can navigate in the call center. However, issues related to the information visualization and the usability of the tool were reported by the users. A usability study was conducted to mitigate these problems.

Usability is the extent to which the product meets the user needs. ISO 9241-11 defines it as the extent to which a product is effective, efficient, and satisfying in a particular context of use. [6] It is not only determined by the ease of use, but also by the extent to which the functional properties and other quality characteristics meet user needs in a specific context of use [6]. Usability is also defined as the quality of a system with respect to ease of learning, ease of use, and user satisfaction. It emphasizes understanding the activities of the users in the real environment, which is beyond what is known as task analysis, involving detailed studies of work practices, roles and concepts [7]. Effectiveness is a measure of the extent in which users can perform the tasks accurately, efficiency measures how fast the user can execute the task, which is related to productivity, and satisfaction measures the degree to which the user likes the product. These three measures were used in the usability study. The evaluation of usability provides feedback on how the design meets the needs of the user and tries to optimize all the factors that are important for an effective system-user interaction in a determined environment. It is determined by the characteristics of the product and the context of use [6].

The cost of rectifying any divergence between the design and the user's needs increases rapidly as development proceeds, which means that user feedback should be obtained as early as possible [6]. If the feedback from the user is not incorporated into design there is a high risk of rework or rejection of the product from the user. Over the past 30 years usability has become a central focus in software development, with almost 50% of system's development devoted to the user interface. [7]. Nevertheless, there are still many cases in which usability assessments are ignored or disregarded.

The use of Gestalt principles, visual metaphors, information models, dynamic displays, and consistency help the user to make sense of the information provided and to relate the information to what they know about their task, their goals and interests [7]. The incorporation of such principles should promote good usability of the systems and products and mitigate the problems reported. The new interfaces proposed were designed based on the user's feedback and related design principles. Therefore, the new designs should be more efficient and effective than the original design, enhancing the usability of the tool.

3. Overview of the System

The tool is designed to monitor the human-machine dialogues, which consist of the conversation between the customer and the computer in the call center. In this specific case the human-machine dialogue routes the callers to the service agents or provides basic services. The data of call routes is exported from call logs to the tool where it is transformed into a graphical representation of the calls trajectories in the dialog system [1]. The call trajectories are generated in real-time and projected as shown in Figure 1. The trajectories are requested by the user in the initial interface by dialogue type. The dialogues are groups of human-machine conversations containing all the possible prompts needed to provide a service that are used in conjunction for a common purpose. Each type of dialogue corresponds to a specific group of task in the system. The dialogues may be composed of small groups of dialogues named sub-dialogues. As an example there might be a dialogue named "Change customer information", which is intended to change the customer's personal information, and which may be composed of sub dialogues such as "Address change", "Status change", and "Phone number change".

Typical tasks of the users are to look for the flow of the customers, detect problems such as incidences of rejection of customer answer by the machine, prompts time-outs, customers' hang-ups, repetition of prompts, and the locations of such incidents in the call flow.

4. Specifications of Proposed Designs

Prior to the experiment, regular users of the current system were observed while using the tool and a series of interviews and focus groups were conducted. A list of problems with the interface and a set of possible design solutions for the most critical concerns (listed in the introduction section) were developed. Proposed designs were created based on the user's feedback and suggestions of using hierarchies of information, focus capabilities and color coding.

Two new designs of the trajectories are proposed, including features such as the consolidation of equal prompts, color and shape coding for types of prompts, the inclusion of the most relevant statistics for each arc, a pop-up with the text of each prompt when rolling the mouse over a node, capabilities to enlarge the view of a specific trajectory, and the separation of dialogues and sub-dialogues in different windows. Figures 2 and 3 show the new designs.

4.1 "Detail-Hidden" Approach

This information model used in this approach follows the structure of information hierarchies, which are information nodes where every node (except the root) has a unique parent node, but any parent node may have multiple child nodes. Hierarchies are a simple and efficient way to organize information by levels of abstraction as it decomposes and organizes navigation in a complex space [7]. In this case the hierarchy consisted of the initial level (window) containing the prompts of the overall dialogue and parent nodes for each sub-dialogue, and a second level (window) containing the trajectories related only to each sub-dialogue and connected to the main level by the parent node of the sub-dialogue. Color and shape coding were used to identify the parent prompt of a sub-dialogue. Relevant statistics of the sub-dialogues such as number of callers are displayed in the parent prompt of each sub-dialogue. The main view of this approach is shown in Figure 2. This design will explore the use of hierarchies for the visualization of call trajectories as a feasible alternative to reduce the cluttering of the data.

The user may focus on the specific prompt or sub dialogue without losing the context of the whole dialogue. This technique combines a large overview of an information model with a local expansion of the portion of the model currently in focus [7]. In this case, the peripheral information is not condensed or shaded, but the information of interest is enhanced when the mouse is rolled over a specific node. A view of this approach is shown in Figure 3.

5. Methodology

An empirical study was conducted to have the users interact with the original interface and the proposed designs. Eighteen subjects (mean age: 25 years) participated in the study, from which 44% have used the tool. All the subjects were familiarized with call flows and call trajectories. Adobe Photoshop and Java were used to create mock-up interfaces for the three designs and to develop the experiment platform. Three different scenarios representing low, medium, and high complexity dialogues were recreated on each interface design. A total of 9 different combinations of scenario-interface (3 levels of complexity, 3 interface designs) were used in the study. The experiment follows a between-subject design, with each subject exposed to only three combinations of scenariointerface that were randomly assigned, assuring every subject was exposed to each type of scenario and each type of interface. The between-subject design required less time for the subjects and mitigates learning effects.

As part of the experiment the subjects were asked to complete a pre-experiment questionnaire to gather their personal information and their expertise with the system. Automated instructions on the use of the three interfaces and a training session consisting of three tasks (one in each interface) were presented to each participant. Copies of the training material were accessible to the subjects during the experiment. Each subject executed three tasks in three different scenario-interface combinations. The tasks were representative of typical tasks executed by the users and were performed randomly. Examples of tasks are: to identify the number of callers who were directed to X prompt after passing through Y prompt, identify the number of callers who enter a sub-dialogue X and proceed successfully through the dialogue. The experimental platform automatically recorded the task time and the answers for each task. A post-experiment survey was conducted to ask the participant's preferences and feedback on each design.

6. Results and Discussion

The outcomes of the experiment are the task time, error rate and subject preferences. The averages for task time and error rate are shown in Table 1. The differences in the mean values of task time are statistically significant (P < .05). The task time increased with an increment in dialogue complexity and it was higher for the original design at all complexity levels. Overall, the average task time was 87.6 seconds for the "detail-hidden" interface, 107.3 seconds for the "mouse-over" interface and 162.9 seconds for the original design. This implies an average reduction of 43.5% in task time when the "detail-hidden" design is compared to the original design. The use of "mouse-over" design provides an average reduction of 35.5% in task time. In high complexity scenarios the average task time was lower for the "detail-hidden" design. In low and moderate complexity scenarios the average task time was lower for the "mouse-over" interface.

The error rates were determined by comparing the answers provided and the real answers for each task. The error rate was higher for the original design at all complexity levels. Overall, the error rate was 10.2% for the "detailhidden" interface, 13% for the "mouse-over" interface and 54.6% for the original design. This implies an average reduction of 83.5% in errors when the "detail-hidden" design is compared to the original design. The use of "mouseover" design provides an average error reduction of 77.3%. In low and high complexity scenarios the error rate was lower for the "detail-hidden" design. In moderate complexity scenarios the error rate was equal for the "detailhidden" and "mouse-over" designs.

The "detail-hidden" design was preferred over the original and the "mouse-over design by 65% of the subjects. About 59% of the subjects felt that when using this interface the execution of the tasks required fewer steps. Approximately 53% reported that the "Detail-hidden" design was the best interface to relate the call statistics with the call paths and 58% reported that it was the best interface to follow the call paths. Around 65% of the subjects reported their preference for the "mouse-over" design to recognize the occurrence of events. All the subjects reported that the color coding was helpful and 95% felt comfortable with the text pop-ups.

The proposed designs provided considerable reductions in the average task time and error rate, improving the efficiency and effectiveness of the monitoring tool. Overall, the lowest average task time and error rate were achieved when using the "detail-hidden" design, which was also the most preferred among subjects. The main features of this design, such as the use of two windows for the display of dialogue hierarchies, may be the best approach to visualize the call trajectories, especially for high complexity dialogues where both task time and accuracy are improved. For low and medium complexity scenarios, where the call trajectories are expected to be less crowded, the "mouse-over" design proved to be more efficient. The other features such as color and shape coding and the display of only the most important statistics proved to mitigate the usability issues. The use of a flexible interface in which the user may choose the type of view for the projection of call trajectories is suggested. The user may choose between the "detail-hidden" design and the "mouse-over" design, depending on the complexity of the dialogue.

7. Conclusions

Call centers are an integral part of many businesses with a significant and growing economic role [2]. The organizations must be able to monitor and evaluate the performance of their call centers, to know their position in the market and to be aware of their customer service levels and other aspects of their business. Dialogue trajectory analysis represents an innovative approach for call center monitoring in which the routes of the calls in the call center may be visualized. The system studied provides a way of visualizing the caller-dialog system interaction, enabling the evaluation of the dialog system performance and providing a feasible, real-time and fine-grained description of the call center status.

The objective of this study was to mitigate the usability problems of the interface used to project the call flows and understand the importance of including the user's preferences in the interface design. The incorporation of user's feedback and design principles improved the efficiency, effectiveness and the user's satisfaction, promoting the usability of the system. Usability methods not only help improve the user interface, but also often provide insight into the extent to which the product will meet user requirements [6]. The improvements in task time, accuracy and preferences obtained imply that the features of the new designs enhanced the usability of the tool. The results confirmed that, as it was hypothesized, the use of design principles and the incorporation of the users' feedback enhance the efficiency, effectiveness and the user preferences to the system.

Acknowledgements

The authors of this paper would like to thank Alicia Abella and Jerry Wright from AT&T Laboratories for their collaboration in this research, and Dr. Frank Ritter for his valuable comments, which improved the quality and scope of the article and its significance.

References

References

1. Abella, A., Wright, J.H., Gorin, A. L., 2004, "Dialog Trajectory Analysis", Proc. of ICASSP-04, Montreal, Canada.

2. Gans, N., Koole, G., and Mandelabauum A., 2003, "Telephone call centers: Tutorial, review, and research prospects," Manufacturing & Service Operations Management, 5, 79-142.

3. Gorin, A.L., Abella, A., Alonso, T., Riccardi, G. and Wright, J.H., 2002, "Automated natural spoken dialog", IEEE Computer Magazine, 35(4), 69-88.

4. Hackos, J. T., 1998, User and task analysis for interface design, Wiley, New York.

5. Nielsen, J. 1993, Usability Engineering, Academic Press, California.

6. Pew, R., Mavor A.S., 2008, Human-System Integration in the System Development Process: A new look, The National Academies, Washington D.C.

7. Rosson, M.B., Carroll, J. M., 2002, Usability Engineering: Scenario-Based Development of Human- Computer Interaction, Academic Press, California

8. Zbikowski, E., The Essential Call center KPI's. Customer Management Insight, Appeared on: Magazine of the International Customer Management Institute, April 2007.