Content area
The addition of multimedia objects such as sound clips, pictures and animations to the traditional text-based learning environments (LE) can provide an enhanced learning experience due to the possibility of employing multiple representations for the content and providing rich background information. Researchers frequently come across teachers who distrust a learning environment as embodying the beliefs of the designers and not their own pedagogy. Following the lead provided by user modeling work carried out in the field of human-computer interaction, there has been much research on student modeling and adaptivity to individual learners; however, the role of the teacher as the manager of the learning process and hence a much more significant user of a learning environment has been ignored. This paper discusses the need for a human teacher model in any computer-based learning environment and recommends configurable, incremental and re-structurable contributive learning environments (CIRCLE) architecture to ensure wider acceptance and greater reuse of the phenomenal creative effort that goes into designing a good learning environment.
Full text
Keywords
Education, Multimedia, Learning, Modeling
Abstract
Researchers frequently come across teachers who distrust a learning environment as embodying the beliefs of the designers and not their own pedagogy. Following the lead provided by user modelling work carried out in the field of human-computer interaction, there has been much research on student modelling and adaptivity to individual learners; however, the role of the teacher as the manager of the learning process and hence a much more significant user of a learning environment has been ignored. This paper discusses the need for a human teacher model in any computer-based learning environment and recommends configurable, incremental and re-structurable contributive learning environments (CIRCLE) architecture to ensure wider acceptance and greater reuse of the phenomenal creative effort that goes into designing a good learning environment.
Electronic access
The research register for this journal is available at http:/Iwww.mcbup.com/research_registers The current issue and full text archive of this journal is available at http://www.emerald-library.com/ft
Introduction
The addition of multimedia objects such as sound clips, pictures and animations to the traditional text-based learning environments (LE) can provide an enhanced learning experience due to the possibility of employing multiple representations for the content and providing rich background information (Patel et al., in press). However, the situation is quite complex in the sense that it is the novice learners who are likely to benefit most from the multiple stimulus provided by richer representations, and yet it is this group of learners who are most likely to get distracted in the absence of directed learning as they may not have developed adequate meta-cognitive skills of setting learning goals, monitoring progress and changing learning strategies where necessary. Different teachers would therefore direct the process of learning by constraining it in different ways, for example, defining an appropriate grain size of learning and level of abstraction through selecting a top-down or bottom-up approach for accessing the contents and also through controlling the amount of contextual information (Kinshuk and Patel, in press).
The benefits of reusability are widely known and accepted to need re-iteration. However, many of the early promises of time and cost savings have not materialised due to a variety of reasons (Coatta, 2000). As Coatta observed, only trivial pieces of code can be taken from one context and reused without effort in another context. To take any significant amount of code out of an existing product and turn it into reusable functionality requires much thought, effort and time. It may also impose constraint on creative thinking processes of the designer.
The critical factor missed out by the LE system designers has been the fact that what holds true for code reuse is indeed true for application reuse for different purpose, especially in the case of any intelligent tutoring system (ITS). An LE system cannot be pulled out of a particular set of contexts and successfully implemented in another set of contexts unless the LE system is designed to be configurable, incrementally extensible and restructurable to allow for the changes in the context of their applications. This paper very briefly examines the contexts of an LE system to encourage a broader perspective in designing LE systems with a view to enhancing their usability and vastly increasing the scope for reusability of their numerous application-specific components.
The environmental contexts surrounding the design and implementation of an LE system are critical in providing the purpose for the system's use and significantly help or hinder in its wider acceptability. Acceptability is also affected by how close the system's pedagogy is to the implementing teacher. If the system's pedagogy is not in proximity to the teacher's pedagogy and there are no easy mechanisms to reconfigure the system's pedagogy, the implementing teacher would be required to reorient teaching strategy and in most cases will be reluctant to do so. To ensure wider acceptability, therefore, there is a need to consider the environmental contexts of an LE system in addition to the interactional context frequently addressed in the existing research (see Patel et al., 1999b). Two aspects that need particular attention are:
(1) What are the various teaching styles and how is the performance of an educational system affected by the possibly divergent teaching styles of the teachers involved in designing and implementing various educational resources?
(2) Does the nature of a particular domain favour a specific teaching approach or method of knowledge representation, providing points of convergence among the possibly divergent teaching styles?
Background
The paper is based on a broader consideration of contexts that led to the development of basic intelligent tutoring tools (ITTs) for introductory accounting topics by the Byzantium project under the Teaching and Learning Technology Programme of the Higher Education Funding Councils of the United Kingdom (Patel and Kinshuk, 1996a). The internal structure (see Patel and Kinshuk, 1996b) and functionality (see Patel et al., 2000) of the ITTs have been discussed in greater detail elsewhere, but the following narrative very briefly describes the scope and structure of an ITT and provides a brief development history. A basic ITT has a narrow focus. It encompasses a single topic or a very small cluster of related topics. It is a mixed-initiative system with an overlay type of student model supported by immediate dynamic feedback for interactive learning and delayed static feedback for assessment. Its inference engine processes knowledge rules stored in a two-fold knowledge base, giving it a degree of intelligence. It
focuses, in particular, on cognitive skills acquisition and is based on the Collins et al. (1989) framework for cognitive apprenticeship-- based learning. The functional requirements of the framework and how the ITT achieves it is shown below with reference to Figure 1, which illustrates the implementation in one of the screens in one of the ITTs:
The learners can study task-solving patterns of experts to develop their own cognitive model of the domain (modelling). The ITTs provide a basic concepts mode presenting textual/graphical explanations and solved examples. The same material is also available through the Help button in the interactive learning mode.
The learners can solve tasks on their own by consulting a tutorial component (coaching). The ITTs offer qualitatively better coaching through interactive guidance and dynamic feedback while a student is attempting to solve a problem.
The tutoring activity of the system is gradually reduced with the learner's improving performances and problem solving (fading). The ITTs provide help "by exception" and the tutoring activity is triggered by an illegal or incorrect attempt. Improved performance will automatically see less tutoring intervention.
While using the interface shown in Figure 1, if a student pressed the Help button, the system will bring up the descriptive explanations dealing with the concept of discount factors. The objectives of the learning environment reflect Collins' (1990) recommendations for constructing robust domain competence.
Accordingly, the system aims to facilitate the learners in:
acquiring the basic domain knowledge which can be used subsequently as a base to integrate all the bits and pieces of knowledge gained from specific situations; applying the basic domain knowledge in abstract and contextual scenarios to generalise the knowledge and skills to be able to apply them in real-world situations.
The scope of an individual ITT can be enlarged by combining various ITTs. An ITT may thus be seen as a building block of a larger and more comprehensive tutoring system. It may also be mixed and matched with other Internet-based technologies (e.g. streaming audio/video) or non-Internet-based technologies (e.g. audio/ video CDs) as well as human teachers (Kinshuk and Patel, 1996), in various configurations of computer-integrated LE to suit classroom based, open and distance learning.
The development of the early ITT prototype commenced in 1990 when the need was established to provide some kind of a tutoring tool to replace at least some aspects of teaching and assessment at the introductory level business studies. The purpose of such a tool was to release some of a lecturer's time with a view to better utilise it for richer interaction with advanced level students. Four ITTs were developed as fully functional advanced prototype applications for the teaching and learning of different techniques - involving dissimilar domain logic and operations, to provide a better understanding of the critical aspects of the interface and internal structural requirements for these diverse applications. The applications are currently operational and have already been used by student numbers in excess of 7,000 at multiple institutions.
Stoner and Harvey (1999) carried out an independent evaluation at the University of Glasgow, involving the Mrs and another widely used, traditional computer-based learning package. They found the results to indicate that student performance had improved significantly over the period since learning technology materials were introduced and that this improvement appeared to be mainly reflected in the students' ability to complete the questions in the area of learning covered by the ITTs. Their student feedback focus groups observed:
The ITTs were useful because you could go over bits you were unsure about. It was better than a book because it was interactive. With the interactive questions you tend to pay more attention than you would to a book.
They prefer the ITTs because the other package waffles on about what you already know and provides no incentives to pay attention to what it says.
The ITT offers instant feedback, is more involving and you can do as many questions as you like. Of the two tutoring systems, 71 per cent students showed a preference for the ITTs while 8 per cent indicated no particular preference. The students wanted more tutoring systems, similar to the ITTs, for other topics and were positive about computer-aided learning (CAL) in general, observing that it was good to use CAL if the tutoring software was good. They also found that the student performance had improved to a greater extent when the ITTs were better integrated into the curriculum. This confirmed the project team's own implementation experience of the test versions that the implementing teacher/tutor plays a crucial role in the success of a tutoring system.
The need to extend the consideration to contexts beyond the traditionally favoured interactional context (as evidenced by the extensive work on learner modelling and adaptivity to learner) emerged from various inter-disciplinary deliberations undertaken during the design, development and implementation of the ITTs. It has also benefited from the ongoing discussion on proposed further developments including implementing the methodology on the World Wide Web with a view to share both the development activities and their outcomes (see Patel et al., 1999). The notion, especially the need for a human teacher model, was presented at the Distance Education Association of New Zealand (DEANZ) conference (Kinshuk et aL, 2000) and was well received, with the attendees agreeing with the need for any LE system to be adaptive to the implementing teacher or at least be adaptable by providing facilities for easy configuration. A brief discussion on the contexts of an LE system may be helpful in obtaining the larger picture.
The contexts of an LE system
Any LE system is designed and used in the context of a wide variety of factors that can be grouped into three main categories, as shown in Figure 2.
Interactional contexts
The need to employ the interactional contexts in an intelligent system arises from the necessity to accommodate the notions of co-operation, explanation and incremental knowledge acquisition (Context-97, 1997). The research, as reported in the literature, points to the notion of context being employed primarily with respect to the tasks of plan recognition, knowledge structuring, knowledge representation, reasoning, and discourse management. Employing context in these tasks improves the human-computer interaction and facilitates intelligent feedback by the system. For example, Widmer and Kubat (1993) described a system called FLORA3 that implements incremental concept learning in dynamic environments where the target concepts may be context dependent and may change drastically over time. Dybkjaer et al. (1995) reviewed a spoken language dialogue system that uses context to provide system-- directed dialogues to enable controlled steps in the direction of mixed-initiative dialogue.
An example of the use of interactional context for providing better reasoning is found in PROTEGE-II system, a meta-tool for constructing task-specific, expert-system shells (Walther et al., 1992). The application of context to plan recognition has been explored by Johnson (1995) who presented a system called REACT, used for training operators of the communication links in NASA's deep space network (DSN). The systems described above paint a general picture of the existing research. They all suggest appropriate design philosophy using the notion of context as applied to the human-computer interaction. We suggest, however, that other classes of contexts, in particular the environmental contexts, are perhaps even more important from the point of view of widespread usability in the actual learning environment, and the LE system designers need to look beyond the student-- system interaction issues.
Objectival contexts
It is quite interesting to compare the objectives of the traditional approaches to planning for teaching and methods of assessment. For planning of teaching, a syllabus is drawn up consisting of all the subject knowledge that is considered essential. This syllabus acts as an indicative teaching plan and, except for some small variation, a teacher endeavours to cover as much of the syllabus as possible. The teaching is thus implicitly based on the necessity of knowing most of the discipline's "essential" subject matter. On the other hand, the assessment methods may only cover 30-60 per cent of the syllabus. Student performance in a typical assessment may be distributed across a range of, say, 20-80 per cent, where a 50 per cent score may indicate a pass mark. Thus, for an assessment based on say 40 per cent of the syllabus, a student needs to achieve a 50 per cent performance, i.e. a 20 per cent proven knowledge of the whole syllabus to pass! This situation raises some interesting questions about the overall objectives of the educational system. Since the acquired knowledge can only be demonstrated through summative assessment, the assessment strategy strongly influences the students' learning activities and provides an overriding objectival context that can undermine the objectives of a teaching and learning system. A badly designed summative assessment system can cause widespread adoption of shallow learning approaches (Patel et al., 1999b). While the objectival contexts are very important in their own right, this paper will focus on the environmental contexts.
Environmental contexts
The environmental contexts of an LE system are analogous to the contexts of office application systems (such as word processing packages, spreadsheets, etc.). While the contexts of office application systems can be defined in terms of the user attributes and nature of the tasks, those of the LE system have to be described by the student, the learning goal, the learning environment, and the practical application environment where the learning results will be employed in due course. Major groupings of the environmental contexts may be listed as:
Student (the student's capabilities, preferences and motivation), also including student peers.
Teacher (the designing and implementing teacher's preferences and outlook).
Discipline (the nature of subject discipline)
Characteristics of knowledge (the characteristics of the domain knowledge).
Characteristics of the medium (the capabilities of the computer hardware and software employed as a tutoring medium).
Social environment (the social environment in which the LE system is designed and used).
These groups of context and their main constituents are depicted in Figure 3.
The teacher as an environmental context
A pedagogic designer of an LE system needs to consider the categories of the students based on several criteria:
(1) The novice and expert users of a tutoring system (prior knowledge of how to use a tutoring system on any particular hardware/ software platform).
(2) The novice and experienced students, based on:
knowledge - due to prior exposure to the subject discipline, and
learning ability - due to prior exposure to academic instruction.
The user sophistication (as determined by factors such as age, experience, socioeconomic background, prior education and so on) may also be a factor to contend with for instance, adult learners may be rich in experience but poor in formal education.
A much improved appreciation of the complexity of the task undertaken by the LE system designer has brought about increasing recognition of two concepts. The first is the general acceptance that knowledge has a contextual component and that the context provides a systematic way to cluster, partition and organise knowledge and its dimensions (Brezillon and Abu-Hakima, 1995). The second is the acceptance of the role of a student's natural intelligence, i.e. common sense and general problem solving abilities in the learning process, to the extent that some researchers argue that the human tutors virtually never provide the sort of explicit diagnosis of student misconceptions that is sought to be provided in the traditional ITS (Cumming, 1991).
The over-ambitious design of traditional ITS attempted to outperform a human teacher though, as Chen (1995) noted, "the methods currently used in areas pertinent to computerbased learning environment are incomplete in addressing the wide range of cognitive and pedagogical issues involved". Given the natural intelligence of the student, perhaps the endeavour at deep understanding of the student's mental processes, though it may be desirable, does not appear to be very critical and the educational purpose may be better served by providing the teacher with better and more efficient tools for teaching. The attempt at eliminating the teacher from the process has perhaps resulted in many years of research with little to show in terms of tutoring systems that are operationally successful in the real learning environment. While the idealism provides a good impetus to research in the laboratories, it is essential to grasp the ground reality. An LE system, realistically, can only be seen as a joint cognitive system (Dalal and Kasper, 1994) comprising not only the tutoring software and a student, but also an implementing teacher and to an extent the peer students. The student-LE system interaction is therefore a convergence of the human psychology of a student and a teacher and, to the degree to which the tutoring software is "intelligent", the cyber-psychology of an LE system - reflecting the psychology of the LE system designers, including their perception of students, teachers and the learning process.
The implementing teacher plays various roles including those of providing the context, selecting and scheduling other educational technologies, managing the curriculum and overseeing the learning progression. In the ensuing power relationship, the preferences of a teacher will certainly be more important than the learning style of a student in gaining wider acceptance of a tutoring system. Identifying these preferences is a difficult task as the teachers may have different personalities and possibly different teaching styles born out of their traditional, progressive or vocational outlooks and their own learning styles (Entwistel, 1981). However, it is recognised that the orientation to teaching strongly influences the teaching methods adopted, learning tasks set, assessment demands made and the overall workload specified (Gow and Kember, 1993). However, unless the LE system has a teacher model and enables its reconfiguration to suit the implementing teacher, it will not find easy acceptance as the last thing the harassed teachers would want to do is to fight against their own past practices and invest time in implementing someone else's pedagogy. We suggest that a human teacher model should formally be incorporated in the design of an LE system and indeed in any educational system: to recognise the different teaching styles, to put on record the teaching styles) adopted in the design and enable manual or automatic adaptation to suit the implementing teacher. An explicit explanation of the teaching style adopted in the design not only enables an implementing teacher to understand the designer's rationale but also helps in dealing with the cognitive dissonance arising from any differences in the teaching styles. In fact, this might also contribute to improvements in the student learning in the case of less adaptive systems, as the rationale behind the adopted teaching strategy is made clear.
It is also important to recognize two facts (Clark, in press):
(1) it is the instructional method and not the media that causes learning, as has been demonstrated in many media comparison studies; and
(2) the human brain, that has evolved over millions of years, does not change rapidly and can be overloaded by the sensory output that the technology is capable of delivering.
To prevent such cognitive overload, the amount of information and, in particular, the richness of some of the contextual information may need to be constrained in early stages of learning. As discussed before, different teachers would direct the learning process by constraining it in different ways. An LE system therefore needs some mechanism for adapting to the teaching style of an implementing teacher. The need for such adaptation to the implementing teacher is even more critical in the case of the tutoring systems that are implemented in far-away places or are run on the Internet and accessed at long distances. The wide cultural differences may make some of the representations difficult to comprehend. Similarly, the teaching style adopted at the design stage may turn out to be unproductive in a different cultural setting. The implementing teachers in such cases need some way to localize the LE system.
Domains and the process of education
The process of education involves traversing the granularity within various domains to varying extents, from detailed to abstract and from intrinsically simple to complex representations of knowledge - the complexity generally arising from implicit knowledge, implied context and inferred semantic. It is observed in current educational practices that learning takes place over a number of topics in a number of subjects over a period of time, with progressively increasing depth and/or breadth. This practice indicates that there are levels in the learning process at which knowledge is instructed and the students advance in the process of an educational model progression along the part-whole dimension.
On the other hand, some students may be exposed to a more abstract representation of a domain, for example, management students may be exposing the financial report and its uses rather than the techniques employed in producing the reports.
To produce separate LE system for learning at these different grain sizes, even if possible, can be a very costly endeavour. In providing facilities for the teachers to reuse application components to relatively easily modify, increment, re-stucture and localise applications to produce different versions that can be collected systematically and accessed from a Web-based repository, not only is it possible to share the development effort and collectively build up vast amounts of educational resources but also to provide for far greater use of such resources.
The CIRCLE architecture
The only way to increase the usability and in the process automatically increase the reusability of various learning resources is to allow the implementing teacher to contribute through configuring the learning space, incrementally adding and re-structuring where necessary, the scope and functionality of various LE system components. While hypermedia technology may allow ease of linking and de-linking various resources, this in itself is not sufficient. A learning system has to be more than a library of inter-linked resources. It is the "learning by doing" and formative assessments accompanied by feedback which are crucial to any learning system. To enable modification in a learning system, the minimum requirement is a flexible architecture and a well thought out infrastructure for storing alternative versions and readily accessing them through an index providing meaningful description. Such architecture greatly helps in bridging any gaps between pedagogic strategies of the designing and implementing teachers. It also greatly reduces the cost and the lead time in implementing new and innovative practices into an existing LE system, enabling the overall performance to be enhanced readily by upgrading a few system components. The flexible and extensible structure greatly reduces penalty for any modification errors and encourages creativity and experimentation. Improvements in assessment systems and integration with the administrative systems can readily be implemented by just adding a few additional components, ensuring less duplication of effort and greater system integrity.
A possible example of an early adoption of CIRCLE architecture is shown in Figure 4 (Patel et at, 1999b). Such architecture complements the hypermedia capabilities of the Web browser with a notion of hyper-tutoring system providing problems for students to solve under intelligent interactive guidance. It is still too early to discuss a concrete architectural design in detail but it would require an interface manager and a tutoring manager with a QUI-based authoring shell to rapidly enable a teacher to select and position interaction objects and controls as well as to elicit expert knowledge and feedback messages in an interactive fashion.
Conclusion
The Internet provides global connectivity and opportunities of co-operation. Supported by appropriate authoring systems and an Internetbased repository of LE system components, rapid incremental development of a newer extended or modified component can be relatively easily enabled though re-use of existing components that can be reviewed and downloaded from a central repository. Adequate technologies are rapidly emerging that can be harnessed for deploying the CIRCLE architecture. For example, distributed component object model (DCOM) for Microsoft development tools or remote method invocation (RMI) for Java have now made it possible to implement such architecture using object model-specific protocols that require specific, homogenous infrastructure on both the client and service machines. Work on overcoming the limitations of such specific component technologies has already commenced. Kirtland (2000) notes that the Microsoft .net framework provides an application model and key enabling technologies to simplify the creation, deployment and ongoing evolution of secure, reliable, scalable and highly available Web services while building on existing developer skills. The effective pedagogic use of such flexible infrastructure will require a great deal of thought, effort and time initially. A further possible obstacle could be the problems in arriving at commonly accepted agreements on royalties payable for use of such systems and mechanism for payment of such royalties.
References
Brezillon, P. and Abu-Hakima, S. (1995), "Using knowledge in its context: report on the IJCAI-93 workshop", Al Magazine, Spring, pp. 87-91.
Chen, M. (1995), "A methodology for characterising computer-based learning environments", Instructional Science, Vol. 23 pp. 183-220.
Clark, R. (in press), in Stolovitch, H. and Keeps, E. (Eds), Handbook of Performance Technology, Chapter 2, Jossey-Bass, San Francisco, CA.
Coatta, T. (2000), "To reuse or not to reuse, that is the question", E-ssentials! Available at: http:ll www.spc.ca/resources/essentials/jun2800 Collins, A. (1990), "Generalising from situated knowledge to
robust understanding", American Educational Research Association Annual Meeting, April, Boston, MA.
Collins, A., Brown, J.S. and Newman, S.E. (1989), "Cognitive apprenticeship: teaching the crafts of reading, writing and mathematics", in Resnick, L.B. (Ed.), Knowing, Learning and Instruction, Lawrence Erlbaum Associates, Hillsdale, NJ, pp. 453-94.
Context-97 (1997), Call for Papers, available at: http:l/wwwlaforia.ibp.fr/, 1997
Cumming, G. (1991), "Using artificial intelligence to achieve natural learning", in Lewis, R. and Otsuki, S. (Eds), Advance Research on Computers in Education, Elsevier, North-Holland, Amsterdam, pp. 51-62.
Dalal, N.P. and Kasper, G.M. (1994), "The design of joint cognitive systems: the effect of cognitive coupling on performance", International Journal of HumanComputer Studies, Vol. 40, pp. 677-702.
Dybkjaer, L., Bernsen, N.O., Dybkaejr, H. and Papazachariou, D. (1995), "On the use of context in building language dialogue systems for large tasks", The IJCAI-95 Workshop on Context in Natural Language Processing, August 19, Montreal, Canada.
Entwistel, N. (1981), "Teaching styles, learning and curricular choice", Styles of Learning and Teaching, John Wiley & Sons, pp. 225-42.
Gow, L. and Kember, D. (1993), "Conceptions of teaching and their relationship to student learning", British Journal of Educational Psychology, Vol. 63, pp. 20-33.
Johnson, W.L. (1995), "Plan recognition in a situational context", The 1995 IJCAI Workshop on The Next
Generation of Plan Recognition Systems, Montreal, Canada.
Kinshuk and Patel, A. (1996), "Intelligent tutoring tools redesigning ITSs for adequate knowledge transfer emphasis", in Lucas, C. (Ed.), Proceedings of 1996 International Conference on Intelligent and Cognitive Systems, IPM, Tehran, pp. 221-6.
Kinshuk and Patel, A. (in press), "Implementation issues in Web-based training", in Khan, B. (Ed.), Web-Based Training, Educational Technology Publishers.
Kinshuk, Patel, A., Oppermann, R. and Russell, D. (2000), "Human teacher's role in emerging Internet-based intelligent tutoring", Supporting the Learner through Open, Flexible and Distance Strategies: Issues for Pacific Rim Countries, The Distance Education Association of New Zealand, pp, 209-17.
Kirtland, M. (2000), "The programmable Web: Web services provides building blocks for the Microsoft.NET framework", MSDN Magazine, September, available at: http:/msdn.microsoft.com/msdnmag/issues/0900/ WebPlatform
Patel, A. and Kinshuk (1996a), "Knowledge characteristics: reconsidering the design of intelligent tutoring systems", in Behrooz, A. (Ed.), Knowledge Transfer, pAce, London, pp. 190-7.
Patel, A. and Kinshuk (1996b), "Applied artificial intelligence for teaching numeric topics in engineering disciplines", Lecture Notes in Computer Science, Vol. 1108,
pp. 132-40.
Patel, A., Kinshuk and Oppermann, R. (in press), "Intelligent multimedia educational systems pedagogically designed systems for harnessing educational technology", The World in The Information Age, Global Knowledge Series For Arts & Sciences.
Patel, A., Kinshuk and Russell, D. (1999a), "HyperITS: a Web-based architecture for evolving a configurable learning environment", Staff and Educational Development International, Vol. 3 No. 3, pp. 265-80.
Patel, A., Kinshuk and Russell, D. (2000), "Intelligent tutoring tools for cognitive skill acquisition in life long learning", in Sinitsa, K. (Ed.), Educational Technology & Society, Journal of IFETS and IEEE-LTTF, available at: http://ifets.ieee.org/periodical
Patel, A., Russell, D. and Kinshuk (1999b), "Assessment in a cognitive apprenticeship-based learning environment potential and pitfalls", in Brown, S., Race, P. and Bull, J. (Eds), Computer Aided Assessment in Higher Education, Kogan Page, London, pp. 139-47.
Stoner, G. and Harvey, J. (1999), "Integrating learning technology in a foundation level management accounting course: an e(in)volving evaluation", CTI-AFM Annual Conference, April, Brighton.
Walther, E., Eriksson, H. and Musen, M.A. (1992), "Plugand-play: construction of task-specific expert-system shells using sharable context ontologies", Proceedings of the AAAI Workshop on Knowledge Representation Aspects of Knowledge Acquisition, AAAI, San Jose, CA, pp. 191-8.
Widmer, G. and Kubat, M. (1993), "Effective learning in dynamic environments by explicit context tracking", Lecture Notes in Artificial Intelligence, Vol. 667, pp. 227-43.
Kinshuk Ashok Patel and David Russell
The authors
Kinshuk is a Senior Lecturer, Information Systems Department, Massey University, Palmerston North, New Zealand.
Ashok Patel is Director, CAL Research, De Montfort University, Leicester, UK.
David Russell is Postgraduate Course Director, De Montfort University, Leicester, UK.
Copyright MCB UP Limited (MCB) 2001