Content area
Teaching and learning programming constitutes a challenge. Although several teaching approaches and programming tools have been proposed, it seems that they have limited impact on classroom practice. This article investigates students' perceptions on five educational programming environments that are widely used and the features that any introductory programming environment should have. The environments investigated are: BlueJ; objectKarel; Scratch; Alice; and MIT App inventor. These environments were studied and used by experienced undergraduate students of Informatics in the context of a fourth year course. The main features of the environments and the way of presenting them to students, as well as the assignments in the context of the course are presented, in order to help the reader realize what experience was gained by the students that evaluated the environments. Based on a questionnaire filled in by students interesting conclusions were drawn. Students identified the main features of the environments and evaluated them positively, although problems were identified. An introductory programming environment should engage students through the development of programs connected to their interests, such as games and mobile apps. Moreover, an ideal introductory programming environment should provide a simple and user-friendly Graphical User Interface (GUI) that supports visualization of objects and classes, includes a puzzle-like editor for program development, reports simple and understandable error messages in natural language, and finally the ability to execute the program in a step by step manner. Although no single environment fulfils all these features, it seems that the most successful environment is Scratch.
Educ Inf Technol (2017) 22:145176
DOI 10.1007/s10639-015-9433-1
Stelios Xinogalos1 & Maya Satratzemi1 &
Christos Malliarakis1
Published online: 16 September 2015# Springer Science+Business Media New York 2015
Abstract Teaching and learning programming constitutes a challenge. Although several teaching approaches and programming tools have been proposed, it seems that they have limited impact on classroom practice. This article investigates students perceptions on five educational programming environments that are widely used and the features that any introductory programming environment should have. The environments investigated are: BlueJ; objectKarel; Scratch; Alice; and MIT App inventor. These environments were studied and used by experienced undergraduate students of Informatics in the context of a fourth year course. The main features of the environments and the way of presenting them to students, as well as the assignments in the context of the course are presented, in order to help the reader realize what experience was gained by the students that evaluated the environments. Based on a questionnaire filled in by students interesting conclusions were drawn. Students identified the main features of the environments and evaluated them positively, although problems were identified. An introductory programming environment should engage students through the development of programs connected to their interests, such as games and mobile apps. Moreover, an ideal introductory programming environment should provide a simple and user-friendly Graphical User Interface (GUI) that supports visualization of objects and classes, includes a puzzle-like editor for program development, reports simple and understandable error messages in natural language, and finally the ability to
* Stelios Xinogalos [email protected]
Maya Satratzemi [email protected]
Christos Malliarakis [email protected]
1 Department of Applied Informatics, School of Information Sciences, University of Macedonia, 156
Egnatia Street, GR-54636 Thessaloniki, Greece
http://crossmark.crossref.org/dialog/?doi=10.1007/s10639-015-9433-1&domain=pdf
Web End = http://crossmark.crossref.org/dialog/?doi=10.1007/s10639-015-9433-1&domain=pdf
Web End = Microworlds, games, animations, mobile apps, puzzle editors and more: What is important for an introductory programming environment?
146 Educ Inf Technol (2017) 22:145176
execute the program in a step by step manner. Although no single environment fulfils all these features, it seems that the most successful environment is Scratch.
Keywords Educationalprogrammingenvironments.Microworlds.Educationalgames. Mobile apps . Teaching and Learning Programming . Pedagogy
1 Introduction
Teaching and learning programming constitutes a challenge both for instructors and students respectively. Extended research has been carried out the last 4 decades on the teaching of introductory programming and several teaching approaches have been proposed and specially designed tools have been devised. However, this research seems to have limited impact on classroom practice, as Pears et al. (2007) state in their highly referenced paper BA survey of literature on the teaching of introductory programming^. This survey covers four aspects of teaching introductory programming: curricula, pedagogy, language choice and tools for teaching. Tools for teaching are grouped in three categories: visualization tools; automated assessment tools; programming environments. Programming environments - that are of interest in the study presented in this paper - are grouped in two subcategories: programming support tools that support students in developing programs using a rather standard language and execution environment and microworlds that are based on physical metaphors and concrete actors (objects) that are depicted during program execution.
It must be stressed that Pears et al. (2007, p. 209) did not intend to provide a survey of the aforementioned tools, but as they state their aim was to Bgive an outline of the field^ and Bprovide examples of work in several sub-fields of tools research that have had demonstrated success or that we consider potentially important in the future^. Of course, other papers surveyed programming tools used for novices. These surveys, which categorized programming tools in different categories, are briefly reviewed in the next paragraphs in order to give a clear picture of the field.
One of the first surveys and critical analysis of tools devised for the teaching and learning of problem solving and programming was carried out by Deek and McHugh (1998). The tools were classified in four categories: visually-based and text-based programming environments; debugging aids; intelligent tutoring systems; and intelligent programming environments. A critical analysis of a big number of tools falling in the aforementioned categories showed that they did not manage to fulfill their goals.
In a subsequent study, Mark Guzdial (2004) investigated the answers to the question BWhat makes programming hard?^ that are provided by programming environments for novices that aim at making the learning of programming easier. In his study, three families of environments that were considered to have influenced the development of programming environments at that time were studied: the Logo family of environments having its roots on Lisp; the rule-based family of environments that originated from Logo, Smalltalk-72 and Prolog; and the traditional programming language family of environments. This study revealed some trends regarding novice programming environments, namely: using more traditional language syntax; giving the chance to work on meaningful computational artifacts; provide immediate feedback on students work.
Educ Inf Technol (2017) 22:145176 147
Concluding, Bthe progress in the field is toward making programming more interesting, more relevant, and more powerful for students^ (Guzdial 2004, p. 152).
However, evaluating such features of programming languages and environments is not an easy task. Furthermore, there is no universally accepted framework for evaluating programming languages and environments. McIver (2002) has proposed one of the most referenced methods for evaluating programming languages, evaluating programming environments in isolation, as well as comparing and evaluating Integrated Development Environments (IDEs), especially for novices. Although, McIver (2002) does not propose specific criteria the following categories of features are proposed: GUI, Editor, Debugger, Error messages, Visualization, Individual method execution and object construction and Students' evaluation.
Georgantaki and Retalis (2007) carried out a comparative analysis of educational tools used for teaching object-oriented design and programming, which was based on the approach proposed by McIver (2002). Georgantaki and Retalis (2007) reviewed: programming microworlds; and educational programming environments. Moreover, a reference is made to: tools for enhancing the capabilities of programming environments, such as Expresso for enhancing error messages; software tools that support an Bobjects-first^ teaching approach, including environments such as BlueJ and Karel microworlds; the Robocode gaming environment; and educational programming languages.
All the aforementioned studies surveyed, compared and/or critically analyzed various programming environments for novices. As mentioned these environments were categorized differently. The two main categories are:
& Programming microworlds (Georgantaki and Retalis 2007; Guzdial 2004; Pears et al. 2007). Guzdial (2004) refers to this type of environments using two subcategories instead of the term microworlds: the logo family; and rule-based family of environments.
& Educational programming environments (Deek and McHugh 1998; Georgantaki and Retalis 2007; Guzdial 2004; Pears et al. 2007). Various terms are used to refer to this kind of environments: visually-based and text-based programming environments (Deek and McHugh 1998); traditional programming language family of environments (Guzdial 2004); and programming support tools (Pears et al. 2007).
and additional categories are:
& Visualization tools (Pears et al. 2007)& Automated assessment tools (Pears et al. 2007)& Debugging aids (Deek and McHugh 1998)& Intelligent tutoring systems and programming environments (Deek and McHugh
1998)
A detailed taxonomy of both programming languages and environments for supporting novices - that was devised based on the pedagogical goals of each tool and not just its technical features - is provided by Kelleher and Pausch (2005).
Both this taxonomy and the aforementioned surveys have made clear that a great number of tools have been developed and are still being developed. David Valentine
148 Educ Inf Technol (2017) 22:145176
(2004) in a meta-analysis of SIGCSE Technical Symposium Proceedings for the period 19842003 investigated the kind of CS Educational Research carried out at the CS1/ CS2 level. Valentine found that 22 % (99 out of 444) of the papers described tools for CS1/CS2. It is clear that not all of these tools have been used in classroom and as a consequence have not been evaluated. Even for the environments that assessment has taken place it is not always easy to replicate and validate the conclusions drawn (Gross and Powers 2005). Of course, assessing an environment is not an easy task. Common shortcomings with assessment of introductory programming environments are the following (Gross and Powers 2005):
& The research questions posed for the environment are too vague and/or open to interpretation.
& The impact of an environment on students learning process and understanding is not usually studied.
& The assessment takes place under opportunistic and not realistic conditions. For example the grades achieved in a course selected by the students themselves are analyzed.
& The assessment is carried out by the developers of the environment that have expertise and deep knowledge of the pedagogy behind the environment and consequently achieve the best possible outcomes.
In this paper, we present an evaluation of five educational programming environments by a group of undergraduate students attending a course on BEducational Programming Environments and Techniques^. In the context of this course students used, studied relevant literature and carried out an assignment in each one of the following environments: the best known Bobjects-first^ IDE of BlueJ; the microworld and learning environment of objectKarel; the game-oriented environment of Scratch; the 3D animation and story-telling environment of Alice; and the mobile app oriented environment of MIT App Inventor. These particular educational programming environments were chosen from a wide variety of environments based on three main criteria. Specifically, our aim was to use introductory programming environments that: are representative examples of the main categories of environments presented in the previous paragraphs; incorporate a wide range of features that aim to support novices in different ways; and have an impact on classroom practice. After the utilization of the aforementioned environments throughout a whole semester students evaluated the pros and cons of each environment; the pedagogical contexts that each environment is most suitable for; and finally the features that every programming environment targeted to novices should have in order to assist them in learning programming. The novelty of this work lies in the fact that five representative educational programming environments that fall into different categories were evaluated under the same conditions by the same group of last-year students having prior experience in programming languages and environments, as well as the difficulties of learning programming.
The rest of the paper is organized as follows. In section 2 a description of the usage of the environments in the context of the course is provided, including a review of the main features of the five educational programming environments and the way that these features were presented and utilized by students. The methodology of the evaluation carried out in the context of the course is presented in section 3, while in section 4 the
Educ Inf Technol (2017) 22:145176 149
results are analyzed. Next, in section 5 a discussion of the results and the main conclusions of this study are presented.
2 Usage of the environments in the context of the course
The course BEducational Programming Environments and Techniques^ is offered as an elective course at the 7th semester of an 8 semester bachelor degree on Applied Informatics. The aim of the course is to support the teaching and learning of programming: (i) technologically through the appropriate educational programming environments; (ii) by studying the difficulties, errors and misconceptions of students recorded in the literature. The course includes 12 lectures and the environments that are studied are the following: BlueJ, objectKarel, Scratch, Alice and MIT AppInventor. In the following paragraphs we present the content of each lecture, the literature studied and the assignment for each one of the aforementioned environments. The aim of this section is to present: what the main features and didactical rationale of each one of the environments is; what experience the group of students that evaluated the environments was. Actually, emphasis is given on the context of using and assessing the environments and their features and not the course per se.
2.1 Lectures 1 & 2: The Bobjects-first^ approach - BlueJ
The goal of the first two lectures was firstly to introduce students to a programming environment that supports the teaching and learning of the object-first approach, i.e. incorporating functionalities such as the interactive creation of objects and invocation of methods; the inspection of objects situation; and the visualization of classes and objects. Secondly, the presentation and study of papers which discuss (a) the features that an educational programming environment should include and (b) the difficulties and misconceptions that students face when learning Object-Oriented Programming (OOP) using BlueJ.
BlueJ (Klling et al. 2003) is Ban integrated Java development environment specifically designed for introductory teaching^ and is widely used in OOP courses. The main window of BlueJ presents students with a simplified UML class diagram of the currently open project. Students can create objects by invoking a constructor through interaction with the classes of the UML diagram, and then interact with the depicted objects by calling methods through each objects popup menu. Furthermore, students can inspect an objects state during method execution. Visualization of the object oriented entities, such as objects and classes, is an important feature that aims at the comprehensive presentation of their differences. BlueJ incorporates a simple editor that uses different background colours for highlighting blocks of code, reports explanatory error messages and provides a java interpreter (code pad), as well as a debugging ability (Klling et al. 2003). Overall, BlueJ is characterized by the simplicity of using all the aforementioned features. Finally, a large number of plug-ins has been developed for providing important features missing from BlueJ, such as a class card for presenting the structure of a class.
BThe BlueJ system and its pedagogy^ (Klling et al. 2003) and the established Bguidelines for teaching object orientation with Java^ (Klling and Rosenberg 2001)
150 Educ Inf Technol (2017) 22:145176
proposed by the team of BlueJ were presented to students using the corresponding papers in the context of the 1st lecture. Besides the activities proposed for the pedagogical use of BlueJ in the aforementioned papers, activities proposed in the accompanying textbook (Barnes and Klling 2004) for the environment were also utilized.
In the 2nd lecture the difficulties, errors and misconceptions of students when introduced to OOP were studied using the widely known paper by Ragonis and Ben-Ari (2005), as well as a paper on a teaching approach for alleviating the difficulties of learning OOP that takes into account the relevant literature and the special features of BlueJ (Xinogalos 2012).
2.1.1 Assignment
Students were asked to devise a project for teaching the basic concepts of OOP following the guidelines proposed by Klling and Rosenberg (2001) and the sequence of activities proposed in (Klling and Rosenberg 2001; Klling et al. 2003). This type of assignment was selected because students had been taught OOP and were familiar with relevant problems.
The aim of the assignment was: (i) to give students the role of the teacher and have them use BlueJ for teaching the main concepts of OOP; (ii) to investigate in what degree can students take into account the findings of the literature regarding a more effective teaching of the main OOP concepts applying the Bobjects-first^ approach and an environment that has the necessary features for supporting this approach.
2.2 Lectures 3 & 4: The microworlds & mini-languages approach - objectKarel
The goal of these two lectures was to present educational microworlds and mini-languages for programming, as well as structure editors as an alternative teaching approach of OOP, or programming in general. The typical approach to teaching programming is based on a conventional programming language and environment and a set of number and symbol processing problems. On the contrast, programming microworlds: have a limited instruction set, which however covers all basic constructs; is usually based on a physical metaphor that includes an actor controlled by students through their programs; program execution takes place in a step by step manner and is depicted the actor(s) execute the statements; a set of problems based on every-day activities are solved. In some cases, students are further supported by incorporating a structure editor in the microworld, so as to eliminate the need to memorize syntax details, avoid syntax errors and help students focus on understanding programming concepts and acquiring problem solving capabilities.
The approach of microworlds and mini-languages as Ba way to learn programming principles^ was presented using the work by Brusilovsky et al. (1997) and also a Bcase study in supporting levels of cognitive development in the computer science curriculum^ based on the microworld JKarelRobot (Buck and Stucki 2001). The technology of structure editors was studied through the Bevolution of novice programming environments developed at Carnegie Mellon University based on structure editors^ (Miller et al. 1994).
Educ Inf Technol (2017) 22:145176 151
The microworld objectKarel was used for studying and applying all the aforementioned technologies. objectKarel (Xinogalos et al. 2006a) is based on Karel++ (Bergin et al. 1997) which uses the well-known metaphor of robots carrying out various tasks on a restricted world. objectKarel incorporates a series of e-lessons and hands-on activities for motivating students through the use of concepts before they are asked to implement them. Programs are developed with a structure editor, which has the form of a menu and aims at avoiding syntax errors and focusing on concepts. Highly informative error messages are reported for the few errors that can occur, explaining the source of the error in natural language. Students can run, trace with a predefined speed or execute programs step by step (only forward). Also, objectKarel incorporates the technology of explanatory visualization, which means that students are presented with explanatory messages for the semantics of the statement being executed.
2.2.1 Assignment
Students were assigned to solve four problems that required using selection and repetition structures and the following OOP concepts: objects-classes, inheritance, polymorphism and overriding. The aim of this assignment was for students to get familiarized with solving a different type of problems from the ones they had already confronted (processing symbols and numbers or developing applications with GUI), as well as with using a structure editor and a programming environment based on a microworld and a mini-language.
2.3 Lectures 5 & 6: The game-oriented approach - Scratch
The goal of these lectures was to present a visual programming language and environment where students do not face syntax errors and where they are able to develop a different type of programs like games, animations, and simulations that can be shared with a universal community on the web. The environment used for this reason was Scratch.
Scratch (Maloney et al. 2010) is a visual, block-based programming language and environment for children. Scratch projects are built of graphical characters (called sprites) on a stage, and each sprites behavior is managed by one or more stack of visual instruction blocks. Scratch scripts are built by putting together blocks illustrating statements, expressions, and control structures. The shapes of the blocks indicate how they fit together, and a drag-and-drop technique allows for the blocks to be connected in meaningful ways. A Scratch program does not need to be compiled or edited/run. The programmer can click on a stack of blocks or a command at any time to see what it does. Similarly, Scratch has no error messages. Blocks fit together only in meaningful ways and thus syntax errors are avoided. The Scratch GUI makes navigation easy. It uses a single window, multi-pane design to ensure that key components are always visible.
With Scratch, children program and create their own stories, games, animations, simulations, interactive art and share their creations with others in an online community.
In order to present the rationale and the special features of Scratch students first watched the talk BLets teach Kids to code^ by Mitchel Resnick on TED (http://www.ted.com/talks/mitch_resnick_let_s_teach_kids_to_code
Web End =http://www.ted.com/talks/mitch_resnick_let_s_teach_kids_to_code ). BThe
152 Educ Inf Technol (2017) 22:145176
Scratch programming language and environment^ were presented using the homonym paper by (Maloney et al. 2010). Students used Scratch, while various tutorials and games were presented.
2.3.1 Assignment
Students had to develop a game with the following requirements: the player has at least three trials (lives); the player has a predefined time for playing; there are at least 3 opponents; the game records the best score for each player; the game gradually becomes more difficult as the player continues playing.
The aim of this assignment was for students to familiarize themselves with the capabilities of Scratch and use it in a creative way according to its design.
2.4 Lectures 7 & 8: The story-telling & 3D animation approach - Alice 2.0
The goal of these two lectures was the same as this of lectures 3 and 4.
Alice (Cooper et al. 2003) is a 3D interactive, animation, programming environment that aims to teach computer programming to students by visualizing objects and their behaviours. Specifically, students can use/modify 3D objects in order to create their own virtual worlds and write scripts for controlling the objects' appearance and behaviour. Alice utilizes a drag-and-drop smart editor for dragging objects into the editor and drop-down menus for selecting messages to send to an object. User-defined methods and functions are automatically added to the dropdown menus. Students can see immediately how their animation programs run each action is animated smoothly - enabling them to easily understand the relationship between the code and its result.
Alice 2.0 was presented using the tutorials and educational material that is available in the web site of Alice (http://www.cs.duke.edu/csed/alice/aliceInSchools/
Web End =http://www.cs.duke.edu/csed/alice/aliceInSchools/ ).
2.4.1 Assignment
The assignment aimed at familiarizing students with Alice, its features and the projects that can be developed with it. This assignment was carried out in the lab in contrast with the rest that were assigned as homework. Students were given the presentations from various workshops carried out at Duke University for developing projects (http://www.cs.duke.edu/csed/web/alice09/tutorials.php
Web End =http://www.cs.duke.edu/csed/web/alice09/tutorials.php ). Most of the above mentioned projects are mainly related to mathematical notions and we wanted to familiarize students with different types of problems that could be solved using a programming language and environment.
2.5 Lectures 9 & 10: The app-oriented approach - MIT App Inventor
The goal of these two lectures was to study a visual programming environment that is similar to Scratch and Alice, which additionally focuses on developing
Educ Inf Technol (2017) 22:145176 153
apps for mobiles. In addition, we wanted to show that it is possible to learn the fundamental notions of programming when the incentive is to develop interesting apps for everyday devices, deploy them on the web, and share them with ones friends and family.
MIT App Inventor is a blocks-based cloud programming tool that enables everyone to develop fully functional apps for Android devices.
In App Inventor the programmer develops the program as in Scratch in an interactive way using the drag and drop technique. He patches together blocks with specific meaning like loops, if-then-else statements or variables. The pieces fit in each other like in a jigsaw puzzle.
App Inventor consists of three parts:
The Designer: where the programmer selects components from a palette (object of a certain class) and drags them to the Viewer (screen layout) to easily build the GUI of his app. Then, the static attributes and properties of the selected component can be changed.
The Block Editor: provides blocks of code (Bbuild-in drawers^) with specific behavior. The programmer drags & drops them to the blocks viewer and he puts them together so as to define his apps dynamic behavior. The app can respond to various events, such as user events (touching the screen) or system events (a new location).
App Inventor provides databases, geolocation, audio and video contents and other features.
Testing app: the user can test an app by using a pre-configured emulator or an android phone while building it without compiling a program.
App Inventor was presented through the tutorials and the projects that are available in the site of the environment (http://www.appinventor.org/
Web End =http://www.appinventor.org/ ). App Inventor 2 was just released.
2.5.1 Assignment
This assignment was carried out by groups of two students. Students were asked to study the material regarding BModule 5: Shared and Web Data An introduction to apps with shared data^ from the site https://sites.google.com/site/appinventorcourse/communications-app-creat
Web End =https://sites.google.com/site/appinventorcourse/ https://sites.google.com/site/appinventorcourse/communications-app-creat
Web End =communications-app-creat and implement an app similar to the ones proposed in this web page.
2.6 Lectures 11 & 12: Literature study
Students were assigned to select an article from the proposed literature regarding the presented educational programming environments, study it, prepare and present a presentation. Students presentations were made in class, followed by discussion with the instructor and classmates.
154 Educ Inf Technol (2017) 22:145176
3 Methodology of the study
3.1 Participants
The study presented in this paper has taken place in the context of the 7th semester elective course BEducational Programming Environments and Techniques^ at a Department of Applied Informatics. The course that was described in Section 2 is being taught for three years now by the second author of this article. During the academic year 201314 thirty seven students selected and attended the course. The majority of students was in the 4th year of their 4 years (8 semesters) bachelor studies (Table 1).
All the students had been introduced to programming in the context of the 1st semester compulsory course BProcedural Programming^ that uses C as the programming language and environments like wxDev-C++ and Code::Blocks. Students were also taught OOP in the 3rd semester compulsory course BObject-Oriented Programming^ with Java and Eclipse. In a relevant question included in the final questionnaire one student stated that s/ he is an expert in OOP and Java, while the rest stated in exactly the same percentages that have either basic or advanced knowledge (Table 2).
3.2 Research questions
The main research questions that we aimed to investigate with this study are the following:
Research Question 1 (RQ1): What do students think of the pedagogical usage of the five educational programming environments?
All the participants of the study had attended, as mentioned above, compulsory programming courses. As is usually the case, several students had faced various difficulties and some of them had not managed to acquire a deep knowledge of some programming concepts, especially OOP concepts that are more advanced. One of the aims of this study was to investigate whether the educational programming environments used would help such students to comprehend or deepen on concepts they had not mastered. Also, the study aimed to investigate which environment students consider the most effective for an introduction to programming no matter the age of the learners, and moreover in which context they consider each environment could be used successfully.
Table 1 Participants Year of studies Percentage
3 2 5.41
4 28 75.68
>4 7 18.92
Total 37 100
%
%
%
%
Educ Inf Technol (2017) 22:145176 155
Table 2 Students knowledge of
OOP and Java Level of knowledge N Percentage
Basic 18 48.65
Advanced 18 48.65
Expert 1 2.7
Total 37 100
Research Question 2 (RQ2): What are the advantages and disadvantages of the five educational programming environments?
RQ2 aimed to record students perceptions on the pros and cons of each one of the environments that were studied and used by students throughout the semester. The data collected from this question was expected to help evaluate the five environments under the same conditions, but most important to draw conclusions on the effectiveness of the wide variety of features incorporated in them. This was considered to assist in answering the main question of the study stated in the title.
Research Question 3 (RQ3): What features are considered important for an introductory programming environment in terms of its GUI, visualization capabilities, editor, error messages and execution?
RQ3 aimed to address directly the main question of the study regarding the features that are important for an introductory programming environment. Students were asked to define important features for each one of the widely accepted and utilized categories proposed by McIver (2002) for evaluating programming environments for novices.
3.3 Data collection and analysis
In order to investigate the aforementioned research questions, students filled in a questionnaire after the end of the lessons. The questions are provided in the Appendix and fall into three categories that correspond to the aforementioned research questions:
& Pedagogical usage of the studied environments (Q1, Q2, Q3)& Evaluation of the studied environments (Q4)& Investigation of main features of an introductory programming environment (Q5)
Q2 was a Likert type question and Q3 provided some choices but students could also provide freely a response. The rest of the questions were open type questions.
The open type questions were analyzed by the three authors in consecutive sessions in order to group their answers in categories and validate them. The results of students replies in the questionnaire are presented and analyzed in the following section.
%
%
%
%
156 Educ Inf Technol (2017) 22:145176
4 Analysis of the results
4.1 Pedagogical usage of educational programming environments
4.1.1 Enhanced comprehension of programming concepts
Question 1 (Q1): Did any of the environments help you comprehend or deepen on concepts (i.e. object, class, inheritance) that you had been taught in other courses and you had not fully comprehended?
In this question 54 % of the students (20 students) stated that they already had good knowledge of OOP. The rest 46 % of the students (17 students) stated that the educational programming environments helped them clarify some OOP concepts. Specifically, these students stated that the educational programming environments used in the context of the course helped them better comprehend both basic (objects, classes, methods) and advanced OOP concepts. Students replies are presented in Fig. 1. The reply Bbasic concepts^ is quite subjective and could refer to different concepts based on the environment. For example, in BlueJ and objectKarel most probably it refers at least to objects and classes, while in Scratch to control structures.
The majority of students (10 out of 17 students - 59 %) were benefited by BlueJ that helped them comprehend the most concepts in comparison with the rest of the
MIT App Inventor
Alice
Scratch
objectKarel
BlueJ
Polymorphism/Overriding
objectKarel
Inheritance
objectKarel
BlueJ
MIT App Inventor
0 1 2 3
Methods/messages
Scratch
Alice
BlueJ
Classes
BlueJ
Objects
Scratch
BlueJ
MIT App Inventor
Basic concepts
Scratch
objectKarel
BlueJ
Number of Students
Fig. 1 Support provided in comprehending programming concepts (N=17)
Educ Inf Technol (2017) 22:145176 157
environments: basic concepts; objects; classes; methods/messages; and inheritance. The concepts of class and inheritance are visualized in the simplified UML class diagram presented in the main window of BlueJ, while objects are also depicted in the main window. Moreover, objects are created interactively using direct manipulation and each objects methods are invoked in the same way. No one from the students benefited by BlueJ made a reference to polymorphism and overriding. This might be due to the fact that these concepts are not directly supported by the visualization and interactivity features of BlueJ and are mostly connected to the source code. A reference both to basic concepts and the more abstract concepts of inheritance, polymorphism and overriding was made for objectKarel, where students can watch simultaneously the line of code being executed each time and the way the object responds. For example, when a polymorphic method call is made for a robot the flow of control is transferred to the method declared in the class that the robot belongs to and the robot responds to this polymorphic method call with the way dictated by its type/class. Having robots from different classes in an inheritance hierarchy execute a polymorphic/overriden method helps students clearly comprehend the underlying concepts. It seems that step by step execution with visualization of the results seems to make abstract concepts more easily comprehended, as recorded in a previous study with objectKarel (Xinogalos et al. 2006a). Scratch was the second more referenced environment after BlueJ that helped 35 % (6 out of 17 students) of the students to comprehend OOP concepts, although its support for OOP concepts is limited (duplicate objects, method calling). Students mentioned that Scratch helped them comprehend basic concepts, as well as the OOP concepts supported by the environment: objects and methods/ messages. Both MIT App Inventor and Alice helped students comprehend basic concepts and the former also methods/messages.
4.1.2 The most effective environment for teaching introductory programming
Question 2 (Q2): What is the most effective environment for an introduction to programming independently of the educational level?
Students replies in this question are summarized in Fig. 2, while more statistical data are presented in Table 3.
As expected, Scratch is considered by students the most effective environment for an introduction to programming for all educational levels (T3.1: Mean=4.49, Std. Dev.=0.77). The next more effective environment with a considerable Bdistance^ is
Alice
objectKarel
BlueJ
MIT App Inventor
Scratch
0 1 2 3 4 5
Fig. 2 Mean value - effectiveness for an introduction to programming (scale 1 to 5)
2,76
3,22
3,38
3,59
4,49
158 Educ Inf Technol (2017) 22:145176
Table 3 The most effective environment for an introduction to programming (N=37)
Mean Std.Dev. Confidence 1 2 3 4 5
T3.1 Scratch 4.49 0.77 0.25 0 1 3 10 23
T3.2 MIT App Inventor 3.59 1.07 0.34 1 4 13 10 9
T3.3 BlueJ 3.38 1.06 0.34 1 7 12 11 6
T3.4 objectKarel 3.22 1.11 0.36 1 11 9 11 5
T3.5 Alice 2.76 1.09 0.35 6 8 13 9 1
considered to be MIT App Inventor (T3.2: Mean=3.59, Std. Dev.=1.07), followed with a small difference by BlueJ (T3.3: Mean=3.38, Std. Dev.=1.06) and objectKarel (T3.4: Mean=3.22, Std. Dev.=1.11), and finally with a big difference by Alice (T3.5: Mean=2.76, Std. Dev.=1.09).
The placement of MIT App Inventor in the second place and Alice in the last was
quite unexpected, since MIT App Inventor is oriented to mobile app development and not to fundamental programming concepts/constructs. This might be due to the great interest for mobile apps nowadays and consequently the motivation provided to students. However, students replies in question 3 that asked students to think separately for each environment what aim(s) it fulfils gave different results.
4.1.3 Usage of educational programming environments in different contexts
Question 3 (Q3): For what aim(s) is each one of the environments appropriate? You can choose one or more from the following or define other(s) aim(s):
Introduction to basic algorithmic structures & concepts Introduction to programming Introduction to OOP Introduction to fundamental OO concepts (e.g. object, class, inheritance)
36 students answered this question for BlueJ and all the students for the other four environments. Students replies in this question are presented in Fig. 3 and can be summarized as follows:
& Scratch and objectKarel are considered to be the most appropriate environments for an Bintroduction to basic algorithmic structures and concepts^ by 54 and 49 % of the students respectively (F3.1).
& Scratch and objectKarel are also considered appropriate for an Bintroduction to programming^ by the majority of students (F3.2: 70 and 65 % respectively).
& BlueJ seems to be the most popular environment both for an Bintroduction to fundamental OO concepts^ (F3.4: 83 %) and an Bintroduction to OOP^ (F3.3: 67 %). Scratch and objectKarel are also considered appropriate for an Bintroduction to OOP^ by the majority of students (F3.3: 57 and 51 % respectively).
Educ Inf Technol (2017) 22:145176 159
14%
MIT App Inventor
27%
22%
19%
30%
Alice
35%
27%
35%
35%
Scratch
70%
[F3.4] Introduction to fundamental OO concepts (e.g. object, class, inheritance)
[F3.3] Introduction to OOP
[F3.2] Introduction to programming
[F3.1] Introduction to basic algorithmic structures & concepts
54%
38%
object-Karel
51%
57%
65%
49%
83%
BlueJ
67%
47%
36%
Fig. 3 Usage of the environments in different contexts
& Alice is considered appropriate for the articulated aims by one-third (and less) of the students, while MIT App Inventor is considered appropriate for less than one-fourth of the students.
4.2 Advantages and disadvantages of educational programming environments
Question 4 (Q4): What are the main advantages and disadvantages of each environment?
4.2.1 BlueJ
Students replies in Q4 regarding BlueJ are presented in Table 4. The two most referenced advantages of BlueJ are the environments simplicity and usability (T4.1: 36 %) and the fact that it generally supports the comprehension of concepts (T4.2: 31 %). Some students mentioned specific features of BlueJ, namely: the presentation of a UML class diagram for a project in the main window (T4.3: 14 %); the visualization of concepts (T4.4: 11 %); and the interactive creation of objects and calling of methods (T4.5: 6 %). Consequently, students identified and evaluated positively all the main features and design decisions made by its developers (Klling et al. 2003).
Moreover, 2 students consider important the fact that the student is Bwriting code in Java^ (T4.6: 6 %) and not some educational language and another student made a reference to the Berror messages^ reported by BlueJ (T4.7: 3 %).
When we examine the disadvantages of BlueJ we see that more than one out of five students states that the environment is Bgood for beginners, but not for more experienced students^ (T4.8: 22 %). However, BlueJ was designed mainly for educational purposes and this cannot be actually considered as a disadvantage. Another disadvantage refers to the Bpoor graphics^ of BlueJs GUI (T4.9: 11 %). Probably this response was due to the GUI graphics and the graphical output of the problems solved in the rest
160 Educ Inf Technol (2017) 22:145176
Table 4 The advantages and disadvantages of BlueJ (N=36)
Advantages Disadvantages
T4.1 Simplicity/Usability 36 % (13) T4.8 Good for beginners, but not for more experienced students
11 % (4)
T4.4 Visualization 11 % (4) T4.11 Requires prior knowledge 8 % (3)
T4.5 Interaction 6 % (2) T4.12 Neglecting control flow during program execution
T4.6 Writing code in Java 6 % (2)
T4.7 Error messages 3 % (1)
of the environments used in the context of the course. The most important of the disadvantages mentioned, in our opinion, lies in the fact that BlueJ does not support program development (T4.10: 11 %). More specifically, two out of the 36 students mentioned as a disadvantage that the student Bwrites code^ and another 2 students that BlueJ Bdoes not protect from syntax errors^. In these students we can add three more students which mentioned that BlueJ Brequires prior knowledge^ (T4.11: 8 %), in the sense that there is no support in program development as in the other environments (e.g. structure/block editor). Of course we have to take into account that BlueJ supports Java and not some kind of a mini-language.
4.2.2 objectKarel
Students replies in Q4 regarding objectKarel are presented in Table 5. The most referenced advantages of objectKarel based on students replies are that the environment is Bgood for learning programming concepts^ (T5.1: 38 %) and it is characterized by simplicity and usability (T5.2: 30 %). Another recorded advantage refers to the incorporation of the structure editor in objectKarel that according to 13 % of the students supports program development (T5.3). More specifically, 3 students stated that there is Bno need for memorizing syntax^ and 2 students stated that the structure editor Bprotects users from syntax errors^. Other advantages recorded are the following: Bentertaining environment^ (T5.4: 11 %); Borganization of activities and theory^ (T5.6: 5 %); Bgood examples are incorporated in the environment^ (T5.7: 3 %). An advantage that we anticipated that would be identified by more students refers to the execution of programs in objectKarel that utilizes both program animation and explanatory visualization. In Q4 just one student mentioned Bstep by step execution^ and another one that BobjectKarel shows how statements work^ (T5.5: 5 %). However, in Q5 that is analyzed in section 4.3 more than one third of the students made a reference in the execution system of objectKarel.
The most obvious disadvantage according to one third of students is program development in objectKarel (T5.8: 30 %). More specifically, 7 (out of 37) students commented as a problem Bprogram development^, 3 students Bcorrecting the
22 % (8)
T4.2 Supports comprehension of concepts 31 % (11) T4.9 GUI: poor graphics 11 % (4)
T4.3 UML class diagram 14 % (5) T4.10 Does not support program development
3 % (1)
Educ Inf Technol (2017) 22:145176 161
Table 5 The advantages and disadvantages of objectKarel (N=37)
Advantages Disadvantages
T5.1 Good for learning programming concepts 38 % (14) T5.8 Program development 30 % (11)
T5.2 Simplicity/Usability 30 % (11) T5.9 Not many programming capabilities
8 % (3)
8 % (3)
T5.6 Organization 5 % (2) T5.13 Not user-friendly 5 % (2)
T5.7 Good examples 3 % (1)
program^ and 1 student the Black of copy-paste^ within the structure editor. To these students we can add 3 more students which mentioned that the environment is Bdifficult to use at the beginning^ (T5.11: 8 %) and 2 students that consider the environment is Bnot user-friendly^ (T5.13: 5 %). Consequently, although the structure editor is one of the main features of objectKarel that it is supposed to support novices in learning programming, a significant number of students consider it as a disadvantage. However, this result is not surprising. Structure editors restrict users in specific actions and make editing more difficult since they do not allow violating syntactic rules, which is sometimes the easiest way for correcting a piece of code that does not give the desired results. Novices overlook these disadvantages and appreciate the support provided by a structure editor: no need to memorize syntax; avoidance of syntax errors that are difficult for novices; guidance through program development. More experienced students that are accustomed to using typical editors for writing code, as is the case in this study, seem to prefer the flexibility of such editors.
Other disadvantages of objectKarel are the following: it Bdoes not offer many programming capabilities^ (T5.9: 27 %); it is Busing the same scenario (robots)^ for teaching various concepts and this fact Bdoes not make it very attractive^ (T5.10: 8 %).
4.2.3 Scratch
The advantages and disadvantages of Scratch as recorded in students replies are presented in Table 6.
Several advantages, but disadvantages as well, were recorded in students replies for Scratch. Nearly one third of the students made a reference to the usability and simplicity of Scratch (T6.1: 32 %). On the other hand, students mentioned various relevant problems: Scratch has Bmany functionalities that distract^ (T6.12: 11 %), while Bin complex programs it is confusing^ (T6.12: 3 %); it is Bdifficult using it at the beginning^ (T6.13: 11 %); Bthere are some bugs^ (T6.15: 5 %); Bworking in one screen^ (T6.18: 3 %); Bno warning for potential bugs^ (T6.20: 3 %).
Several students were fond of the Bnice graphics^ (T6.2: 27 %) and the fact that the environment offers Bmany capabilities^ (T6.3: 24 %), such as Beasy creation of games^ (T6.9: 5 %). Another advantage of Scratch is the support provided in program
27 % (10)
T5.3 Supports program development 13 % (5) T5.10 Not very attractive 8 % (3)
T5.4 Entertaining 11 % (4) T5.11 Difficult to use at the beginning
T5.5 Execution 5 % (2) T5.12 Using the same scenario
(robots)
162 Educ Inf Technol (2017) 22:145176
Table 6 The advantages and disadvantages of Scratch (N=37)
Advantages Disadvantages
T6.1 Usability/Simplicity 32 % (12) T6.12 Distracting/confusing environment
T6.2 Nice graphics 27 % (10) T6.13 Difficult using it at the beginning
9 % (3)
T6.4 Supports program development 19 % (7) T6.15 There are some bugs 5 % (2)
T6.5 Good for teaching introductory programming
3 % (1)
T6.7 Mother tongue 8 % (3) T6.18 Working in one screen 3 % (1)
T6.8 Critical thinking/Creativity 8 % (3) T6.19 Not OO enough 3 % (1)
T6.9 Easy creation of games 5 % (2) T6.20 No warning for potential bugs
T6.10 Interactivity 5 % (2)
T6.11 Community 5 % (2)
development (T6.4: 19 %) and more specifically the Bdrag and drop code, simple statements, categorization of statements^ (14 %) and the fact that it Bprotects from syntax errors^ (5 %). A relevant problem that was referenced by just one student is that the environment Bdoes not explain why a block does not fit^ (T6.17: 3 %). Also, 3 students consider as a disadvantage that there is actually Bno code writing^, that the Bcode is not realistic^ and that there is a Brestricted number of statements^ (T6.14: 9 %).
Finally, the following advantages that refer to important features of Scratch were recorded, but in low percentages: it is Bgood for teaching introductory programming^ (T6.5: 14 %); it is Bentertaining^ (T6.6: 14 %); it Bpromotes critical thinking and creativity^ (T6.8: 8 %); it is characterized by a high degree of Binteractivity^ (T6.10: 5 %); and has a large Bcommunity^ (T6.11: 5 %).
4.2.4 Alice
The advantages and disadvantages of Alice, as recorded in the replies of 35 out of the 37 students, are presented in Table 7. The most referenced advantage is the many capabilities offered by Alice and the impressive results, while the most important disadvantage recorded in the majority of students replies is the complexity of the environment.
Forty percent of the students commented on the various capabilities offered by Alice and the very nice results (T7.1: 40 %): the fact that Alice offers Bmany capabilities (even modeling, math,)^ was mentioned by 10 students; the Bnice, even impressive results^ was mentioned by 3 students; and the Beasy creation of games^ by 1 student.
14 % (5)
11 % (4)
T6.3 Many capabilities 24 % (9) T6.14 No code writing/codeis not realistic/ restrictednumber of statements
14 % (5) T6.16 Limited capabilities 5 % (2)
T6.6 Entertaining 14 % (5) T6.17 Does not explain why a block does not fit
3 % (1)
Educ Inf Technol (2017) 22:145176 163
14%(5)
Table7TheadvantagesanddisadvantagesofAlice(N=35)
AdvantagesDisadvantages
T7.1Offersmanycapabilities(modeling,math,games,)andveryniceresults40%(14)T7.11Complex/notuser-friendlyenvironment57%(20)
T7.23Dgraphics17%(6)T7.12Programdevelopmentisdifficult(fornovices)17%(6)
T7.3Supportsprogramdevelopment11%(4)T7.13Discouragingfornovices(volumeofinformation,
familiarization)
T7.4(Relatively)user-friendly9%(3)T7.14Manipulationofobjects6%(2)
T7.5Goodforintroductiontoprogramming\code9%(3)T7.15Problemsinsynchronizationofversions3%(1)
T7.6Entertaining\resemblesagame9%(3)T7.16GUIcouldbenicer3%(1)
T7.7Visualizationofprogramdevelopment\execution6%(2)T7.17Limiting3%(1)
T7.8Interactivity3%(1)
T7.9ThegeneratedcodeisclosertorealcodeincomparisontoScratch3%(1)
T7.10Creativethinking3%(1)
164 Educ Inf Technol (2017) 22:145176
These impressive results are mainly due to the use of B3D graphics^ (T7.2: 17 %). Another advantage is the support provided for program development (T7.3: 11 %) through the use of the Bblock editor^ (2 students) that helps Bavoid memorization of syntax^ and Bsyntax errors^ (1 student). Alice is considered Bgood for an introduction to programming or code^ (T7.5: 9 %) and it provides an Bentertaining, different teaching approach resembling a game^ (T7.6: 9 %). 2 students mentioned Bvisual representation of program execution^ and that a Bproject is visible while developed and can be easily improved^ (T7.7: 6 %). BInteractivity^ and Bpromotion of creative thinking^ were mentioned by just 1 student each (T7.9, T7.10: 3 %).
The main disadvantage of Alice refers to its complexity (T7.11: 57 %). Specifically, students made the following comments:
Bcomplex environment^ (34 %) Bnot user-friendly^ (14 %) Bthe initial screen should be simpler^ (3 %) Bhaving different functionalities in various places of the main screen confuses^ (3 %) Bcould not decide what and how to do it - more complex than the other environments^ (3 %)
Moreover, some students brought up two issues that they think are discouraging for novices:
& Program development is difficult (for novices) (T7.12: 17 %): Alice Brequires prior knowledge of programming^ according to 3 students; the Bstatements are not simple for novices^ according to 2 students; and Bimplementation is difficult^ according to 1 student.
& The environment is discouraging for novices (T7.13: 14 %): Benough time is required for get familiarized^ as stated by 2 students; Bthe volume of the information provided can confuse novices^ as stated by another 2 students; or it generally Bdiscourages novices^ as stated by 1 student.
4.2.5 MIT App Inventor
Students replies in Q4 regarding MIT App Inventor are presented in Table 8. Students reported several advantages for MIT App Inventor. As noted by students Bmobile apps are interesting nowadays^ (T8.5: 19 %) and MIT App Inventor gives the chance of Bdesigning and implementing easily^ (T8.3: 22 %) Bgood quality apps^ (T8.4: 19 %). More than one fourth of the students made a reference to the usability and simplicity (T8.1: 28 %) of the environment that constitutes it Bgood for an introduction to Android^ (T8.2: 28 %). MIT App Inventor is good for novices, since it supports them in the whole process:
& Bgood tutorials^ (T8.9: 8 %)
Educ Inf Technol (2017) 22:145176 165
Table 8 The advantages and disadvantages of MIT App Inventor (N=36)
Advantages Disadvantages
T8.1 Usability/Simplicity 28 % (10) T8.13 Not so user-friendly/ complete
19 % (7)
T8.2 Good for an introduction to Android
28 % (10) T8.14 Faced problems 19 % (7)
T8.3 Designing/implementing apps easily
22 % (8) T8.15 Limited capabilities 11 % (4)
T8.4 Implementing good quality apps
19 % (7) T8.16 Not so appropriatefor an introduction to programming
6 % (2)
T8.5 Mobile apps are interesting nowadays
19 % (7) T8.17 Not many graphics 3 % (1)
T8.6 Supports program development
17 % (6) T8.18 No coding the transition to an editor will be difficult
3 % (1)
T8.7 Easy testing 14 % (5)
T8.8 Special features/capabilities
(no installation, web viewers)
14 % (5)
T8.9 Good tutorials 8 % (3)
T8.10 Using complex tools through the GUI
3 % (1)
T8.11 Close to programming philosophy with the usage of external libraries
3 % (1)
T8.12 Promotes creative thinking 3 % (1)
& program development (T8.6: 17 %): Bdrag and drop code^ was mentioned by 3 students; Bavoidance of syntax errors^ was mentioned by 2 students; and Bvery good organization of statements^ was mentioned by 1 student.
& easy testing (T8.7: 14 %): Bwatching the project while being developed gives the chance for improving it^ (2 students); Byou can download the app in your mobile device and test it^ (1 student); Bfunctionality and communication with the emulator was very good and useful^ (1 student); Bway of presentation^ (1 student).
& Busing complex tools through the GUI^ (T8.10: 3 %)
As expected, some students also mentioned the Bspecial features/capabilities^ (T8.8: 14 %) of MIT App Inventor, such as the fact that: Bno installation is required^; Bfunctionalities not supported by other environments (such as web viewers)^ are offered; it constitutes Bthe most complete environment^.
Besides all the aforementioned advantages, 19 % of the students considered that the environment is not so user-friendly (T8.13: 19 %). These students mentioned that the environment Bis not so user-friendly/complete^ or Bthat is tiring^ (5 students), Bthe user must devote some time in order to comprehend the environment^ (1 student), and that is
166 Educ Inf Technol (2017) 22:145176
Bdifficult for a novice to understand and use its capabilities^ (1 student). The same percentage of students faced problems with MIT App Inventor (T8.14: 19 %), namely: Bslow emulator with limited capabilities^ (2 students); Bbugs (i.e. source code is unexpectedly deleted)^ (2 students); Bhidden functionalities^ (1 student); Bsecurity issues with Java problems with Java 7^ (2 students). The limitations of MIT App Inventor are the following (T8.15: 11 %): Blimited size of apps^ (2 students); Bnot for professional use limited capabilities^ (1 student); Bno ability to duplicate code^ (1 student).
4.2.6 The main advantages and disadvantages of the environments
The main advantages and disadvantages of the five environments, presented analytically in Tables 4, 5, 6, 7 and 8, are summarized in Fig. 4 and Fig. 5 respectively. The advantages/disadvantages that were reported by at least 10 % of the students are included in Fig. 4 and Fig. 5.
Regarding the advantages, students made references to features that could be grouped into the following categories:
& GUI: simplicity/usability (F4.1: BlueJ-36 %, objectKarel-30 %, Scratch-32 %, MIT
App Inventor-28 %); nice graphics (F4.2: Scratch-27 %); 3D graphics (F4.3: Alice-17 %).& Pedagogical support: supports comprehension of concepts (F4.4: BlueJ-31 %); good for learning programming concepts (F4.5: objectKarel-30 %); teaching
BlueJ objectKarel Scratch Alice MIT App Inventor
MIT App Inventor
objectKarel, Scratch, Alice, MIT App Inventor
BlueJ
objectKarel, Scratch
MIT App Inventor
Scratch
Alice
objectKarel, Scratch
BlueJ
Alice
BlueJ, objectKarel, Scratch, MIT App Inventor
MIT App Inventor
[F4.17] Special features/capabilities (no installation, web viewers)
14%
[F4.16] Easy testing
14%
[F4.15] Implementing good quality apps
19%
[F4.14] Designing/implementing apps easily
22%
[F4.13] Supports program development
13%
19%
11%
17%
[F4.12] Visualization
11%
[F4.11] UML class diagram
14%
[F4.10] Entertaining
11%
14%
[F4.9] Mobile apps are interesting nowadays
19%
[F4.8] Many capabilities
24%
[F4.7] Offers many capabilities (modeling, math, games,...) and very nice results
[F4.6] Good for an introduction to Android
28%
[F4.5] Good for learning programming concepts/ teaching introductory programming
30%
14%
[F4.4] Supports comprehension of concepts
31%
[F4.3] 3D graphics
17%
40%
[F4.2] Nice graphics
27%
Scratch
[F4.1] Simplicity/Usability
36%
30%
32%
28%
Fig. 4 The main advantages of the environments
Educ Inf Technol (2017) 22:145176 167
BlueJ objectKarel Scratch Alice MIT App Inventor
[F5.12] Good for beginners, but not for more experienced students
BlueJ
[F5.11] Limited capabilities
11%
MIT App Inventor
[F5.10] Not many programming capabilities
27%
objectKarel
[F5.9] Does not support program development
11%
22%
30%
BlueJ
[F5.8] Program development/editing
17%
objectKarel, Alice
[F5.7] GUI: poor graphics
11%
BlueJ
[F5.6] Difficult using it at the beginning
11%
Scratch
[F5.5] Distracting/confusing environment
14%
[F5.4] Faced problems
19%
MIT App Inventor
[F5.3] Not so user-friendly/complete
19%
[F5.2] Discouraging for novices (volume of information, familiarization)
14%
Alice
[F5.1] Complex / not user-friendly
57%
Alice
Fig. 5 The main disadvantages of the environments
introductory programming (F4.5: Scratch-14 %); good for an introduction to Android (F4.6: App Inventor-28 %).& Capabilities/types of programs: offers many capabilities (modeling, math, games,) and very nice results (F4.7: Alice-40 %); many capabilities (F4.8: Scratch-24 %); mobile apps are interesting nowadays (F4.9: App Inventor-19 %); entertaining (F4.10: objectKarel-11 %, Scratch-14 %).& Visualization: UML class diagram (F4.11: BlueJ-14 %); visualization (F4.12:
BlueJ-11 %).& Program development (editor): supports program development (F4.13: objectKarel-13 %, Scratch-19 %, Alice-11 %, App Inventor-17 %); designing/ implementing apps easily (F4.14: App Inventor-22 %); implementing good quality apps (F4.15: App Inventor-19 %).& Testing: easy testing (F4.16: App Inventor-14 %).
Regarding the disadvantages, students made references to features that could be grouped into three main categories:
& GUI: complex/not user-friendly (F5.1: Alice-57 %); discouraging for novices (F5.2:
Alice-14 %); not so user-friendly/complete (F5.3: App Inventor-19 %); faced problems (F5.4: App Inventor-19 %); distracting/confusing environment (F5.5: Scratch-14 %); difficult using it at the beginning (F5.6: Scratch-11 %); poor graphics (F5.7: BlueJ-11 %).& Program development (editor): program development/editing is difficult (F5.8: objectKarel-30 %, Alice-17 %); does not support program development (F5.9: BlueJ-11 %).
168 Educ Inf Technol (2017) 22:145176
& Capabilities/types of programs: not many programming capabilities (F5.10: objectKarel-27 %); limited capabilities (F5.11: App Inventor-11 %).
We must stress that some of the disadvantages that refer to the GUI and the editor might be attributed to the fact that the students were not novices. As already mentioned, some students that have experience in programming seem to have difficulty in implementing programs that are not based on text. Moreover, some students may simply be non-visual learners. Finally, mapping the semantics of shapes to already known statements might cause a cognitive overload to some students (Green and Petre 1996), with no added value in the case they already have experience in programming.
4.3 Important features for an introductory programming environment
In section 4.2 students replies on an open-ended question (Q4) regarding the advantages and disadvantages of each one of the five environments were analyzed. Students made references to features regarding the GUI of the environment, the pedagogical support provided, the capabilities or else the types of programs developed in the environment, its visualization capabilities, and finally the way program development is accomplished in the incorporated editor. In this section, students replies regarding the main characteristics that any introductory programming environment should have under specific categories are analyzed. Some of these categories are the same as the ones proposed by students themselves, namely: GUI, visualization, and editor. Moreover, question 5 (Q5) made a reference to compiler/error messages and execution. The question posed was the following:
Question 5 (Q5): What are the main characteristics that an introductory programming environment should incorporate for each one of the following modules: GUI, Visualization, Editor, Compiler/Error messages, Execution? You can make reference to the environments you used.
Students references to the five environments when describing the features they consider important for an introductory programming environment are summarized in Fig. 6, while the features themselves are presented in Fig. 7. Students were asked to define the main features for the categories proposed in McIver (2002) with the following exceptions:
& BDebugger^: students were not asked to provide features for this category, since: this ability was not presented to students due to time restrictions; this capability is offered only by BlueJ; some environments use some kind of debugger as the standard execution system, such as objectKarel.
& BIndividual method execution and object construction^: this category was substituted by the more general category BExecution^.
Educ Inf Technol (2017) 22:145176 169
9%
6%
Execu on
21%
36%
3%
9%
6%
Compiler Error messages
9%
6%
15%
31%
Editor
40%
[F6.5] App Inventor
[F6.4] Alice
[F6.3] Scratch
[F6.2] objectKarel
[F6.1] BlueJ
9%
9%
14%
Visualiza on
34%
40%
25%
6%
9%
11%
GUI
42%
6%
9%
14%
Fig. 6 References to the environments regarding fundamental features
4.3.1 Graphical User Interface (GUI)
Nearly one third of the students identified as the most important feature for the GUI of an educational programming environment simplicity and user-friendliness (F7.1:31 %). Usability and simplicity was identified in Q4 regarding the advantages and disadvantages of the educational programming environments used as one of the main advantages for 4 out of the five environments with nearly the same percentages: BlueJ (T4.1: 36 %); objectKarel (T5.2: 30 %); Scratch (T6.1: 32 %); MIT App Inventor (T8.1: 28 %). The only exception was Alice that was characterized as B(relatively) user-friendly^ by just 9 % (T7.4) of the students, while the majority (T7.11: 57 %) of them identified as the most prevalent disadvantage that the environment is Bcomplex and not user-friendly^. For App Inventor also 19 % of the students stated that they find the environment not so user user-friendly (T8.13).
Other features identified by students regarding GUI, but with lower or even minimal percentages are graphics (F7.3: 8 %) and feedback/interactivity (F7.6: 3 %). Two features referenced by one student each that are not related to the GUI in general but to programming more specifically is the use of characters (F7.4: 3 %), such as the ones used in microworlds, and adoption of visual programming (F7.5: 3 %).
The environment that seems to better fulfill collectively the aforementioned features is Scratch that was referenced by 42 % (F6.3) of the students in Q5, followed by MIT App Inventor (F6.5: 25 %).
4.3.2 Visualization
Visualization of classes and objects seems to be important for more than one third of the students. 29 % of them mentioned the visualization of both classes and objects (F7.7),
170 Educ Inf Technol (2017) 22:145176
GUI VISUALIZATION EDITOR COMPILER/ ERROR MESSAGES EXECUTION
Fig. 7 Important features for an introductory programming environment
while an additional 9 % (F7.9) of them considered important the usage of characters/ actors as objects in the programs (like robots and so on). Some students considered important even the visualization of statements as puzzle pieces (F7.8: 14 %).
BlueJ was referenced by 40 % of the students (F6.1) and Scratch by 34 % (F6.3) in their replies regarding visualization (in the context of Q5). In Q4 students identified the following advantages for BlueJ that are relevant to visualization: UML class diagram (T4.3: 14 %); visualization (T4.4: 11 %); interaction with classes and objects (T4.5: 6 %). Moreover, the Bsupport provided for comprehending concepts^ in BlueJ (T4.2: 31 %) could be largely attributed to the visualization and direct manipulation features of the environment.
4.3.3 Editor
Puzzle-like statements seem to be the most popular feature regarding the editor of an educational programming environment. Nearly one third of the students identified a puzzle-like editor (F7.10: 31 %) and just 6 % (F7.11) of the students identified a structure editor as an important feature. Adding the 6 % of the students which think that the editor must prevent errors (F7.12) we see that 43 % of the students consider that novices should be supported in developing programs and should not write code.
As expected, most students referenced Scratch (F6.3: 40 %) and MIT App Inventor (F6.5: 31 %) when describing the features of the editor they consider important.
4.3.4 Compiler/error messages
Error messages are a great source of difficulties not only for novices, but for more experienced students as well. According to students a good compiler must report Bsimple and understandable messages^ (F7.15: 21 %), as well as Bthe exact line of the error^ (F7.16: 18 %). The messages reported should Bpropose ways of correcting the error^ (F7.17: 12 %) and use Bnatural language^ (F7.18: 12 %). A small
Educ Inf Technol (2017) 22:145176 171
percentage of students mentioned that a Bprogram should compile during development^ (F7.19: 6 %) and Bpop-up warnings^ should be reported (F7.20: 6 %), features that students knew from their prior experience with Eclipse. Finally, one student mentioned that errors should be prevented in the first place, which of course requires some kind of structure editor.
4.3.5 Execution
The most referenced feature not only for the category of execution, but for all categories included in Q5, was step by step execution (F7.22: 42 %). BInspecting elements^ (F7.23: 3 %), Bwatching variables^ (F7.24: 3 %) and Bexecuting during development^ were mentioned by 1 student each.
The execution system that seems to suit better to novices needs is objectKarel that was referenced by 36 % of the students (F6.2), followed by Scratch that was referenced by 21 % of the students (F6.3).
5 Discussion and conclusions
The study presented in this paper investigated the perceptions of students regarding 5 educational programming environments that are widely used and the features that an ideal introductory programming environment should incorporate. The study was based on a questionnaire filled in by 4th year undergraduate students of an Informatics Department attending a course on BEducational Programming Environments and Techniques^. The same group of students having prior experience in programming languages and environments, as well as the accompanying difficulties of learning programming, used the five environments in the context of the course and evaluated them. The main conclusions that are drawn are presented in the following paragraphs.
Regarding the pedagogical usage of the educational programming environments (RQ1) examined the most important results can be summarized as follows:
Based on the replies in Q1, 54 % of the students had already had a good knowledge of OOP programming. All the environments helped more or less the rest of the students comprehend or deepen on various programming concepts: basic concepts (all the environments); objects (BlueJ, Scratch); classes (BlueJ); methods/messages (BlueJ, Scratch, App Inventor); inheritance (BlueJ, objectKarel); polymorphism/overriding (objectKarel). The term Bbasic concepts^ of course is quite subjective and can refer to different concepts based on the environment: objects and classes in BlueJ and objectKarel, control structures in Scratch and so on. The environment that benefited the majority of students (59 %) to comprehend or deepen on all the recorded concepts, with the exception of polymorphism and overriding, was BlueJ. Consequently, this study confirms that an educational programming environment cannot be used only for an introduction to programming, but also for helping students in dealing with difficulties from a prior introduction to programming with a conventional programming language and environment (Xinogalos et al. 2006a).
Regarding the environment that is considered by students to be the most effective for an introduction to programming for all educational levels (Q2), Scratch was a clear winner. In the second position but with a considerable Bdistance^ students
172 Educ Inf Technol (2017) 22:145176
placed MIT App Inventor, followed in the third position with a small difference by BlueJ and objectKarel, and finally with a bigger difference by Alice. Both Scratch and App Inventor engage more students most probably due to the fact that they give them the chance to write programs connected to their interests, such as games, simulations, stories (Utting et al. 2010) and mobile apps (Papadakis et al. 2014). Alice also gives the chance to develop programs with impressive results connected to students interests, however several students in this study considered it complex and not user-friendly (Q4) and this is the most prominent reason for considering it the least effective from the examined environments for an introduction to programming. So, an environment used for an introduction to programming should engage students through the development of programs connected to students interests using a simple and user-friendly interface.
However, an environment that engages students and is simple is not always the best choice for an introduction to either basic or object-oriented programming (concepts). In Q5 that asked students to specify for which of the following aims
& introduction to basic algorithmic structures and concepts & introduction to programming& introduction to fundamental OO concepts& introduction to fundamental OOP
each environment is appropriate, quite different results were recorded in comparison with the open-type question Q4. Specifically, Scratch was still among the first choices, but App Inventor was considered appropriate for less than one-fourth of the students. Scratch and objectKarel were considered to be the most appropriate environments both for an Bintroduction to basic algorithmic structures and concepts^ (by 54 and 49 % of the students respectively) and an Bintroduction to programming^ (by 70 and 65 % respectively). Scratch and objectKarel were also considered appropriate for an Bintroduction to OOP^ by the majority of students (57 and 51 % respectively). BlueJ was the most popular environment both for an Bintroduction to fundamental OO concepts^ (83 %) and an Bintroduction to OOP^ (67 %), while Alice was considered appropriate for the articulated aims by one-third (or less) of the students.
Taking into account the results regarding the pedagogical use, the advantages and disadvantages recorded for the five environments (Q4) in combination with students replies regarding the features that any introductory environment should have (Q5) the following conclusions can be drawn regarding students perceptions for the five environments (RQ2):
The most important advantages of BlueJ according to students are its simplicity/ usability (F4.1), the incorporated visualization features (F4.11 & F4.12) and the support provided for comprehending concepts (F4.4). Students distinguished BlueJ amongst the other environments for its visualization features (Fig. 6) and especially the visualization of classes and objects (F7.7). Also, the majority of students consider BlueJ as the most effective environment for an introduction both to fundamental OO concepts (F3.3) and programming (F3.4). However, they do not consider it as good for more experienced students (F5.12). The disadvantages outlined by a small percentage of students (F5.7 & F5.9: 11 %) are the poor
Educ Inf Technol (2017) 22:145176 173
graphics used in the interface of BlueJ and the lack of support for program development.
Scratch is clearly students first choice for an introduction to programming irrespectively of the educational level of the learners (Table 3). Specifically, Scratch was considered by the majority of students appropriate for an introduction to basic algorithmic structures and concepts, programming, even OOP (Fig. 3). The advantages of Scratch are several: it has nice graphics (F4.2); it offers many capabilities (F4.8); it is entertaining (F4.10); and it supports program development (F4.13). Students distinguished Scratch for its GUI as a whole (Fig. 6) and its editor with the puzzle-like statements (Fig. 6, F7.14). Simplicity and usability were referenced by 32 % of the students as one of the advantages for Scratch (F4.1), however several students mentioned the following relevant drawbacks: the environment is distracting/confusing (F5.5:14 %); and difficult using it at the beginning (F5.6: 11 %).
The most important advantages of objectKarel according to students are its simplicity/usability (F4.1) and the entertaining nature of the environment/problems (F4.10), while they consider the environment good for learning programming concepts (F4.5). Specifically, objectKarel was considered by nearly the same percentages with Scratch appropriate for an introduction to basic algorithmic structures and concepts, programming and OOP (Fig. 3). Students distinguished objectKarel amongst the other environments for its execution system (Fig. 6) that supports step by step execution (F7.22) using highlighting of the current statement with simultaneous depiction of its result on the robots world and explanation in natural language of the relevant semantics (explanatory visualization). The main disadvantage of objectKarel lies in the fact that it does not support many programming capabilities (F5.10). However, it was a main design decision of objectKarel to support the most fundamental OOP concepts in a straightforward manner. Of course objectKarel was designed 15 years ago and students interests and needs have changed towards programming games, simulations and mobile apps. Also, 30 % of the students reported difficulties with the structure editor (F5.8), in contrast with 13 % of the students that considered the editor to support program development (F4.13). As recorded in a previous study with two groups of students with different experience, a combination of a structure editor targeted to novices and a text editor for more experienced students would be ideal (Xinogalos et al. 2006b).
Alice is distinguished for the many capabilities it offers, such as modeling, math and games, and the very nice or even impressive results (F4.7) and its 3D graphics (F4.3). Although 11 % of the students state that Alice supports program development (F4.13), more students (as was the case for objectKarel as well) consider that program development is difficult in the environment (F5.8: 17 %). However, the main disadvantage of Alice is not this. The majority of students consider Alice to be complex and not user-friendly (F5.1), while some students (14 %) consider it even discouraging for novices, as recorded in their comments. Alice is considered appropriate for an introduction to algorithmic concepts (F3.1: 35 %), programming (F3.2: 27 %), OO concepts (F3.4: 30 %) or OOP (F3.3: 35 %) by approximately one-third (more or less) of the students.
MIT App Inventor differentiates from the other environments for the type of applications it supports. Mobile apps are interesting nowadays (F4.9) and App Inventor supports students in designing and implementing good quality apps (F4.15) easily (F4.14), testing is easy (F4.16) and it offers special features and
174 Educ Inf Technol (2017) 22:145176
capabilities (no installation, web viewers) (F4.17). So, App Inventor is considered good for an introduction to Android (F4.6). App Inventor was considered the second more effective environment after Scratch for an introduction to programming irrespectively of the educational level (Q2), probably due to the motivation provided by the development of mobile apps. However, when students were asked (Q3) to define explicitly if they consider each one of the examined environments appropriate for an introduction to basic algorithmic concepts and/or programming and/or OO concepts and/or OOP, App Inventor was considered appropriate for less than one fourth of the students (27 % for OOP). Finally, although 28 % of the students mentioned simplicity and usability (F4.1) as an advantage, 19 % of the students mentioned that the environment is not so user-friendly (F5.3) and 19 % that faced problems (F5.4). Frequent problems in usage are also recorded in (Papadakis et al. 2014).
Generally, students identified and evaluated positively the main features of the examined environments with few exceptions.
Regarding the main features that an introductory programming environment should have (RQ3), the main results that were recorded are the following:
& GUI: the most important feature of the GUI is to be simple and user-friendly (F7.1).
A good example of such an environment is Scratch (Fig. 6).& Visualization: visualization of classes and objects is the most important feature in this category (F7.7) and an exemplary environment that has successfully implemented this feature is BlueJ (Fig. 6).& Editor: the support for puzzle-like statements is clearly the most important feature for the editor (F7.10) and a good example of such an editor is Scratch (Fig. 6).& Compiler/error messages: it is important to report simple and understandable error messages (F7.15) with the exact line of the error (F7.16), and moreover to use natural language (F7.18) and propose ways of correcting the error (F7.17).& Execution: step by step execution (F7.22) was the most referenced feature among all the features referenced by students as important. A good example of an execution system is offered by objectKarel (Fig. 6).
Consequently, an ideal introductory programming environment should provide a simple and user-friendly GUI that supports visualization of objects and classes, includes a puzzle-like editor for program development, reports simple and understandable error messages in natural language that provide help for correcting them, and finally the ability to execute the program in a step by step manner. Although students find these features in the examined environments, there is no single environment that fulfils all these features. However, the most successful environment seems to be Scratch. Interested researchers could consider the aforementioned features as recommendations for designing new introductory programming environments.
Concluding, it must be highlighted that this study attempted to investigate the wide variety of features that have been incorporated in introductory programming environments and find out what is actually important for students. Relevant studies tried to evaluate the features and didactical rationale of a specific environment each time (Buck and Stucki 2001; Cooper et al. 2003; Xinogalos et al. 2006a) or reviewed various environments based on some sort of criteria (Deek and Mchugh 1998; Guzdial 2004;
Educ Inf Technol (2017) 22:145176 175
Kelleher and Pausch 2005). On the contrary, in this study five representative educational programming environments that fall into different categories were evaluated under the same conditions by the same group of last-year students having experienced the difficulties of learning programming using various programming environments during their studies. This helped us draw conclusions not only on the pedagogical value and pros and cons of each one of the five specific environments, but also on the most important features that any introductory programming environment should have. This is considered to be the main contribution of this work. The conclusions drawn could be utilized by anyone interested in the teaching and learning of programming, as well as in devising the next generations of programming environments. Future work could include developing a programming environment that encompasses all these features, as well as repeating the study with even more environments that are also used in practice.
Appendix - Questionnaire
Question 1 (Q1): Did any of the environments help you comprehend or deepen on concepts (i.e. object, class, inheritance) that you had been taught in other courses and you had not fully comprehended?
Question 2 (Q2): What is the most effective environment for an introduction to programming independently of the educational level?
For each environment give one answer in the scale of 1 to 5 where 1=least effective and 5=most effective.
1 2 3 4 5
BlueJ
objectKarel
Scratch
Alice
MIT App Inventor
Question 3 (Q3): For what aim(s) is each one of the environments appropriate? You can choose one or more from the following or define other aim(s):
Introduction to basic algorithmic structures & concepts Introduction to programming Introduction to OOP Introduction to fundamental OO concepts (e.g. object, class, inheritance)
Question 4 (Q4): What are the main advantages and disadvantages of each environment?
Question 5 (Q5): What are the main characteristics that an introductory programming environment should incorporate for each one of the following modules: GUI, Visualization, Editor, Compiler/Error messages, Execution? You can make reference to the environments you used.
176 Educ Inf Technol (2017) 22:145176
References
Barnes, D., & Klling, M. (2004). Objects First with Java: A practical introduction using BlueJ. EnglewoodCliffs: Prentice Hall.
Bergin, J., Stehlik, M., Roberts, J., & Pattis, R. (1997). Karel++ - a gentle introduction to the art of object-oriented programming, 2nd edn. New York: John Wiley and Sons.
Brusilovsky, P., Calabrese, E., Hvorecky, J., Kouchnirenko, A., & Miller, P. (1997). Mini-languages: a way to learn programming principles. International Journal of Education and Information Technologies, 2, 6583.
Buck, D., & Stucki, D. (2001). JKarelRobot: a case study in supporting levels of cognitive development in the computer science curriculum. ACM SIGCSE Bulletin, 33(1), 1620.
Cooper, S., Dann, W., & Pausch, R. (2003). Teaching objects-first in introductory computer science. ACMSIGCSE Bulletin, 35(1), 191195.
Deek, F., & McHugh, J. (1998). A survey and critical analysis of tools for learning programming. Journal ofComputer Science Education, 8(2), 130178.
Georgantaki, S., & Retalis, S. (2007). Using educational tools for teaching object oriented design and programming. Journal of Information Technology Impact, 7(2), 111130.
Green, T. R. G., & Petre, M. (1996). Usability analysis of visual programming environments: a cognitive dimensions framework. Journal of Visual Languages & Computing, 7(2), 131174.
Gross, P., & Powers, K. (2005). Evaluating assessments of novice programming environments. In Proceedings of the first international workshop on Computing education research, 99110.
Guzdial, M. (2004). Programming environments for novices. In Computer science education research, 2004,127154.
Kelleher, C., & Pausch, R. (2005). Lowering the barriers to programming: a taxonomy of programming environments and languages for novice programmers. ACM Computing Surveys (CSUR), 37(2), 83137. Klling, M., & Rosenberg, J. (2001). Guidelines for teaching object orientation with Java. In Proceedings of the 6th annual conference on Innovation and technology in computer science education (ITiCSE '01). ACM, New York, NY, USA, 3336.
Klling, M., Quig, B., Patterson, A., & Rosenberg, J. (2003). The BlueJ system and its pedagogy. Computer
Science Education, Special issue on Learning and Teaching Object Technology, 13(4), 249268. Maloney, J., Resnick, M., Rusk, N., Silverman, B., & Eastmond, E. (2010). The Scratch Programming
Language and Environment. Trans. Comput. Educ. 10, 4, Article 16 (November 2010), 15 pages.
McIver, L. (2002). Evaluating languages and environments for novice programmers. In Fourteenth Annual Workshop of the Psychology of Programming Interest Group (PPIG 2002), Brunel University, Middlesex, UK.
Miller, P., Pane, J., Meter, G., & Vorthmann, S. (1994). Evolution of novice programming environments: he structure editors of Carnegie Mellon University. Interactive Learning Environments, 4(2), 140158. Papadakis, S., Kalogiannakis, M., Orfanakis, V., & Zaranis, N. D2014]. Novice Programming Environments.
Scratch & App Inventor: a first comparison. In H. M. Fardoun & J. A. Gallud DEds.], Proceedings of the 2014 Workshop on Interaction Design in Educational Environments DIDEE '14] Dp. 1]. New York: ACM. 7 pages.
Pears, A., Seidman, S., Malmi, L., Mannila, L., Adams, E., Bennedsen, J., Devlin, M., & Paterson, J. (2007). A survey of literature on the teaching of introductory programming. ACM SIGCSE Bulletin, 39(4), 204 223.
Ragonis, N., & Ben-Ari, M. (2005). A Long-Term Investigation of the Comprehension of OOP Concepts byNovices. International Journal of Computer Science Education, 15(3), 203221.
Utting, I., Cooper, S., Klling, M., Maloney, J., & Resnick, M. (2010). Alice, Greenfoot, and Scratch ADiscussion. Transactions on Computing Education, 10, 4, Article 17, 11 pages.
Valentine, D. W. (2004). CS educational research: a meta-analysis of SIGCSE technical symposium proceedings. SIGCSE Bull, 36(1), 255259.
Xinogalos, S. (2012). Didactical Approach for Object-Oriented Programming: application in BlueJ. Proceedings of the 6th PanHellenic Conference on Didactics of Informatics, Florina 2022 April, 63 72 (in Greek).
Xinogalos, S., Satratzemi, M., & Dagdilelis, V. (2006a). An introduction to object- oriented programming with a didactic microworld: objectKarel. Computers and Education, 47(2), 148171.
Xinogalos, S., Satratzemi, M., & Dagdilelis, V. (2006b). Evaluating objectKarel - an educational programming environment for object oriented programming. In A. Mendez-Vilas et al. (Eds.), Current Developments in Technology-Assisted Education, vol. 2, 821825, Formatex Press.
Education and Information Technologies is a copyright of Springer, 2017.