Content area
Purpose
The purpose of this study is to develop an intelligent tutoring system (ITS) for programming learning based on information tutoring feedback (ITF) to provide real-time guidance and feedback to self-directed learners during programming problem-solving and to improve learners’ computational thinking.
Design/methodology/approach
By analyzing the mechanism of action of ITF on the development of computational thinking, an ITF strategy and corresponding ITS acting on the whole process of programming problem-solving were developed to realize the evaluation of programming problem-solving ideas based on program logic. On the one hand, a lexical and syntactic analysis of the programming problem solutions input by the learners is performed and presented with a tree-like structure. On the other hand, by comparing multiple algorithms, it is implemented to compare the programming problem solutions entered by the learners with the answers and analyze the gaps to give them back to the learners to promote the improvement of their computational thinking.
Findings
This study clarifies the mechanism of the role of ITF-based ITS in the computational thinking development process. Results indicated that the ITS designed in this study is effective in promoting students’ computational thinking, especially for low-level learners. It also helped to improve students’ learning motivation, and reducing cognitive load, while there’s no significant difference among learners of different levels.
Originality/value
This study developed an ITS based on ITF to address the problem of learners’ difficulty in obtaining real-time guidance in the current programming problem-solving-based computational thinking development, providing a good aid for college students’ independent programming learning.
Introduction
Computational thinking is the ability to solve problems through the development of computer programs (Leonard et al., 2016). It requires students to adopt basic concepts of computer science like computer scientists, solve problems, design systems and understand human behavior (Wing, 2006). It follows that computational thinking requires not only problem-solving strategies but also the knowledge to develop programs and perform computational processing (Hsu et al., 2018; Chao, 2016).
The essence of programming problem solving is to describe the steps to solve a problem in a computer-understandable way (Ma et al., 2022). Therefore, most of the current learning of computational thinking is in the form of programming learning, where programming language is used as a shell for computational thinking, and programming problem-solving activities are used to motivate learners to use computer science concepts and programming language to solve problems to improve students’ computational thinking (Lye and Koh, 2014). Previous studies have also shown that there is a strong link between computational thinking and programming problem-solving activities. Yi et al. (2022) constructed an evaluation model of CT based on digital games for programming problem-solving activities by correlating the aspects of CT used in problem-solving with the behaviors in programming problem-solving games; Jiang and Huang (2019) categorized computational thinking into three steps: discovering the problem, designing a solution and solving the problem, in which the formation of each step corresponds to the knowledge points of the computer science discipline; Boom et al. (2022) on the other hand, by investigating the relationship between different forms of computational thinking and two different measures of programming quality, showed that only computational thinking processes significantly predicted program quality; Some studies have even clearly indicated that problem-based programming learning has better learning outcomes and thinking skills (Basogain et al., 2018; Harms et al., 2019).
However, computational thinking education focuses on the development and training of a mindset, it has different requirements and goals than previous teaching. In practice, programming problem-solving activities often do not contribute to the development of computational thinking. The reason for this is that learners tend to focus too much on syntactic knowledge during programming problem-solving, which leads them to focus more on the syntax and mechanics of their programs at the expense of logic and structure. In addition, learners often learn to program by using programming software to check whether the code they write has syntax errors and whether the final answer is correct. As a result, it is difficult for their logical thinking to be exercised and even more difficult to conclude problem-solving strategies. To address these problems, many studies have begun to improve programming instruction in terms of classroom teaching models (Ren et al., 2021), student learning styles (Martín-Ramos et al., 2017) and programming tools (Wu and Su, 2021).
Nevertheless, most of these studies focus on primary and secondary schools and require deep involvement of teachers and peers in the learning process. For college students, especially noncomputing majors, their opportunities to learn computer programming in a formal educational setting are often limited (Jiang and Huang, 2019). The desire to improve their computational thinking through programming learning often requires them to engage in independent learning. While this is a laudable goal, it is difficult to achieve. This is because technology often reinforces social shafts under the guise of openness (Bruillard, 2020). Especially for programming, which is a relatively abstract knowledge in itself, beginners tend to show extremely high cognitive load when they learn programming independently due to their insufficient knowledge of programming syntax and lack of a systematic knowledge framework. Past surveys have also shown that most introductory programming students perceive computer programming as a technical activity rather than a sequence of cognitive skills (Porter et al., 2013). Students spend almost all of their cognitive resources in learning programming on how to express their solutions through a new discipline of syntax and grammar, which causes them to carry out a computer project, often skipping key steps in developing computational thinking such as analysis and design (Kazimoglu, 2020). At the same time, they will also find the mechanics of learning computer programming often neither interesting nor relevant, which in turn leads to low motivation and high dropout rates (Beaubouef and Mason, 2005).
Informative tutoring feedback (ITF), a common support strategy in problem-based online learning, offers valuable insights to guide error identification and successful task completion (Narciss, 2004). Some researchers have even suggested that ITF is one of the most powerful factors influencing learning in online learning environments (Hattie and Gan, 2011). Its common application to the problem-solving process (Taxipulati and Lu, 2021; Duijnhouwer et al., 2012). Especially in computer-based or multimedia-based self-directed learning environments, ITF is of interest to researchers as an important factor in supporting the learning process (Narciss and Huth, 2004).
ITF can be categorized into two types (i.e. simple feedback and detailed feedback). Simple feedback includes the knowledge of result or response (KR), knowledge of the correct response and answer until correct (provide KR and ask to stay on the same item until it is answered correctly) (Narciss and Huth, 2004). While detailed feedback provides more instructional information and often outperform simple feedback in promoting learning (Moreno, 2004). Therefore, many studies have tailored richer feedback content to specific goals (e.g. interest, perseverance and engagement). For example, Yong et al. (2023) developed a board game with rich human–computer interaction as hints to encourage students to reflect on and improve their moves, so as to develop their computational thinking. Marwan et al. (2020) aimed to improve students’ academic performance, perseverance and engagement by designing an adaptive real-time feedback system that provided feedback on, among other things, students’ current task completion progress, the tasks they were doing and the level of completion of the current task; Sun et al. (2019) instead designed a detailed feedback containing both encouraging and warning messages, through the provision of which the emotional and cognitive engagement of learners could be facilitated.
However, it is not true that the more complex an ITF is, the more effective it is at facilitating learning (Van der Kleij et al., 2015). If the feedback provided is too intrusive (e.g. warning feedback) or too complex, it can lead to students spending more time on feedback processing, and their emotions and motivation to learn are disturbed, which in turn increases the complexity or difficulty of the learning content, resulting in higher cognitive load. In addition, overly detailed feedback might reduce deep thinking motivation and encourage superficial knowledge acquisition (Sharma and Harkishan, 2022; Schimmel, 1988). However, learning motivation and cognitive load significantly shape learning outcomes (Su, 2016), and they are also closely related to learners’ willingness to learn consistently (Wu et al., 2022; Whitehill and McDonald, 1993), which is crucial for computational thinking that requires sustained learning to be developed. It can be seen that the level of design of the ITF has a direct impact on the effectiveness of the ITF for the learner.
Based on the above issues, this study combines the correlation between computational thinking and programming problem-solving and their respective characteristics for ITF design and develops a programming learning intelligent tutoring system (ITS) oriented to computational thinking cultivation based on the designed ITF, with a view to realizing the effectiveness of computational thinking cultivation in practical applications. In addition, it is considered that learners’ cognitive load and learning motivation largely affect their willingness to use the system for continuous self-directed learning (Zulu et al., 2018; Park and Lee, 2021). Therefore, in this study, in conducting research on the design, development and application of an ITF-based question-answer design system, its effects on learners’ cognitive load and learning motivation were further explored.
The following questions address the research purposes:
Design and development of informative tutoring feedback-based intelligent tutoring system
Concept model of informative tutoring feedback
Different types of ITF use different learning tasks or learning objectives, and the impact effects of different quality ITF vary (Klueger and DeNisi, 1996). This study combines the characteristics of programming problem-solving activities and the needs of computational thinking development, and refers to the ITS design process proposed by Narciss and Huth to design the ITF from the three aspects of functions of feedback, contents of feedback and presentation of the feedback (Narciss and Huth, 2004).
Functions of feedback are functions related to learning objectives. The ultimate goal of this study is to enable learners of different levels of computational thinking to achieve an improved level of computational thinking, and the most important thing for the development of computational thinking is for learners to gain a complete problem-solving experience, so that they can acquire a systematic process of problem-solving and various strategic approaches to problem-solving. Therefore, feedback should assist students in going through a systematic and complete programming problem-solving process and provide personalized feedback content based on the errors made by the students.
Based on the problem-solving process proposed by Polya and the characteristics of programming problem-solving activities, this study divides programming problem-solving into identifying the problem, extracting knowledge, developing plan and verification review (Polya, 2004). In the extracting knowledge session, students may have difficulties in extracting problem-solving strategies quickly due to insufficient understanding of the topic or confusing knowledge structure; in the developing plan session, students may have difficulties in expressing the conceived strategies in computer language; in the verification review, students may have difficulty determining the feasibility of their solutions and revising them according to the problems, or they may become frustrated or even give up because of mistakes in their individual problem solutions. To address these problems, the main functions of the ITF in the computational thinking development process should be to:
provide process prompts to provide timely assistance to learners who encounter difficulties in the problem-solving process;
perform problem solution diagnosis, which should include knowledge of problem-solving logic and programming, and report the gap between the learner’s current answer and the answer; perform error source analysis; and
provide emotional encouragement to facilitate cognitive processing of feedback, including selection, organization and integration, through emotional interaction between the learner and the computer (Moreno and Mayer, 2007).
Contents of feedback is based on the functions of feedback, and it also needs to combine certain design principles and the knowledge and skills involved in the learning task, and then determine the information related to ITF in the learning process and the contents of feedback that should be given based on this information (Narciss and Huth, 2004).
From the process prompting function, the information of ITF should be sourced with the setting of the topic, and according to the characteristics of the proposed project, students should be provided with hints about the analysis of the topic information and knowledge points or strategies related to the solution of the topic.
From the diagnostic function, the information of ITF should come from the errors in students’ answers. This form of feedback is called bug-related tutoring feedback (BRT-feedback). This type of feedback allows the learner to make multiple attempts at problem-solving by providing strategic and useful error correction information, without providing a complete solution to the problem, and requires the learner to continually correct any deficiencies in their original answer. With the support of such feedback, learners are able to go through the complete problem-solving process and get the final answer after continuous trial and error, which leads to better motivation (Narciss and Huth, 2006).
From the function of emotional support, ITF should then infer the emotional state of students through their learning behavior and give them some emotional feedback when they experience negative emotions. Such a feedback component is known as reactive empathy in previous studies, which focuses on identifying students’ emotions and providing information that can regulate them (Terzis et al., 2012).
Presentation of the feedback refers to the way and form in which the feedback is provided. In the process of extracting knowledge and developing plan, the feedback is acting on the students’ confusion resolution in the process of problem-solving. Therefore, feedback should be designed to avoid providing all topic-related prompts without filtering. It is important to start from the learner’s confusion and provide different feedback contents according to the learner’s level of confusion. This step-by-step feedback approach can effectively prevent learners from receiving too much information at once, which may result in an excessive cognitive load. (Mayer and Moreno, 2002). At the same time, such prompts throughout the student’s formal problem-solving process also act as a companion to a certain extent and can reduce the isolation of independent learning students.
In the process of validation review, for BRT-feedback, the feedback information comes from the learner’s mistakes, on the one hand, it is necessary to avoid presenting the correct answer directly, which leads to the students failing to get the complete experience of problem-solving. On the other hand, it is necessary to consider avoiding the learner’s psychology of giving up due to confusion. Therefore, the feedback in this session should provide more revealing hints for program improvement, for which a combination of graphic and textual feedback can be used, i.e. graphic information as a supplement to the textual information, to help learners revise their understanding of what they have read (Kulhavy and Stock, 1989), and, thus, better refine their programming problem-solving strategies.
Similarly, in this session of emotional feedback, the procedural form of emotional feedback is often carried out in the form of a combination of pictures and texts to better convey the situation, and these pictures and texts often have a more obvious encouraging or warning meaning, thus, enabling students to be more actively involved in the improvement of their problem solutions.
System development and algorithm design
The studied ITS is developed through PyCharm, scratch’s API interface and MySQL database. MySQL is a relational database that supports flexible data models and diverse data structures, as well as efficient query and indexing mechanisms. Therefore, it is used to store information about topics and feedback.
Scratch is a visual programming software. It divides the programming statements into actionable blocks, and the learner only needs to drag and drop the blocks and modify some key parameter values to complete the programming process. This allows the student to avoid thinking too much about programming syntax because the blocks are described in a common language and allow the student to dynamically modify the parameters during the writing process, depending on the problem context, without having to write an algorithm. This makes it extremely easy to understand and master (Montiel and Gomez-Zermeño, 2021). Students can, therefore, focus their attention to the problem-solving process (Wu and Su, 2021; Zhan et al., 2022), which can reduce the cognitive load caused by grammar use as well as increase student motivation (Liu et al., 2022). As they go through the programming process, they need to concentrate on the step-by-step process of block building, as the drag-and-drop system does not allow blocks to be connected in a way that has no effect on the programming logic (Montiel and Gomez-Zermeño, 2021). This facilitates learners’ mastery of problem-solving strategies and the logic of the thinking process. In addition, scratch supports its ability to translate the programming blocks built by the user into Python in real time, which facilitates the user to be able to move further from graphical programming to textual programming in the programming process, and also facilitates code analysis and, thus, feedback. Moreover, compared to other visual programming software (e.g. App Inventor, Hopscotch, Alice, etc.), scratch can use extensions to support hardware programming, which facilitates learners to turn their designed problem solutions into reality.
With these supports, this study developed an ITS based on the ITF and divided four modules based on the needs of the four segments of programming problem-solving: the question bank module, the prompting module, the strategy detection module and the emotional feedback module (see Figure 1).
Module functions of intelligent tutoring system
The question bank module corresponds mainly to the identifying the problem session, allowing learners to choose questions based on their programming level and desired practice. Drawing 10 questions from the database based on this selection, it presents them in ascending difficulty. Problems are rooted in real-life scenarios, which is conducive to learners’ realization of the close connection between computational thinking and life, thus, preserving good motivation for learning. At the same time, this also enables the subsequent evaluation to fairly and objectively test the students’ real abilities in a more natural situation, avoiding the increase of nonessential cognitive loads on the students (LI et al., 2022).
To achieve these goals, the study uses knowledge tags, programming level tags and user-selected features for personalized question supply. The ITS organizes topic data in a MySQL database (see Figure 2), including tables for KnowledgePoints, Questions, Prompts, Answers and ErrorCases. The structure links knowledge, questions, hints, correct answers and error cases efficiently.
ITS provides learners with multiple knowledge point tags to choose from based on the knowledge point attributes in the database, and also provides a programming level selection button, which divides the programming level into three levels: high, medium and low, so that learners can choose accordingly based on their programming level. Based on the knowledge point label selected by the learner, ITS queries the KnowledgePoints table in the database to obtain the topic ID corresponding to the knowledge point and obtains the information of the topic related to it from the Questions table based on the topic ID. Then, based on the programming level label selected by the learner, ITS calculates the difficulty coefficient threshold of the questions by using the sort_values function and the NumPy library to filter the questions suitable for the learner from the set of candidate questions.
The hint module acts as a knowledge extraction link and a planning environment, in which the main task of the learner is to select an appropriate problem-solving strategy based on the problem, or relevant programming knowledge, and to develop a complete problem solution, which is presented in the form of graphical programming. However, many self-directed learners lack systematic learning opportunities or have not formed a mature knowledge framework for knowledge transfer, and, thus, they are often unable to extract relevant knowledge points quickly. For these learners, the ITF uses the Hints module to provide learners with a “bag of tricks” of problem explanations, strategy choices, knowledge applications or similar problem-solving strategies, depending on the characteristics of each topic. This enables learners to apply their existing knowledge in a better way. According to the design of the ITF, the content of these hints should be provided gradually, i.e. as the number of times the learner actively accesses the “bag of tricks” increases, the tips provided by “bag of tricks” became more detailed (see Figure 3).
The strategy detection module and the emotional feedback module work together in the validation review phase, where learners need to submit completed problem solutions, and the strategy detection module in ITS analyzes learners’ problem-solving strategies, while the emotional module provides corresponding feedback based on the results of the strategy diagnosis analysis and process data.
The strategy detection module realizes result diagnosis and error analysis by comparing and analyzing the problem-solving strategies entered by students and the preset problem-solving strategies, and accordingly presents the feedback information in the form of graphic and text combination (see Figure 4). The result diagnosis involves comparing the student’s code with various preset answers, constructed based on different solution ideas to cover multiple correct solutions. Both the student’s code and each correct answer are abstracted into syntax trees, and then the student’s syntax tree is compared to each standard answer syntax tree.
Based on the results of the similarity comparison the answer with the highest similarity is selected and considered as the student’s solution. If the highest similarity is not 100%, i.e. the student’s code is different from the most similar answer, the policy detection module starts invoking the error cause analysis function, i.e. based on the answer IDs provided in the answer table, it finds multiple error cases in the ErrorCases table corresponding to the correct answer with the highest similarity, as well as the error causes or potential problems to which the error cases point. Then, the student code is matched against the multiple error cases extracted from the ErrorCases table and an error cause analysis is implemented based on the degree of match. Finally, the strategy detection module returns the highest similarity calculated and the identified error causes to the learner.
The policy detection module’s algorithm uses syntax trees derived from the program’s tree-like representation. Each node in the tree represents a structure (e.g. statement, expression and function) in the user’s code, constructed using Python’s abstract syntax trees (AST) module. It analyzes the learner’s input, extracts lexical properties through semantic analysis and transforms each word into a node, forming a syntax tree. After the syntax trees of learner input answers and standard answers are generated, the strategy detection module also needs to evaluate the similarity of the two syntax trees. In this regard, this study compares various similarity calculation methods. Specifically, they include node-based comparison, subtree-based comparison, edit distance-based and node-sequence-based comparison (see Table 2). After combining the advantages and disadvantages of the above-mentioned comparison methods, this study finally chose the node sequence-based comparison, which can consider the structural and semantic similarity of the programs, thus, capturing the structural and semantic features of the programs more accurately and obtaining more accurate similarity scores. Meanwhile, this study obtains several important IF-THEN rules based on the AST comparison model and the characteristics of the longest common subsequence (LCS) algorithm, where X is the AST of the student input answers, Y is the AST of different predefined answers and the length of sequence X is m and the length of sequence Y is n. The study use LCS[i][j] to denote the longest common sub between the first i elements of sequence X and the first j elements of sequence Y length of the sequence. The specific representation is as follows: (1) (2) (3) (4)
The emotional feedback module corresponds to emotional feedback information, which focuses on the learner’s high cognitive load or reduced motivation due to frustration (Lang et al., 2022). This module guides learners by monitoring their performance and responses, offering encouragement or warnings to maintain motivation during repeated program revisions. Key indicators for assessing learning status include prompt acquisition frequency, answer results and response duration. The number of hints is sourced from the hint button in the module, answer results are based on node sequence similarity from the strategy detection module, and answer length is primarily determined using the time function time.time(). The content of the emotional feedback is judged according to these three values in accordance with certain principles. For example, if the prompt is given several times but the similarity remains low, indicating that the student is experiencing difficulty, the feedback will encourage and remind the student to analyze the grammatical structure of the standard answer. If the response is short and the similarity is low, the student may have been careless or misunderstood, and the feedback will prompt the student to think carefully and use the prompt if necessary.
Method
Participants
The participants in this study came from a university in Guangdong Province, China, and the students who participated in this experiment consisted of 15 college students, ranging in age from 18 to 23. These students were all noncomputer majors and had very limited opportunities to learn system programming. The participants had completed and passed the required university computer science course and had some programming knowledge, but had not learned programming systematically during the course.
The intelligent tutoring system
The design of the ITF.
The ITF is designed in terms of both content and presentation. In the design of ITF content, this study focused on the errors that learners may make during the problem-solving process and further helped students to analyze the reasons behind their errors based on the errors that occurred (see Table 1).
In the design of the ITF presentations, in order for the feedback to start from the learner’s confusion, this study provided step-by-step detailed prompting information, that is, the initial prompts provided some basic guidance to help the students start the problem-solving thinking process. As the number of active prompts increased, the prompts became more detailed. In this way, learners were given multiple opportunities to receive prompts during the problem-solving process, and the problem of excessive cognitive load was avoided. Also considering the issue of companionship, the human-computer dialogue format was chosen in this study to provide these cues to maximize the effectiveness of this companion learning experience:
The development of the system.
According to the design of the ITF, the content of ITF should be provided gradually. For this purpose, the system provides a button that students can click to actively access the hints. A buried code is added to the button click event to record each time a student clicks the button to get a hint. If the student clicks the Hint Get button, the Hint module determines the level of hints provided and presents them to the learner based on the topic and the number of times the student has accessed the hints (see Figure 5).
In addition to the process ITF, the diagnostic ITF is also included, as shown in Figure 6. According to the design of the strategy detection module and the affective feedback module, the highest similarity computed and the reason for the error identified will be provided to the learner as the diagnostic ITF, and the corresponding affective feedback determined by combining the diagnostic result, the number of times the learner was prompted and the duration of the answer will be presented in graphical form. In addition, considering that multiple forms of feedback help students’ comprehension, the syntax tree of the student code and the syntax tree of the abstracted answer with the highest degree of similarity are also presented to the learner as graphical information during the resultant diagnostic process (see Figure 7).
Procedure
A single-group experimental design assessed the impact of an ITS on learners with limited programming knowledge. To address participant variations, a problem set focusing on the fundamental structures of a program was chosen. Prior to programming instruction, participants underwent assessments for cognitive load, learning motivation and computational thinking. Subsequently, they viewed a 5-min instructional video on program structures before tackling 10 problem-solving tasks. Questions were sourced from a question bank module and problem-solving occurred through graphical programming and hints modules. The strategy detection module analyzed responses, providing feedback to learners. Correct answers advanced participants to the next question, while incorrect responses prompted strategy adjustments until mastery. Posttask, participants were reevaluated for cognitive load, motivation and computational thinking. In addition, interviews were conducted (see Figure 8).
Measurement
The study focuses on exploring the impact of ITF-based ITS on learners in terms of computational thinking, learning motivation and cognitive load.
Computational thinking
Computational thinking was evaluated using the computational thinking scale developed by Korkmaz et al. (2017), which contains four dimensions of creativity, algorithmic thinking, critical thinking, collaboration and problem-solving (Cronbach a: 0.822). Comprising four dimensions: creativity, algorithmic thinking, critical thinking and collaboration (Cronbach’s α: 0.822). Using a five-point Likert scale, participants rated items from “strongly disagree” to “strongly agree.” SPSS was used to analyze the scale, and the results showed that Cronbach’s alpha was 0.822, indicating that the scale can be trusted.
Learning motivation
Learning motivation is an important indicator of students’ sustained use of the ITS developed in this study for computational thinking development. To assess motivation, the ARCS model by Keller (1983) was chosen, aiding in sustaining motivation during instructional activities. Wu et al. (2022) developed a motivation assessment scale based on this model (Cronbach a. 0.93). The study further analyzed the reliability of the scale and Cronbach’s alpha value was 0.891, which is greater than 0.8, indicating that the scale has good reliability and stability.
Cognitive load
Cognitive load assessment used Hwang’s scale (Hwang et al., 2013), using a five-point Likert scale. Divided into “mental load” (five items) and “mental effort” (three items), Cronbach’s a-values for mental load and mental effort were 0.84 and 0.68, respectively, with overall a-values at 0.87.
Results
Computational thinking
Paired-samples t-test assessed impact of ITS on participants’ computational thinking. The results are shown in Table 3 which reveal that the means of the data of the posttest are higher than the means of the pretest, where the participants’ creativity (t = −4.087, p = 0.001**), algorithmic thinking (t = −3.311, p = 0.005**), critical thinking (t = −2.882, p = 0.011*) and collaborative (t = −3.444, p = 0.004**) all showed significant improvements.
To further investigate the effect of the ITS system on participants with different levels of computational thinking, this study divided the participants into three groups according to their computational thinking scores (high level for computational thinking scores >100, medium level for scores >=100 and >=90 and low level for scores <90) based on their pretest data, and since the data did not violate the regression chi-squared assumption (F = 1.803, p > 0.05), the ANCOVA method was chosen to exclude the effect of different levels of computational thinking on the effect of ITS in this study (see Table 4). The results showed that there was no significant difference in the effect of different levels of computational thinking on the level of computational thinking enhancement in the programming problem-solving activity with the application of ITS (p > 0.05). However, as seen by the mean values, the enhancement effect of participants in the low-level group is better than the other two groups, which is possibly due to the insignificant p-value caused by the small sample data.
In addition, this study grouped the students based on the recorded behavioral data from the ITS. The data showed that the number of times students actively obtained hints during problem-solving ranged from 0 to 17, and mainly clustered in the interval of 5–9. Therefore, this study classified students who obtained hints >7 as a group and <=7 as a second group. And an analysis of covariance was used to include the students’ pretest levels of computational thinking as covariates. The results shows that the level of computational thinking in Group 1 has significantly improved compared to Group 2 (F = 4.932, p < 0.05).
Learning motivation and cognitive load
In this study, the effects of ITS on learners’ learning motivation and cognitive load were analyzed for different levels of computational thinking using paired-samples t-test and ANCOVA, respectively (F = 0.009, p = 0.991). The results of the study showed that both motivation and cognitive load were enhanced (for motivation: t = 2.476, p = 0.026; for cognitive load: t = 3.953, p = 0.001). ANCOVA results showed that there was no significant difference in motivation and cognitive load based on computational thinking levels after learning with ITS (F = 0.293, p > 0.05).
Student interview results
In this study, each student’s future intention to use a similar ITS was interviewed after the students completed the study, and four students were randomly selected for in-depth interviews, and these students were asked to evaluate the ITS and describe how they felt about the ITS experience. The results of the interviews were as follows:
Computational thinking: students think they are getting faster at problem-solving.
A number of students indicated that they felt that there was a gradual improvement in his speed of problem-solving, especially in the areas of problem-solving method conceptualization and the transformation of the conceptualization into a graphical programming language. This suggests that the students are able to use their creativity more fluently to solve problems and express themselves using algorithmic thinking, which agrees with the results of the questionnaire analysis:
Cognitive load: topics are challenging but won’t burn out students.
Some of the students with programming learning experience indicated that it was the first time they had completed so many programming topics at once, but it was more relaxed than their previous programming experience, and although some of the topics also felt challenging to them, overall it felt more like they were playing a particular kind of pass-and-fail game. This suggests that the cognitive load of the students’ learning process using ITS was reasonable and did not make them feel overloaded:
Learning motivation: students show a strong desire to continue learning programming with ITS.
Students exhibited significant enthusiasm for using ITS for both programming and computational thinking enhancement. When asked about their preference between ITS, traditional instructional videos or book-based learning, they unanimously favored ITS due to its interactive nature, which aided continuous learning. One student stated, “The intelligent tutor’s encouragement during our dialogue motivated me to persist despite challenges”:
Students’ attitudes toward feedback in the verification review phase varied widely.
Regarding verification review feedback, student attitudes diverged. Some found it valuable for rectifying incorrect answers, while others struggled due to unfamiliarity with the grammar tree or difficulty in extracting useful information for program adjustments. One student remarked, “Interpreting the graphical feedback and supplementing textual feedback for effective changes posed a challenge.”
Discussion
This study analyzes the current problems encountered by college students in programming problem-solving; proposes an ITF mechanism for computational thinking development based on the existing ITF design process; incorporates this mechanism into the design and development of ITS, which is integrated into the four key aspects of programming problem-solving for learners; and finally achieves personalized problem supply, immediate process cues, feedback on results based on problem-solving logic and emotional feedback based on learning behavior data analysis.
In the development of computational thinking, ITS achieves a significant increase in creativity, algorithmic thinking, critical thinking and cooperativity. This is predictable because when students engage in a variety of active learning activities, they acquire a broader range of higher-order competencies (Evans, 2013). And the present acting ITS incorporates learning tasks, process prompts and feedback, which are supposed to be the main ways to promote more learning outcomes for self-directed learners (Nicol et al., 2014).
For the improvement of algorithmic thinking, researchers have shown that providing students with suggested feedback is one of the key conditions for fostering creativity (Root-Bernstein, 2015), which is facilitated by the ITF provided by this study’s ITS.
On the dimension of algorithmic thinking, on the one hand, may be the result of the role of visual programming and feedback (Wu and Su, 2021; Brusilovsky et al., 1997). Because algorithmic thinking is a method used to describe problem-solving, and the scratch API embedded in this study’s ITS, this allowed students to focus more on the description of the problem-solving solution rather than the application of programming syntax. The graphical feedback was also able to present the overall structure of the problem solution in a visual form, helping learners to better recognize the problems in the problem-solving solution descriptions.
As for critical thinking and cooperativity, their enhancement is mainly attributed to the personalization of feedback, which is able to quickly uncover the problems present in them based on students’ problem solutions and present them to learners in a variety of formations, changing the traditional programming learning in which both teacher and peer feedback are hardly effective (Asiri et al., 2018). On the one hand, students are able to explore the problems in their solutions and correct them through feedback, a process that effectively promotes critical thinking (Ndolo, 2021). On the other hand, students gain a better learning experience by collaborating with an intelligent tutor and are more willing to collaborate in future learning because of their ability to obtain information from the feedback. As for the problem-solving dimension, it is not consistent with the findings of previous studies. Previous studies have shown that an important factor of feedback for problem-solving improvement is the revision of answers, and although this study provided multiple responses, it did not save the data of the students’ last response and required them to reformulate the complete problem solution, which may lead to new solutions in subsequent responses because they did not remember the solution of the last response and faced new errors.
Regarding ITS for the development of computational thinking of different learners, it differs from previous findings that feedback is difficult to assist low-level computational thinking learners to obtain improvement (Moreno, 2004). This could be due to several reasons. One is that for low-level learners, they usually face difficulties in acquiring information about the problem, or they often need more behavioral paths to extract information about the features of the problem, which leads to inefficient problem-solving (Shen et al., 2022). The process prompts provided by ITS in this study helped these students to solve this problem by instructing them to quickly acquire valid information and in the process to acquire the ability to analyze the problem. The second reason may be due to the personalization of the ITS feedback in this study. Because the feedback provided in this study was based on students’ learning data, allowing diverse learners to extract desired insights. In addition, the ITS-generated ITF differs from traditional feedback; it is accurate, impartial, student-oriented, devoid of ability assessment and does not involve an assessment of student ability (Ndolo, 2021). While learners with lower proficiency displayed notable improvements, limited sample size hindered statistical significance in covariance analysis. This might stem from ITS aiding lower-level learners with increased cues, enhancing motivation and reducing cognitive load (as evidenced in the analysis of cognitive load and motivation across different computational thinking levels). Consequently, this facilitated more pronounced advancements in their computational thinking abilities.
Furthermore, the analysis of the effect of the number of hints on computational thinking showed that learners who obtained more hints were able to improve their computational thinking more significantly, and in the follow-up survey it was also found that most of the low-level learners, who obtained more hints, were able to improve their computational thinking more. This shows that ITS is an effective tool for noncomputer learners to learn independently and develop computational thinking.
In addition to computational thinking, this study also investigated students’ motivation and cognitive load. Results showed that the ITF was able to promote students’ learning motivation, which is consistent with previous studies (Shute, 2008). This is likely since the ITS in this study presented problem topics and process prompts in a human-computer dialogue format, which improved students’ perception of community and influenced students’ satisfaction and motivation (Li et al., 2020). Also, the personalized feedback and immediate prompting features of the ITS have a role in promoting positive learning behaviors among learners (Hsu and Ching, 2015). The reduction in cognitive load is also consistent with the results of previous studies. While cognitive load itself represents the total burden that a task imposes on the learner’s cognitive system, the presence of ITF allows students to identify their presence more quickly and reduces their cognitive resource support (Fyfe et al., 2015). In addition, the ITS in this study supports students to answer multiple times, which to some extent reduces the psychological burden of students due to the fear of making mistakes (Wancham and Tangdhanakanond, 2022).
Implications and future research
This study systematically analyzes the dilemmas faced by today’s college students, especially for noncomputer science majors, in programming learning and computational thinking development, and based on these problems and difficulties, the design of a programming problem-solving-based ITF and the development of a corresponding ITS were conducted. The study shows that the ITS designed in this study can improve students’ computational thinking and keep their cognitive load from being too high and their motivation high.
These findings deepen our understanding of how ITF promotes the development of learners’ thinking and how ITF integrates with computer-assisted learning, providing new ideas for the development of computational thinking. In the future design of ITS or self-directed learning models, ITF can be integrated into the whole learning process of the learner. On the one hand, for the learner, the ITS integrating ITF can act as an intelligent tutor agent to help him/her to monitor the level of personal computational thinking and problems, and to conduct targeted learning. On the other hand, for teachers, ITF-based ITS can reduce the instructor’s pressure. At the same time, ITS enables accurate evaluation of learners’ knowledge acquisition and analyzes the logic and innovation of learners’ work. This enables teachers to understand the learner’s situation comprehensively and provide guidance, truly realizing assessment for learning and teaching. In addition, teachers can further refer to the ITF ideas proposed in this study, and provide feedback according to the specific situation of the students in the process of classroom teaching or project-based learning.
Future research can also start from the multimodality of the acquired data and combine more knowledge tracking or emotion recognition techniques to improve and enrich the content of the feedback, so that the ITF process can be more targeted and the results of the diagnostic feedback can be more accurate and diversified. At the same time, it can also start from the diversity of the feedback form, past research has indicated that feedback that provides learners with more instructional information about their current learning status in real time can better facilitate the regulation of the learning process and subsequently acquire or improve the competencies needed to master the learning task (Narciss, 2013). Therefore, future research can consider further combining knowledge graphs, event graphs and other related technologies to visualize the learners’ mastery of knowledge and the process of using computational thinking, so that learners can know more about their learning situation.
This research was financially supported by the National Natural Science Foundation in China (62277018; 62237001), Ministry of Education in China Project of Humanities and Social Sciences (22YJC880106), the Major Project of Social Science in South China Normal University (ZDPY2208), the Degree and graduate education Reform research project in Guangdong (2023JGXM046).
Figure 1.ITS role in programming problem-solving process
Figure 2.Structure of relational database
Figure 3.Hint module algorithm flow
Figure 4.Strategy detection module algorithm flow
Figure 5.The Hint module
Figure 6.The strategy detection module and the emotional feedback module
Figure 7.Graphical feedback for the strategy detection module
Figure 8.Experimental procedure
Table 1.
The design of contents of ITF
| Design principles | Information related to ITF in learning tasks |
|---|---|
| Error analysis | • Syntax error – the program does not compile properly |
| Analysis of potential sources of errors | • Problem misunderstanding |
| Selection of information that can be provided as feedback when errors occur | • Indicates the location of the error |
| Selection of information that can be provided as feedback when learners encounter difficulties in problem-solving | • Provide a similar problem and demonstrate the correct problem-solving strategy |
Source: Created by authors
Table 2.
Comparison of different similarity calculation methods
| Calculation methods | Advantage | Disadvantage |
|---|---|---|
| Node-based comparison | Capable of capturing small changes between nodes | Ignore the hierarchical relationship between nodes, resulting in insensitivity to changes in the structure of the whole tree |
| Subtree-based comparison | Can compare changes in the structure of the entire tree | Insensitive to small changes between nodes |
| Edit distance based | Can compare the structure of the entire tree and small changes between nodes | High time complexity |
| Node-sequence based | Considers semantic information of the code, can handle cases such as variable renaming and has lower time complexity compared to edit distance-based comparison | Cannot handle code addition, deletion and change operations |
Source: Created by authors
Table 3.
Results of paired-sample t-test for computational thinking
| Dimension | Mean | SD | t-test | p-value |
|---|---|---|---|---|
| Creativity | ||||
| Pretest |
3.55 |
0.49 |
−4.09 | 0.001** |
| Algorithmic thinking | ||||
| Pretest |
3.10 |
0.89 |
−3.31 | 0.005** |
| Critical thinking | ||||
| Pretest |
3.61 |
0.49 |
−2.88 | 0.011** |
| Cooperativity | ||||
| Pretest |
3.52 |
0.70 |
−3.44 | 0.004** |
| Problem-solving | ||||
| Pretest |
2.77 |
0.61 |
−0.43 | 0.673 |
| Computational thinking | ||||
| Pretest |
3.22 |
0.30 |
−13.36 | 0.000*** |
Notes:***p < 0.001; **p < 0.01; *p < 0.05
Source: Created by authors
Table 4.
ANCOVA for the effect of ITS on computational thinking
| Pretest | Posttest | |||
|---|---|---|---|---|
| Group | Mean (SD) | Mean (SD) | F | p-value |
| High | 3.53 (0.19) | 4.01 (0.21) | 2.275 | 0.153 |
| Medium | 3.20 (0.15) | 3.69 (0.15) | ||
| Low | 2.85 (0.11) | 3.73 (0.28) |
Source: Created by authors
© Emerald Publishing Limited.