Content area
This article explores the potential of chatbots to bridge the gap for students transitioning from structured programming to object-oriented programming (OOP). We delve into the advantages and disadvantages of using chatbots like ChatGPT 3.5 and 4.0, Gemini, and Bing to assist with OOP learning. While these tools offer benefits like providing code examples and explanations, limitations exist in accurate problem interpretation and adherence to best practices.
Building upon the previous student-focused analysis, we present the results of a new survey targeting software engineering teachers: ‘Survey on the Use of AI Chatbots in Teaching Software Engineering’. This survey sheds light on how educators integrate chatbots into their curriculum and their perspectives on their effectiveness.
The original student survey revealed positive impressions of chatbots as study aids, with a high percentage utilizing and finding them valuable. The teacher survey will provide further insights into their role in the teaching and learning process, ultimately contributing to the development of more efficient software engineers. Recognizing and addressing the limitations of chatbots remains crucial to maximizing their potential in OOP education. This paper is an extension of work originally presented in Conisoft 2023 [1].
INTRODUCTION
Software engineering is a demanding and multifaceted field that necessitates a broad skill set and extensive knowledge. Students in this discipline must grasp and implement numerous concepts and methodologies, spanning from software design to testing and ongoing maintenance [1]. Traditional software engineering education often relies on lectures, labs, and projects, but this method may not effectively address every student’s needs or provide individualized support [2]. Meanwhile, current teaching of OOP often focuses on explaining theoretical concepts, using integrated development environments (IDEs) in labs, and implementing standard projects. However, this approach may not fully reflect the principles of OOP or the complexities of software development in practice, nor does it adequately prepare students to face real-world challenges [4].
AI chatbots present an innovative way to enhance software engineering education by creating an interactive and personalized learning experience. These tools allow students to learn at their own pace and better understand complex concepts in a more engaging way.
In conventional education, teaching involves two key players: students and teachers. The teacher typically dictates the learning pace and addresses questions with examples and discussions. However, this approach isn’t ideal for every student, leading some to fall behind. While personalized teaching would be ideal, it’s often impractical with current human resources. Technology, particularly AI chatbots, can help bridge this gap by offering tailored support.
Chatbots are software programs designed to simulate conversations with users. The first chatbot, named ELIZA, was developed by Joseph Weizenbaum in 1966 [3]. While commonly used in customer service to answer questions and assist customers, chatbots also have educational potential, aiding students in learning new concepts.
Chatbots offer several benefits when used as educational tools:
• Personalization: Chatbots can tailor their content and responses to the individual needs of each student. This can be achieved through the use of artificial intelligence techniques such as natural language processing (NLP) and machine learning (ML). NLP enables chatbots to understand the meaning of user inputs, while ML allows them to learn from their interactions with users and improve their ability to provide accurate and relevant responses.
• Interactivity: Chatbots can provide students with an interactive learning environment that allows them to explore concepts and try out different solutions. This can be especially helpful for learning abstract or complex concepts, as it allows students to visualize and experiment with different ideas in a more concrete way.
• Availability: Chatbots are available 24/7, meaning students can access them anytime, anywhere. This can be particularly useful for students who have difficulty attending traditional classes or who need additional help outside of school hours.
• Motivation: Chatbots can help motivate students by providing them with an engaging and rewarding learning environment. Chatbots can use gamification techniques, such as offering points or badges for completing tasks, to make learning more fun and appealing.
There has been growing research on the use of chatbots in this area, mainly in teaching programming. Studies have shown that chatbots can be effective in helping students learn software engineering concepts, improve their problem-solving skills, and increase their motivation [5–7]. For example, Jill Watson is a chatbot developed by IBM that helps students learn about web development [8]. Jill Watson can answer questions about web technologies such as HTML, CSS, and JavaScript, and can help students create their own web applications.
AI chatbots have the potential to revolutionize software engineering education. By providing students with a personalized, interactive, and engaging learning environment, chatbots can help them learn more effectively and efficiently.
Although there are already works like those mentioned in the previous paragraph, it is necessary to further research to better understand the implications of using chatbots in education given that we are only at the beginning of this new way of teaching and we still do not have feedback on the effects of this new form of learning. This research should focus on developing chatbots that are adaptable, engaging, and able to provide students with feedback and support they need to succeed. Our previous research [9] explored the pros and cons of using chatbots such as ChatGPT versions 3.5 and 4.0, Gemini, and Bing as assistants in OOP learning. We also provided a concise comparison of their performance in generating code from natural language. In this paper, we summarize the findings of the survey and delve into how educators are integrating chatbots into their curricula and their perspectives on their effectiveness. The paper is organized into six sections. Section 2 provides a concise overview of the current state-of-the-art chatbots. In Section 3, the methodology employed for the surveys is outlined. Section 4 presents the results obtained from the surveys. A discussion of the results is provided in Section 5. Finally, the paper concludes with a summary of the findings in Section 6.
A REVIEW OF STATE-OF-THE-ART CHATBOTS
One of the more popular chatbots at the time of writing this paper was ChatGPT. ChatGPT is a large language model trained by OpenAI with a vast amount of data, the first version available to the general public was the 3.5 [10]. It is designed to understand natural language from written texts and generate responses similar to those of a human. The model uses machine learning algorithms and deep neural networks to analyze patterns in language and generate coherent and relevant responses to user queries. It has been trained on a large corpus of text data, including books, articles, and websites, to develop its language processing capabilities.
As a passive entity, ChatGPT cannot initiate communication on its own. Instead, it responds to user inputs and attempts to provide informative and helpful responses to a wide range of questions. The model can be integrated into various applications, such as chatbots, virtual assistants, and customer service platforms, to provide a more personalized and interactive experience for users. With its advanced language processing capabilities, ChatGPT represents a significant advancement in the field of natural language processing and has the potential to revolutionize the way humans interact with technology.
There are several solutions similar to ChatGPT that are designed to perform natural language processing and generate responses similar to those of a human. Here are some examples:
• GPT-4: This is another language model developed by OpenAI that is similar to ChatGPT but much larger in scale. It has been trained on a massive dataset of text and can perform a wide range of language tasks, including text completion, language translation, and answering questions.
• BERT: This is a language model developed by Google that is designed to perform natural language processing tasks such as sentiment analysis and text classification. It has been pretrained on a large dataset of text and can be fine-tuned for specific language tasks.
• Google’s Gemini [11]: Gemini (before called Bard) is an experimental conversational AI service by Google. It utilizes LaMDA (Language Model for Dialogue Applications) technology [12] and works by extracting information from the web to provide fresh and high-quality responses. Gemini is independent of Google Search but can be integrated with it. It included assistance for code generation in April 2023 [13].
• Microsoft’s Copilot: Microsoft Copilot is an AI chatbot that uses OpenAI’s most advanced LLM, GPT-4 Turbo. It has access to the internet and works like a search engine with information on current events. It was unveiled in February 2023 and moved from a limited preview to an open preview in May.
• Claude [14]: Anthropic, a research and development company specializing in safe general artificial intelligence (AGI), has developed a large language model (LLM) chatbot called Claude. It is capable of performing an impressive variety of tasks, including text summarization, answering questions, generating text, language translation, creative content writing, and even coding. Although Claude is still in development, it has already attracted the attention of several companies and organizations, such as Google, Notion, Quora, and DuckDuckGo, which are utilizing its unique capabilities to enhance their respective businesses. An interesting aspect of this chatbot is that it is governed by a “constitution.” Claude’s Constitution is a set of written moral values that Anthropic used to train and safeguard Claude, its rival to the technology behind OpenAI’s ChatGPT1. The moral value guidelines are based on various sources, including the United Nations Declaration of Human Rights and even Apple Inc.'s data privacy rules [15].
ChatGPT and some other chatbots are based on the Transformer architecture [16]. The Transformer architecture was first proposed in the paper “Attention is all you need” [17]. This architecture was conceived as an alternative to recurrent neural network (RNN) and convolutional neural network (CNN) architectures that were commonly used for natural language processing tasks and had some limitations in terms of efficiency and ability to capture long-term dependencies in text. The Transformer architecture is based on the concept of attention, which allows the network to learn dependency relationships between different parts of the text without the need for a recurrent or convolutional structure. This has proven to be very effective in a variety of natural language processing tasks. In particular, ChatGPT uses a modified version of the GPT (Generative Pretrained Transformer) architecture that was first introduced in 2018 by OpenAI. The GPT architecture utilizes a transformer neural network to generate text in a process called “natural language generation.” This architecture is highly effective in tasks such as text generation, text summarization, machine translation, and question answering, among others. ChatGPT is an upscaled and enhanced version of the GPT architecture that uses more parameters and has been trained with large updated amounts of data to improve its responsiveness and understanding of natural language.
A comparative table of some of the most popular natural language models can be seen in Table 1 [18– 22]. It is worth noting that this table is not exhaustive, and there are many other natural language models available on the market, each with its own strengths and weaknesses. It is important to mention that not all evaluated chatbots were included in the tables because reliable published information on the most recent models was not found at the time of article development.
Table 1. . Comparison of the most popular natural language models
Model | Year | Model Type | Training | Number of Parameters | Supported Tasks |
|---|---|---|---|---|---|
ELIZA | 1966 | Rule-based | Heuristic rules | N/A | Psychotherapist simulation |
BERT | 2018 | Transformer-based | Supervised learning | 110 million | Text classification, question answering, named entity recognition, among others |
GPT-3 | 2020 | Transformer-based | Self-supervised learning | Up to 175 billion | Text generation, machine translation, question answering, dialogue, code generation, among others |
T5 | 2019 | Transformer-based | Supervised learning | 11 billion | Machine translation, text generation, question answering, among others |
RoBERTa | 2019 | Transformer-based | Self-supervised learning | 355 million | Text classification, named entity recognition, among others |
ChatGPT | 2022 | Transformer-based | Self-supervised learning | Based on GPT-3.5 | Text generation, machine translation, question answering, dialogue, code generation, among others |
Claude (Anthropic) | 2023 | Transformer-based | Self-supervised learning | N/A | Text generation, machine translation, question answering, dialogue, code generation |
Bard (Google) | 2023 | Transformer-based | Self-supervised learning | N/A | Text generation, machine translation, question answering, dialogue, code generation |
GPT-4 (OpenAI) | 2023 | Transformer-based | Self-supervised learning | N/A | Text generation, machine translation, question answering, dialogue, code generation |
Gemini (Google) | 2023 | Transformer-based | Self-supervised learning | N/A | Text generation, machine translation, question answering, dialogue, code generation |
On the other hand, ChatGPT 4 represents a significant improvement over ChatGPT 3.5. It has a larger language model, more parameters, and more up-to-date training data. It also has higher top-1 and top-5 accuracy. ChatGPT 4 is also multimodal, meaning it can process different types of inputs such as text, images, and audio. ChatGPT 4 has also been adversarially trained, meaning it has been trained to resist malicious attacks. Finally, ChatGPT 4 has lower latency, meaning it can respond to requests more quickly.
As can be seen, the main chatbots currently are ChatGPT, in its different versions, and Bard. Chatbots are an emerging and constantly developing area, so they are changing rapidly. The results of this generation of chatbots, while more reliable than those of past generations, are still far from perfect. For example, The Guardian newspaper reported that ChatGPT has quoted fake articles attributed by the chat to the newspaper itself [23], [24]. Similarly, USA Today accused ChatGPT of providing false references to alleged research [25]. Google with Bard has been slightly more cautious, but not overly so; according to The New York Times, both Google and Microsoft (with Bing) have sacrificed caution about the quality of results in order to gain a place in the current chatbot race [26]. Recent advances have been so surprising that some industry leaders have called for a pause in the development of systems more advanced than GPT-4 for at least 6 months [27]. The goal is to take that time to agree on and implement auditable security protocols for the development of these artificial intelligence systems.
The advancements in chatbot technology have revolutionized the field of natural language processing. Chatbots’ ability to understand and generate human-like text has opened up new possibilities for their integration into various applications, from customer service to virtual assistants. Its advanced language processing capabilities enable it to handle a wide range of tasks, making it a versatile tool for businesses and individuals alike. However, the strengths of chatbots come also with notable weaknesses so far. Despite the impressive language generation capabilities, these models can still produce incorrect or misleading information. Instances of chatbots fabricating references or misattributing sources highlight the need for caution and verification when using these tools. Moreover, while the ability to generate human-like responses is beneficial, it can also lead to over-reliance on the technology without critical evaluation of the output. The rapid development and deployment of these models, sometimes prioritizing speed over accuracy, further exacerbate these concerns.
METHODOLOGY
Two surveys were conducted, the first one aimed at programming students to assess their experience in using chatbots, and the second one targeted professors with experience in teaching software engineering.
The first survey applied to 51 undergraduate students. The second one was applied to 45 professors. Both surveys were applied between 2023 and 2024.
Survey to Students
The survey titled “Use of chatbots in object-oriented programming teaching” was administered to collect information on how students from various majors use chatbots, such as ChatGPT and Bard, in their programming studies, specifically in object-oriented programming in engineering.
The survey design was validated by a group of professors in the field. It was conducted in May 2023 among three undergraduate student groups taking the object-oriented programming course. These students represent the target population of the survey. The survey was distributed to participants directly by their class instructor, with the assistance of an online learning platform. Before administering the survey to students, they were informed about the nature of the research, the approximate duration of the survey, and the privacy and anonymity measures. The questions focused on five main areas:
1. General usage: Students were asked if they had used chatbots for their studies and, if so, which chatbots they had used and how often.
2. Utility: Students were asked about the usefulness of chatbots for their studies, for which specific tasks they used them, and if there were tasks or topics where they found chatbots to be less useful.
3. Experience: Students were asked about their overall experience with chatbots, the features they appreciated the most, and any issues or drawbacks they encountered.
4. Comparative questions: Students were asked to compare the effectiveness of chatbots with other resources and, if they had used multiple chatbots, which one they preferred.
5. Future usage: Students were asked if they planned to continue using chatbots in the future and if they would recommend them to other students.
Once the data were collected, quantitative responses provided objective data, while qualitative responses offered greater depth and context. The survey results were integrated into this article in an aggregated form, preserving the anonymity of the participants. The results were used to assess the utility of chatbots in object-oriented programming teaching and to inform future teaching strategies. The applied questionnaire can be seen in Table 2.
Table 2. . Questionnaire of the survey “Use of chatbots in object-oriented programming teaching”
General usage: 1. Have you ever used a chatbot (e.g., ChatGPT, Bard) to assist you with your object-oriented programming studies? (Yes, No) 2. Which chatbot(s) have you specifically used for this purpose? (ChatGPT 3.5, ChatGPT 4, Bing Chat, Bard, other (specify), none) 3. How often do you use these chatbots for your programming studies? (Never, rarely, occasionally, frequently, very frequently) |
Utility: 4. On a scale of 1 to 5, how useful do you find chatbots for learning object-oriented programming? (1 being “Not useful at all” and 5 being “Extremely useful”) 5. For which specific tasks in your programming studies do you use chatbots? (Understanding concepts, debugging, learning new languages, other (specify), none) 6. Can you provide an example of a moment when a chatbot significantly helped you in your object-oriented programming studies? (Open-ended response) 7. Are there tasks or topics in object-oriented programming where you feel that chatbots are not useful? (Open-ended response) |
Experience: 8. On a scale of 1 to 5, how would you rate your overall experience with chatbots in the context of object-oriented programming? (1 being “Very poor” and 5 being “Excellent”) 9. What features do you appreciate the most about the chatbot(s) you use for programming studies? (Open-ended response) 10. Have you encountered any issues or drawbacks when using chatbots for object-oriented programming? If so, can you describe them? (Open-ended response) |
Comparative questions: 11. If you have used multiple chatbots for programming studies, which one do you prefer and why? (Open-ended response) 12. Compared to other resources (e.g., textbooks, online tutorials, instructors), how would you rate the effectiveness of chatbots for learning object-oriented programming? (Much less effective, less effective, equally effective, more effective, much more effective) |
Future usage: 13. Do you plan to continue using chatbots to assist you with your programming studies? (Yes, no, not sure) 14. Would you recommend chatbots to other students studying object-oriented programming? (Yes, no, not sure) |
Open comments: 15. Please provide any additional comments or feedback about your experience using chatbots for object-oriented programming studies. (Open-ended response) |
Survey to Software Engineering Professors
The survey “Survey on the use of AI chatbots in teaching software engineering” was administered to university professors of courses related to Software Engineering with the aim of assessing the perception and experience of professors regarding the use of AI chatbots in teaching software engineering.
The survey’s structure is described below:
• Section 1: Previous Experience. This section aims to understand the respondents' prior experience with using AI chatbots in teaching software engineering. It includes questions about whether they have used chatbots, if so, which ones, the courses or stages where they were used, the purposes for their use, outcomes observed, and challenges encountered.
• Section 2: Perception and Perspectives. This section explores the respondents' perceptions and perspectives regarding the future use of AI chatbots in teaching software engineering. It includes questions about the likelihood of future use, potential advantages and challenges, required resources or support for effective use, and any recommendations or suggestions for improvement.
• Section 3: Demographic Information. This section collects demographic information about the respondents, such as age, gender, professional information (type of educational institution, country and region of work), and years of experience in teaching software engineering. This information helps provide context and understand the demographics of the respondents.
Each section serves a specific purpose: Section 1 provides insights into past experiences, Section 2 delves into perceptions and future intentions, and Section 3 gathers demographic information for context. Together, they provide a comprehensive view of the respondents' experiences, perceptions, and backgrounds regarding the use of AI chatbots in teaching software engineering.
The survey design was validated by a group of professors in the field. It was conducted in April 2024 among university teachers mainly from Mexico. After gathering the data, quantitative responses yielded factual information, while qualitative responses provided more detailed insights and background. The findings from the survey were consolidated in this article, maintaining the anonymity of the respondents. These results were then utilized to evaluate the effectiveness of chatbots in teaching object-oriented programming and to shape forthcoming instructional approaches. The specific questionnaire utilized is available in Table 3.
Table 3. . Questionnaire of the survey “Survey on the use of AI chatbots in teaching software engineering”
Section 1: Previous Experience |
1. Have you used AI chatbots in your software engineering classes? Why? |
* Yes (briefly describe the experience) |
* No |
Explain (Yes/No): |
2. If you have used AI chatbots in your software engineering classes, please indicate which one(s): |
3. If you have used chatbots or are considering using them in software engineering classes, in which courses or stage of software engineering have you used them? (Check all that apply) |
* Programming |
* Introduction to Software Engineering |
* Software Requirements Engineering |
* Software Design |
* Software Architecture |
* Model-Driven Software Development |
* Software Testing |
* Software Project Management |
* Software Quality |
* Software Maintenance |
* Other (specify) |
4. What purpose did you use or plan to use AI chatbots for? (Check all that apply) |
* Providing support to students in solving exercises and problems. |
* Offering personalized feedback on students' code. |
* Guiding students in the learning process of specific concepts. |
* Detecting errors and common problems in students' code. |
* Customizing the learning pace for each student. |
* Other (specify). |
5. What were the outcomes of using AI chatbots? (Check all that apply) |
* Improvement in students' academic performance. |
* Increased participation and interaction in class. |
* Reduction in the teacher’s workload. |
* Enhancement in student satisfaction. |
* Other (specify). |
6. What were the main challenges or difficulties you encountered when using AI chatbots? (Check all that apply) |
* Difficulty in configuring or integrating the chatbot into the educational platform. |
* Limitations in the chatbot’s ability to understand students' questions. |
* Lack of accuracy in the responses or feedback provided by the chatbot. |
* Difficulty in adapting the chatbot to different levels of knowledge or experience. |
* Lack of resources or technical support for chatbot use. |
* Other (specify). |
Section 2: Perception and Perspectives |
7. How likely are you to consider using or continue using AI chatbots in your Software Engineering classes in the future? |
* Very likely |
* Likely |
* Unlikely |
* Very unlikely |
8. What would be the main advantages of using AI chatbots in teaching software engineering? (Check all that apply) |
* Personalization of learning. |
* 24/7 support for students. |
* Early detection of errors and difficulties. |
* Freeing up time for teachers to engage in other activities. |
* Fostering autonomy and individual learning pace. |
* Other (specify). |
9. What would be the main challenges or concerns you would take into account when using AI chatbots in teaching software engineering? (Check all that apply) |
* Implementation and maintenance costs. |
* Possible negative impact on teaching quality. |
* Difficulty in assessing true student learning. |
* Data privacy and security issues. |
* Resistance or lack of acceptance from students or colleagues. |
* Technical response errors from chatbots. |
* Other (specify). |
10. What type of resources or support would you need to effectively use AI chatbots in your software engineering classes? (Check all that apply) |
* Training and education on the use of AI chatbots. |
* Guides and documentation on the implementation of AI chatbots. |
* Platforms or educational tools integrated with AI chatbots. |
* Technical support and assistance for the configuration and use of AI chatbots. |
* Research and case studies on the effective use of AI chatbots in education. |
* Training in the use of prompts. |
* Other (specify). |
11. What recommendations or suggestions do you have for improving the use of AI chatbots in teaching software engineering? |
Section 3: Demographic Information |
Personal Information: |
Age: |
[ ] 18-24 years |
[ ] 25-34 years |
[ ] 35-44 years |
[ ] 45-54 years |
[ ] 55 years or older |
Gender: |
[ ] Female |
[ ] Male |
[ ] Non-binary |
[ ] Prefer not to answer |
Professional Information: |
What type of educational institution do you work at? |
[ ] Public university |
[ ] Private university |
[ ] Other (specify): |
In which country do you work? |
[ ] (List of countries) |
In which region or state do you work? |
(If applicable, specify regions or states of the selected country) |
How many years have you been teaching in the field of Software Engineering? |
[ ] 0-2 years |
[ ] 3-5 years |
[ ] 6-10 years |
[ ] 11-15 years |
[ ] More than 15 years |
RESULTS OF THE SURVEYS
Results of the Survey to Programming Students
The survey was administered to three groups, obtaining a total of 51 responses from students, with the results explained below.
General usage. For the question “Have you ever used a chatbot (e.g., ChatGPT, Bard) to help you with your object-oriented programming studies?” 56.9% responded affirmatively, while 43.1% indicated that they have not used chatbots in that context. See Fig. 1.
Fig. 1. [Images not available. See PDF.]
Have you ever used a chatbot (e.g., ChatGPT, Bard) to assist you with your object-oriented programming studies?
In response to the question “Which chatbot(s) have you specifically used for this purpose?” Bard obtained 9.1%, Bing Chat 22.7%, ChatGPT 3.5 54.5%, ChatGPT 4 59.1%, and Claude 4.5%, as shown in Fig. 2.
Fig. 2. [Images not available. See PDF.]
Which chatbot(s) have you specifically used for this purpose?
As seen in Fig. 3, in response to the question “How often do you use these chatbots for your programming studies?” The results were: Never 0%, Rarely 31.8%, Occasionally 50%, Frequently 9.1%, and Very Frequently 9.1%.
Fig. 3. [Images not available. See PDF.]
How often do you use these chatbots for your programming studies?
Utility. In this section of the survey, for the first question: On a scale of 1 to 5, how useful do you find the chatbots for learning object-oriented programming? (1 being “Not useful at all” and 5 being “Extremely useful”). 0% selected 1, 4.8% selected 2, 3 was chosen by 28.6%, the highest percentage (38.1%) selected the value of 4, while 28.6% of the responses corresponded to the value of 5. See Fig. 4.
Fig. 4. [Images not available. See PDF.]
On a scale of 1 to 5, how useful do you find chatbots for learning object-oriented programming? (1 being “Not useful at all” and 5 being “Extremely useful”).
The next question, with the graph shown in Fig. 5, “For what specific tasks in your programming studies do you use chatbots?” 36.4% say they use them to understand concepts; 22.7% say they use them for debugging; meanwhile, 13.6% mention they use chatbots to learn the language; and 27.3% use them to explain class examples.
Fig. 5. [Images not available. See PDF.]
For which specific tasks in your programming studies do you use chatbots?
Experience. For this section, to the question: On a scale of 1 to 5, how would you rate your overall experience with chatbots in the context of object-oriented programming? (1 being “Very poor” and 5 being “Excellent”). We can see in Fig. 6 that 0% answered 1 or “very poor”; 9.1% selected the value of 2; meanwhile, 27.3% chose the value of 3; the highest percentage selected 4, with 45.5%; finally, the value of 5 was chosen by 18.2%. See Fig. 6.
Fig. 6. [Images not available. See PDF.]
On a scale of 1 to 5, how would you rate your overall experience with chatbots in the context of object-oriented programming? (1 being “Very poor” and 5 being “Excellent”).
Comparative Questions. In this section, the first question, shown in Fig. 7, was: If you have used multiple chatbots for programming studies, which one do you prefer? 4.5% mentioned Bard; similarly, 4.5% selected Bing Chat; ChatGPT 3.5 was chosen by 22.7% of respondents; ChatGPT 4 by the majority, 54.5%; 0% use the chatbot Claude; and 13.6% have only used one chatbot.
Fig. 7. [Images not available. See PDF.]
If you have used multiple chatbots for programming studies, which one do you prefer and why?
To the question: In comparison to other resources (e.g., textbooks, online tutorials, teachers), how would you rate the effectiveness of chatbots for learning object-oriented programming? 4.5% answered much less effective; 13.6% selected less effective; meanwhile, 18.2% considered it equally effective; whereas 54.5% determined that chatbots are more effective than other resources; and 9.1% determined that it is much more effective. The results can be seen in Fig. 8.
Fig. 8. [Images not available. See PDF.]
Compared to other resources (e.g., textbooks, online tutorials, instructors), how would you rate the effectiveness of chatbots for learning object-oriented programming?
Results of the Survey to Software Engineering Professors
The data for this study was collected through a survey instrument administered in Spanish. The respondents were all educators in the field of software engineering, with experience levels ranging from novice instructors to those with over twenty years of teaching experience. The sample population reflects a geographically diverse group of educators, with respondents hailing from Mexico, Ecuador, Spain, Peru, Chile, and Colombia.
Previous Experience.Figure 9 represents the responses to a survey question, “Have you used AI chatbots in your software engineering classes?”. The survey collected a total of 45 responses. The chart displays an equal division of 50% for ‘Yes’ and 50% for ‘No’. This indicates that half of the respondents have utilized AI chatbots in their software engineering classes, while the other half have not. This data provides an insightful view into the adoption of AI chatbots in Software Engineering education. While half of the respondents are already utilizing them, there is still a significant number of individuals who have not, suggesting an opportunity for increased awareness and training on the benefits of AI chatbots in Software Engineering education.
Fig. 9. [Images not available. See PDF.]
Have you used AI chatbots in your software engineering classes?
Figure 10 presents intriguing insights into the potential application of software bots across various stages of software engineering. The question posed to the respondents was: “If you have used or not chatbots or are considering using them in software engineering classes, in which courses or stage of software engineering have you used them?”
Fig. 10. [Images not available. See PDF.]
If you have used or not chatbots or are considering using them in Software Engineering classes, in which courses or stage of software engineering have you used them?
The survey found that AI chatbots are most commonly used or considered for use in “programming” courses (52.2%). This is likely because programming is a very hands-on subject, and chatbots can provide students with immediate feedback and assistance. Chatbots are also commonly used or considered for use in “Introduction to Software Engineering” courses (19.6%) and “Software Testing” courses (30.4%). This is likely because these courses cover a wide range of topics, and chatbots can help students learn and review material at their own pace. Other courses or stages where AI chatbots are used or considered for use include: requirements engineering (39.1%), software design (28.3%), software architecture (34.8%), model-driven software development (8.7%), software quality (23.9%), and software maintenance (15.2%).
Figure 11 shows the results of a survey on the use of AI chatbots in teaching software engineering. The survey asked respondents to select all of the purposes for which they have used or plan to use AI chatbots in their teaching. The most common purposes were:
Fig. 11. [Images not available. See PDF.]
What purpose did you use or plan to use AI chatbots for?
• Provide support to students (47.8%)
• Guide students through the learning process (65.2%)
• Offer feedback on student work (30.4%)
Less common purposes included:
• Detect errors and problems in student work (19.6%)
• Personalize the pace of learning (2.2%)
• Answer questions about concepts (2.2%)
• Propose code for students to use (2.2%)
AI Chatbots are being used for a variety of purposes in teaching software engineering. The most common purposes are to provide support to students and guide them through the learning process. This suggests that AI chatbots can be a valuable tool for helping students learn software engineering.
For the question “What were the outcomes of using AI chatbots?” the next results were found, as can be seen summarized in Fig. 12. The survey identified several perceived benefits of using AI chatbots in teaching software engineering. The most frequently cited benefits included:
Fig. 12. [Images not available. See PDF.]
What were the outcomes of using AI chatbots?
• Improved student performance: 15.2% of respondents indicated that they believe chatbots can improve student performance.
• Increased student engagement and interaction: 37% of respondents indicated that chatbots can increase student engagement and interaction.
• Reduced teaching workload: 26.1% of respondents indicated that chatbots can reduce the teaching workload.
• Improved student satisfaction: 32.6% of respondents indicated that chatbots can improve student satisfaction.
Figure 13 presents the challenges faced by respondents when using AI chatbots in teaching software engineering. The most common challenges were:
Fig. 13. [Images not available. See PDF.]
What were the main challenges or difficulties you encountered when using AI chatbots?
• Lack of accuracy in responses (44.2%): This suggests that the chatbots were not always able to provide students with accurate information, which could have frustrated students and made it difficult for them to learn.
• Difficulty adapting the chatbot to the specific needs of the course (30.2%): This suggests that the chatbots were not always able to be tailored to the specific content and learning objectives of a particular software engineering course.
• Limitations in the chatbot’s capabilities (23.3%): This suggests that the chatbots were not always able to handle all of the tasks that respondents wanted them to do, such as providing feedback on code or answering complex questions.
• Difficulty configuring or integrating the chatbot (11.6%): This suggests that respondents had some difficulty getting the chatbots up and running, which may have discouraged some from using them altogether.
• Lack of resources or technical support (20.9%): This suggests that respondents may have needed more help from developers or other experts in order to use the chatbots effectively.
Perception and perspectives.Figure 14 shows the results of a survey on the use of AI chatbots in teaching Software Engineering in the future.
Fig. 14. [Images not available. See PDF.]
How likely are you to consider using or continue using AI chatbots in your software engineering classes in the future?
• 52.2% of respondents said they were very likely to use or continue using AI chatbots in their Software Engineering classes.
• 30.4% of respondents said they were somewhat likely to use or continue using AI chatbots in their Software Engineering classes.
• Only 8.7% of respondents said they were somewhat unlikely to use or continue using AI chatbots in their Software Engineering classes.
• Also, 8.7% of respondents said they were very unlikely to use or continue using AI chatbots in their Software Engineering classes.
Overall, the results of the survey suggest that a majority of students are very open to using AI chatbots in their software engineering classes. This suggests that there is a potential for AI chatbots to be a valuable tool for teaching software engineering.
Figure 15 shows the results of the question “What would be the main advantages of using AI chatbots in teaching software engineering?” with the following relevant results:
Fig. 15. [Images not available. See PDF.]
What would be the main advantages of using AI chatbots in teaching software engineering?
• Promoting autonomy and self-paced learning. The majority of the respondents (63%) believe that AI chatbots can promote autonomy and self-paced learning for software engineering students. This is because chatbots can allow students to learn at their own pace and in their own way.
• Personalization of learning. 32.6% of the respondents believe that AI chatbots can be used to personalize the learning experience for software engineering students. This is because chatbots can be tailored to the individual needs of each student, providing them with the level of support and guidance that they require.
• 24/7 support for students. More than half of the respondents (56%) believe that AI chatbots can provide 24/7 support for software engineering students. This is a significant advantage, as it means that students can get help with their assignments and coursework at any time of day or night.
• Early detection of errors. A third of respondents (32.6%) believe that AI chatbots can be used to detect errors in students' code early on. This can help students to identify and fix problems before they become major issues.
• Freeing up time for professors. A quarter of respondents (26.1%) believe that AI chatbots can free up time for professors, allowing them to focus on more complex tasks such as research and teaching development.
Figure 16 shows the results of the question “What would be the main challenges or concerns you would take into account when using AI chatbots in teaching software engineering?” The question results indicate that there are a number of potential challenges and concerns associated with the use of AI chatbots in teaching software engineering. The most common challenges and concerns are:
Fig. 16. [Images not available. See PDF.]
What would be the main challenges or concerns you would take into account when using AI chatbots in teaching software engineering?
• Difficulty evaluating the true learning of students: 56.5% of respondents indicated that they are concerned about the difficulty of evaluating the true learning of students when using AI chatbots. This is because chatbots can provide students with immediate feedback, which may not be an accurate reflection of their understanding of the material.
• Technical errors in responses: 56.5% of respondents indicated that they are concerned about technical errors in responses from chatbots. This could include errors in the information provided by the chatbot or errors in the way the chatbot interacts with students.
• Possible negative impact on teaching quality: 37% of respondents indicated that they are concerned about the potential negative impact of AI chatbots on the quality of education. This is because chatbots may not be able to provide the same level of personalized instruction as human teachers.
• Cost of implementation and maintenance: 30.4% of respondents indicated that they are concerned about the cost of implementing and maintaining AI chatbots. This includes the cost of the software, the cost of training staff, and the cost of ongoing maintenance.
• Privacy and security concerns: 26.1% of respondents indicated that they are concerned about privacy and security concerns associated with the use of AI chatbots. This is because chatbots may collect personal data from students, and this data could be vulnerable to security breaches.
• Resistance or lack of acceptance from students or faculty: 17.4% of respondents indicated that they are concerned about resistance or lack of acceptance from students or faculty. This is because some students and faculty may be skeptical of the use of AI chatbots in education.
Figure 17 shows the results to the question of “What type of resources or support would you need to effectively use AI chatbots in your Software Engineering classes?” It shows the results on the resources that would be needed for using AI chatbots effectively in teaching Software Engineering. Educators were asked to select all of the resources that they would need from a list of seven options.
Fig. 17. [Images not available. See PDF.]
What type of resources or support would you need to effectively use AI chatbots in your software engineering classes?
The most important resource, according to the respondents, was Platforms or educational tools (71.7%). This suggests that instructors need access to tools that are specifically designed for using AI chatbots in the classroom. Other important resources included Training and workshops on AI chatbots (56.5%), Research and case studies on using AI chatbots in education (52.2%), Training on the use of prompts (41.3%), Guides and documentation on AI chatbots (37%) and Technical support and assistance for AI chatbots (32.6%).
The findings of this question suggest that there is a need for a variety of resources to support the effective use of AI chatbots in Software Engineering education. Educators need access to tools, research, training, and support to use AI chatbots effectively in their classrooms.
Demographic information. The demographic data gathered from participants in the “Survey on the use of AI chatbots in software engineering education” reveals a diverse profile within the field. The age range of respondents spans from 25 to 55 years and older, showcasing a broad spectrum of experience levels and perspectives with most having between 16 and 20 years of experience teaching. While the majority of participants identified as male, there was also notable representation from female educators, albeit to a lesser extent.
In terms of the educational landscape, the survey indicates that a significant portion of participants are affiliated with public universities, indicating a prevalence of AI chatbot integration within publicly funded educational institutions. However, there are also educators from private universities, suggesting a broader adoption across different types of academic settings.
Geographically, the survey reflects a concentration of respondents from Mexico, with contributions from other countries such as Ecuador, Colombia, Spain, Peru, and Chile. Within these countries, participants are dispersed across various states, regions, and cities, highlighting the widespread interest and engagement with AI chatbots in software engineering education.
Finally, the data concerning years of teaching experience underscores the diverse expertise within the respondent pool. While some educators are relatively new to the field, with teaching experience ranging from 0 to 5 years, others boast extensive tenure, with over two decades of experience shaping their perspectives and practices in software engineering education. This rich diversity of backgrounds and experiences among participants provides a multifaceted view of the integration and impact of AI chatbots in the teaching of software engineering.
DISCUSSION
Chatbots can be used to help students learn object-oriented programming concepts by providing them with step-by-step instructions on how to solve problems. The integration of chatbots in programming learning is transforming how students learn, providing new opportunities, and also presenting unique challenges. In this section, we will discuss how chatbots are changing programming teaching, as well as the advantages and disadvantages associated with their use.
Overall, chatbots are changing the way students learn programming by providing instant, personalized, and on-demand feedback. Students can interact with chatbots to solve programming problems, which can increase their confidence and autonomy in learning. Additionally, as observed in this study, chatbots can generate code from a problem description, which can help students visualize and understand how to translate problems into functional code.
However, while chatbots offer a range of advantages, there are also challenges and limitations to consider. One such disadvantage is the excessive dependence of students on chatbots to solve programming problems. This can hinder their ability and independence to develop problem-solving skills, as they may become too reliant on automated assistance.
Furthermore, although chatbots can generate code, they do not always interpret problems correctly or follow best programming practices, as demonstrated in this study. This can lead to errors or misunderstandings and may limit their effectiveness as teaching tools. These issues could be the reason why some teachers do not wish to or think about using these tools.
On the other hand, chatbots can present a new way of learning programming, as they provide a safe environment where students can experiment and learn through practice. By interacting with chatbots, students can gain a deeper understanding of programming concepts and improve their programming skills. We believe that the use of chatbots could be integrated to enhance not only learning but also the effectiveness of programmers in their future careers, as tools of this kind are being shown and since tools derived from such technology are rapidly improving [28].
Survey to Programming Students
Regarding the survey results, they provide a very interesting insight into how chatbots are being integrated, independently of teachers, in object-oriented programming learning. Overall, we can observe a positive reception of chatbots as study tools, with more than half of the respondents (56.9%) having used a chatbot to support their studies. ChatGPT 4, in particular, seems to be the most popular, with 59.1% of respondents having used it specifically for this purpose.
In terms of usage frequency, most students use chatbots occasionally (50%) for their programming studies. This may suggest that while chatbots are recognized as useful tools, they have not yet become the primary resource for programming study among students in the course where the experiment was applied.
Regarding perceived utility, the results are interesting. The majority of respondents find the use of chatbots to learn the paradigm useful. 38.1% of participants rated them a 4 on a scale of 1 to 5. Additionally, chatbots are used for a variety of tasks, such as understanding concepts, debugging code, learning the programming language, and explaining class examples.
Lastly, in relation to overall experience, most students have a favorable opinion of chatbots in the context of object-oriented programming. Moreover, many of them prefer ChatGPT 4 among the chatbots they have used. Furthermore, more than half of the respondents consider chatbots to be more effective for learning compared to other resources.
Chatbots are changing the process of teaching and learning programming, providing new opportunities for interactive and personalized learning. Chatbots can also be an additional tool that, when incorporated into teaching, improves the effectiveness of future professional programmers. However, it is also crucial to recognize and address the limitations and challenges associated with their use to maximize their potential as teaching tools. It is important to consider that the way students interact with chatbots is still being carried out empirically and without considering how they could generate better results through the proper use of prompts. At the time of conducting this study, the use of chatbots is still experimental, and the optimal use of these tools is not yet taught. This study is precisely an exploratory analysis of the potential of formally incorporating such tools into students' curricula. It is also necessary to remember that much of the current success of bots is due to interaction through natural language, without yet considering requirements in detailed and syntactic specifications of the required programming language. Defining how accurate a student should be when consulting a bot was not considered in this study since such a trained use would imply mastery, which is not yet the case for the evaluated students or many of us. As artificial intelligence and chatbot technology continue to evolve, we expect to see more natural and straightforward interactions as well as even more exciting developments in this field. Another aspect to consider in future research is the inclusion of programming assistants and their effect on students’ programming learning.
Survey to Software Engineering Professors
The survey results provide valuable insights into the current state of AI chatbot adoption in software engineering education and potential future directions. The findings reveal an even split between educators who have utilized chatbots in their courses and those who have not, highlighting opportunities for increased awareness and training.
The data indicates that chatbots are most commonly employed in programming courses, aligning with the hands-on nature of the subject and the potential for immediate feedback. Additionally, their use spans various stages of the software engineering lifecycle, from requirements engineering to software testing, underscoring their versatility as educational tools.
A key finding is the diverse range of purposes for which chatbots are being utilized, with the primary aims being to provide student support, guide the learning process, and offer feedback on work. This suggests that chatbots are perceived as valuable assistants in facilitating student learning and engagement. However, less common applications, such as personalized pacing or error detection, indicate untapped potential for AI in education.
The survey revealed several perceived benefits of chatbot use, including improved student performance, increased engagement and interaction, reduced teaching workload, and enhanced student satisfaction. These outcomes align with the potential of AI to augment and enhance the learning experience, while simultaneously alleviating instructors’ workloads.
Despite the potential benefits, respondents highlighted challenges such as lack of accuracy in responses, difficulties in adaptation and configuration, and limitations in capabilities. These concerns underscore the need for continued development and refinement of chatbot technologies to ensure their effectiveness in educational settings.
Looking ahead, the survey data indicates a strong inclination towards continued or future use of chatbots in software engineering education, with over 80% of respondents expressing a likelihood of adoption. This positive outlook reflects the perceived value of AI in enhancing the learning experience and aligning with the technological advancements in the field of software engineering itself.
Respondents identified key advantages of chatbot use, including promoting autonomy and self-paced learning, personalization, 24/7 support, early error detection, and freeing up time for instructors. These advantages speak to the potential of AI to revolutionize education by providing tailored, on-demand assistance and enabling instructors to focus on higher-level tasks.
However, the survey also highlighted potential challenges and concerns, such as difficulties in evaluating true learning, technical errors in responses, potential negative impacts on teaching quality, cost considerations, privacy and security concerns, and resistance or lack of acceptance from stakeholders. These findings underscore the need for careful implementation, robust security measures, and effective change management to ensure successful integration of chatbots into educational environments.
To support effective chatbot adoption, respondents emphasized the importance of platforms and educational tools specifically designed for AI integration, as well as training, research, guides, technical support, and assistance with prompt engineering. This holistic approach to resource provision is crucial for equipping educators with the knowledge, skills, and tools necessary to leverage the full potential of AI in their classrooms.
CONCLUSIONS
The integration of chatbots in programming education represents a transformative shift in the way students learn and instructors teach. Our discussions on the surveys conducted with both students and software engineering professors underscore the growing recognition of chatbots as valuable tools in enhancing learning outcomes and facilitating the teaching process. From the student perspective, there is a clear acknowledgment of the utility of chatbots in supporting programming studies, with a majority of respondents expressing positive experiences and perceptions of effectiveness. Despite some limitations and challenges, such as over-dependence and occasional inaccuracies in responses, chatbots are viewed favorably as aids for understanding concepts, debugging code, and reinforcing learning. Similarly, among software engineering professors, there is a notable interest in the adoption and continued use of chatbots in educational settings. The perceived benefits, ranging from improved student performance to reduced teaching workload, underscore the potential of AI to augment traditional teaching methods and provide personalized, on-demand assistance to students. However, it is essential to address the identified challenges and concerns to ensure the successful integration of chatbots into programming education. This includes enhancing the accuracy and adaptability of chatbot responses, providing robust technical support and training for educators, and implementing effective change management strategies.
As AI and natural language processing advance, future work should explore advanced prompt engineering techniques and fine-tuning chatbot models specifically for educational use, enabling more accurate and tailored interactions. Investigating adaptive, personalized learning experiences driven by AI, where chatbots dynamically adjust content and feedback based on individual student needs, could enhance learning outcomes. Developing specialized tools, chatbots or AI tutors for specific programming languages, paradigms, or domains could provide targeted support. Research into the long-term impacts on student learning, problem-solving, and knowledge retention will inform best practices for effective curricular integration. The synergy between human instructors and AI-powered educational tools holds promise for revolutionizing how programming is taught and learned.
FUNDING
This work was supported by ongoing institutional funding. No additional grants to carry out or direct this particular research were obtained.
CONFLICT OF INTEREST
The authors of this work declare that they have no conflicts of interest.
Publisher’s Note.
Pleiades Publishing remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
AI tools may have been used in the translation or editing of this article.
REFERENCES
1 Díaz-Herrera, J.L. and Hilburn, T.B., Software Engineering 2004, Curriculum Guidelines for Undergraduate Degree Programs in Software Engineering, IEEE Computer Society, Association for Computing Machinery, 2004.
2 Vives, L.; Melendez, K.; Dávila, A. ISO/IEC 29110 and software engineering education: A systematic mapping study. Program. Comput. Software; 2022; 48, pp. 745-755. [DOI: https://dx.doi.org/10.1134/S0361768822080229]
3 Weizenbaum, J. ELIZA–A computer program for the study of natural language communication between man and machine. Commun. ACM; 1966; 9, pp. 36-45.
4 Semenov, V.A.; Dragalov, K.V.; Ilyin, D.V.; Morozov, S.V.; Sidyaka, O.V. On complementary principles of object-oriented constraint programming. Program. Comput. Software; 2010; 36, pp. 264-275.
5 Abdellatif, A.; Badran, K.; Costa, D.E.; Shihab, E. A comparison of natural language understanding platforms for chatbots in software engineering. IEEE Trans. Software Eng.; 2021; 48, pp. 3087-3102. [DOI: https://dx.doi.org/10.1109/TSE.2021.3078384]
6 Daun, M. and Brings, J., How ChatGPT will change software engineering education, in Proc. Conf. on Innovation and Technology in Computer Science Education, Turku: ACM, 2023, vol. 1, pp. 110–116. https://doi.org/10.1145/3587102.35888152023
7 Paschoal, L.N., de Oliveira, M.M., and Chicon, P.M.M., A chatterbot sensitive to student’s context to help on software engineering education, Proc. 44th IEEE Latin American Computer Conf.(CLEI), Sao Paulo, 2018, pp. 839–848. https://ieeexplore.ieee.org/abstract/document/8786291/. Accessed Apr. 9, 2024.
8 Rajeshwari, M.; Krishna Prasad, K. IBM watson industry cognitive education methods. Int. J. Case Stud. Bus. IT Educ.; 2020; 4, pp. 38-50.
9 Fernández- Fernández, C.A., Sánchez-Soto, E., and Cisneros, J.R.A., De textbooks a textbots: Una nueva fronteraen la ensenanza de la programacionorientada a objetos, Abstr. Appl., 2024, vol. 2024, no. 45.
10 Hines, K., History of ChatGPT: A timeline of the meteoric rise of generative AI chatbots, Search Engine J. https://www.searchenginejournal.com/history-of-chatgpt-timeline/488370/. Accessed April 10, 2024.
11 An important next step on our AI journey. https://blog.google/technology/ai/bard-google-ai-search-updates/. Accessed June 5, 2023.
12 LaMDA: our breakthrough conversation technology. https://blog.google/technology/ai/lamda/. Accessed March 24, 2023.
13 Code and debug with Bard. https://blog.google/technology/ai/code-with-bard/. Accessed April 21, 2023.
14 Introducing Claude, Anthropic. https://www.anthropic.com/index/introducing-claude. Accessed May 10, 2023.
15 Claude’s Constitution, Anthropic. https://www.anthropic.com/index/claudes-constitution. Accessed May 10, 2023.
16 Transformer: A novel neural network architecture for language understanding. https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html. Accessed March 24, 2023.
17 Vaswani, A., et al., Attention is all you need, Proc. 31st Conf. on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, 2017.
18 Weizenbaum, J. Eliza–A computer program for the study of natural language communication between man and machine. Commun. ACM; 1983; 26, pp. 23-28. [DOI: https://dx.doi.org/10.1145/357980.357991]
19 Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K., Bert: pre-training of deep bidirectional transformers for language understanding, 2018. arXiv181004805.
20 Brown, T. Language models are few-shot learners. Adv. Neural Inf. Process. Syst.; 2020; 33, pp. 1877-1901.
21 Raffel, C. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res.; 2020; 21, pp. 5485-5551.
22 Liu, Y. et al., Roberta: A robustly optimized bertpretraining approach, 2019. arXiv:190711692.
23 Moran, C., ChatGPT is making up fake Guardian articles. Here’s how we’re responding, The Guardian, April 6, 2023. https://www.theguardian.com/commentisfree/2023/apr/06/ai-chatgpt-guardian-technology-risks-fake-article. Accessed April 12, 2023.
24 Newspaper Alarmed When ChatGPT References Article It Never Published, Futurism. https://futurism.com/newspaper-alarmed-chatgpt-references-article-never-published. Accessed April 12, 2023.
25 Williams, M.A., ChatGPT made up research claiming guns aren’t harmful to kids. How far will we let AI go?, USA TODAY. https://www.usatoday.com/story/opinion/2023/03/16/chatgpt-openai-dangers-misinformation-threat-facts/11464869002/. Accessed Apr. 12, 2023.
26 Grant, N. and Weise, K., In A.I. race, Microsoft and Google choose speed over caution, The New York Times, Apr. 07, 2023. https://www.nytimes.com/2023/04/07/technology/ai-chatbots-google-microsoft.html. Accessed Apr. 12, 2023.
27 Pause giant AI experiments: An open letter, Future of Life Institute. https://futureoflife.org/open-letter/pause-giant-ai-experiments/. Accessed March 29, 2023.
28 Mankowitz, D.J., et al., Faster sorting algorithms discovered using deep reinforcement learning, Nature, 2023, vol. 618, no. 7964, p. 7964. https://doi.org/10.1038/s41586-023-06004-9
Copyright Springer Nature B.V. Dec 2024