CodeContrast: A Contrastive Learning Approach for Generating Coherent Programming Exercises

Abstract

Generating high-quality programming exercises with well-aligned problem descriptions, test cases, and code solutions is crucial for computer science education. However, current methods often lack coherence among these components, reducing their educational value. We present CodeContrast, a novel generative model that uses contrastive learning to map programming problems, test cases, and solutions into a shared feature space. By minimizing the distance between matched components and maximizing it for non-matched ones, CodeContrast learns the intricate relationships necessary to generate coherent programming exercises. Our model architecture includes three encoder networks for problem descriptions, test cases, and solutions. During training, CodeContrast processes positive triplets (matching problem, test case, solution) and negative triplets (non-matching combinations) and uses a contrastive loss to position positive triplets close in the feature space while separating negative ones. Comprehensive evaluations of CodeContrast—through automatic metrics, expert ratings, and student studies—demonstrate its effectiveness. Results show high code correctness (92.3% of test cases passed), strong problem–solution alignment (BLEU score up to 0.826), and robust test case coverage (85.7% statement coverage). Expert feedback and student performance further support the pedagogical value of these generated exercises, with students performing comparably to those using manually curated content. CodeContrast advances the automated generation of high-quality programming exercises, capturing relationships among programming components to enhance educational content and improve the learning experience for students and instructors.

1. Introduction

The ability to automatically generate high-quality programming problems, test cases, and solutions is a valuable asset in computer science education. It enables the creation of diverse and challenging exercises, facilitating effective learning for students in introductory programming courses. However, developing a generative model that can coherently map problem descriptions, test cases, and code solutions to a shared representation space remains a significant challenge.

Recent works have explored automated generation of programming exercises, emphasizing coherence between problem descriptions and solutions. For instance, Sarsa et al. (2022) demonstrated the use of large language models for generating programming exercises. Saieva et al. (2023) introduced a code-to-code search technique leveraging both static and dynamic features, utilizing similar and dissimilar examples during training to improve semantic similarity detection. Zhu et al. (2024) proposed a contrastive quantization approach to enhance semantic code representations for generative recommendation tasks. These works highlight the importance of aligning components for educational effectiveness, a focus that this study aims to extend.

Traditional approaches often treat these components separately, failing to capture the intricate relationships between them. This limitation hinders the generation of coherent and semantically aligned programming problems, test cases, and solutions. Additionally, existing methods may struggle to ensure the correctness and pedagogical soundness of the generated content, which is crucial for effective learning in educational settings.

To address these challenges, we propose CodeContrast, a novel generative model that leverages contrastive learning to map programming problems, test cases, and code solutions into a shared feature space. The key idea behind CodeContrast is to learn representations where matching problem-test case-solution triplets are close together in the feature space, while non-matching triplets are pushed apart. This approach aims to capture the semantic relationships between these components, enabling the generation of coherent and aligned programming exercises.

The objective of this work is to develop a model that leverages contrastive learning for generating programming exercises that exhibit strong coherence and alignment among problem descriptions, test cases, and code solutions. By achieving this, the model aims to enhance the pedagogical value and practical usability of generated content in computer science education.

The remainder of this paper is organized as follows: Section 2 presents the state-of-the-art in code generation models and programming problem generation, focusing on machine learning approaches. In Section 3, we introduce the CodeContrast model, detailing its architecture, training procedure, and evaluation methodology. Following that, Section 4 presents the results of our experiments, including both automatic evaluation metrics and human evaluations. We discuss code correctness, problem-solution alignment, test case coverage, diversity, expert ratings, student studies, and qualitative analysis in this section. Finally, Section 5 concludes the paper by summarizing key findings and outlining future research directions in the field of code generation models and AI-driven programming assistance.

2. State-of-the-Art

Generating programming problems, test cases, and solutions has been an active area of research in computer science education and automatic program generation. Several approaches have been proposed to tackle this challenging task, ranging from rule-based systems to machine learning models. In this section, we review the current state-of-the-art in this domain and discuss their strengths and limitations.

2.1. Rule-Based Systems

Early efforts in generating programming exercises relied on rule-based systems and expert-curated templates. These approaches typically involve defining a set of rules or templates for constructing problem statements, generating test cases, and providing solution skeletons.

Rule-based systems have been explored extensively in works such as A. Kumar (2005); A. N. Kumar (2015), where structured templates and rule sets are used to generate problem descriptions and solutions in programming education. While rule-based systems can produce well-structured and pedagogically sound exercises, they heavily rely on manually crafted rules and templates, which can be time-consuming to develop and may lack diversity and flexibility. Additionally, ensuring the correctness and completeness of the generated solutions can be challenging, as the systems often rely on predefined solution skeletons.

2.2. Constraint-Based Generation

Another line of work focuses on constraint-based generation, where the problem and solution spaces are defined by a set of constraints. These constraints can include input-output specifications, resource limitations, or coding style guidelines. The generation process involves finding solutions that satisfy the specified constraints.

Constraint-based generation methods, such as those presented in Brailsford et al. (1999); Martin and Mitrovic (2002); Sovietov (2021), focus on defining detailed constraints for both problem and solution generation, enabling the creation of diverse programming exercises tailored to specific learning outcomes. Constraint-based approaches can produce exercises with well-defined input-output behavior and ensure the correctness of the generated solutions. However, they often struggle with generating natural language problem descriptions and may require extensive manual effort to define the constraints for different problem domains.

2.3. Machine Learning Approaches

With the recent advancements in machine learning, particularly in natural language processing and generative models, researchers have explored data-driven approaches for generating programming exercises. These methods leverage large datasets of existing programming problems, solutions, and test cases to learn patterns and generate new content. One prominent approach is the use of sequence-to-sequence (seq2seq) models, such as recurrent neural networks (RNNs) or transformer-based architectures like GPT Radford et al. (2019). These models can be trained on pairs of problem descriptions and code solutions to learn the mapping between natural language and code. While seq2seq models can generate code solutions from problem descriptions Al-Hossami and Shaikh (2022); Beau and Crabbé (2022); Wang et al. (2023), they often struggle with accurately capturing the relationships between problem descriptions, test cases, and solutions. Additionally, ensuring the correctness and coherence of the generated solutions can be challenging, as these models do not explicitly model the input-output behavior or constraints.

Another line of work focuses on using generative adversarial networks (GANs) and large language models (LLMs) for generating programming exercises. For example, Sarsa et al. (2022) explore using OpenAI Codex to automatically create programming exercises and code explanations. Sun et al. (2022) propose a sequence GAN-based approach for automatic code generation, using LSTM as a generator and CNN as a discriminator, demonstrating improved generation speed and accuracy. More recently, Azaiz et al. (2024) explore using GPT-4 Turbo for generating feedback on programming exercises, showing notable improvements in feedback quality, structure, and consistency compared to earlier models. Jacobs and Jaschke (2024) designed a web application that uses GPT-4 to provide feedback on programming tasks. Wei et al. (2024) introduce SelfCodeAlign, a novel pipeline for self-aligning code LLMs that generates high-quality programming tasks and validated responses without extensive human annotation. Their approach achieves state-of-the-art performance on code generation benchmarks, even with smaller models. While these approaches show significant promise for automated exercise generation and feedback, human oversight remains important for ensuring pedagogical value and content quality.

Prather et al. (2023) delve into the transformative impact of generative AI (GenAI), driven by large language models (LLMs), on computing education. The report anticipates ongoing discussions and advancements in leveraging GenAI for more effective, inclusive, and personalized learning experiences in computing classrooms.

Kotsiantis et al. (2024) delve into the integration of code embeddings and transformers in AI-assisted programming tasks. They highlight how code embeddings capture semantic essence, enabling tasks like code summarization, bug detection, and code completion, while transformers excel in learning contextual representations for tasks such as code generation, translation, and refinement. This comprehensive approach, as outlined in the paper, showcases the potential of combining code embeddings and transformers to enhance efficiency, accuracy, and context-awareness in software development processes.

Soliman et al. (2024) leverage pre-trained language models for code generation, exploring the use of pre-trained transformer language models like BERT, RoBERTa, ELECTRA, and LUKE in code generation tasks. The authors introduce hybrid models combining these pre-trained models with the Marian Causal Language Model, demonstrating enhanced precision and efficiency in code generation. Despite limitations such as dataset size and focus on single-line code generation, the paper identifies future directions like multimodal code generation, explainable AI, and human-AI collaboration, marking a significant advancement in AI-driven software development and code generation productivity.

Sharma et al. (2024) provide a thorough overview of how machine learning (ML) techniques are being used in software engineering tasks related to source code analysis. They cover twelve categories of tasks and discuss the increasing adoption of ML methods in this area. The paper also highlights the challenges faced, such as dataset availability and reproducibility, and emphasizes the growing importance of pre-trained language models like GPTx, BERT, CodeBERT, and others in shaping future software engineering research.

Denny et al. (2024) introduce “Prompt Problems”, a novel type of programming exercise tailored for the generative AI era. It focuses on teaching students how to construct effective prompts for code-generating models, emphasizing the shift towards reading, comprehending, and evaluating code generated by large language models (LLMs). Student feedback highlights enthusiasm for Prompt Problems, their engagement in computational thinking, and exposure to new programming concepts.

Jordan et al. (2024) explore the potential of large language models (LLMs) in generating non-English programming exercises to support non-native English speakers (NNES) in computing education. Using OpenAI GPT-3.5, exercises were generated in English, Tamil, Spanish, and Vietnamese, focusing on sensibility, readability, accuracy, and cultural relevance. While English, Spanish, and Vietnamese exercises showed promise, Tamil exercises exhibited challenges, indicating the limitations in LLMs’ cross-language generalizability. Despite these findings, the study highlights the value of personalized and culturally relevant resources for NNES in their native languages.

Del Carpio Gutierrez et al. (2024) evaluate the effectiveness of automatically generated contextualized programming exercises, aiming to address the need for diverse and engaging problem contexts in introductory programming courses. Leveraging OpenAI’s GPT-4, the research explores different prompting strategies to generate a variety of high-quality programming exercises with contextualized problem descriptions. The evaluation focuses on assessing the novelty and quality of the exercises produced, offering insights into the potential of large language models in automating the creation of diverse programming exercises.

While GAN-based approaches can generate diverse and potentially more realistic programming exercises, they often suffer from mode collapse and instability during training. Additionally, ensuring the correctness and pedagogical value of the generated content can be challenging, as GANs do not explicitly model the relationships between problem descriptions, test cases, and solutions.

Despite these efforts, existing approaches still face several limitations, including:

1. Lack of coherence: Many current methods struggle to generate programming exercises where the problem description, test cases, and code solution are coherent and well aligned.

2. Limited correctness: Ensuring the correctness of the generated code solutions and the validity of the test cases is a significant challenge for many existing approaches.

3. Pedagogical considerations: Existing methods often overlook the pedagogical aspects of generating programming exercises, such as ensuring the exercises are suitable for introductory programming courses and align with learning objectives.

4. Generalization and diversity: Many approaches struggle to generalize to new problem domains or generate diverse and varied programming exercises, limiting their practical applicability.

The CodeContrast model proposed in this work aims to address these limitations by leveraging contrastive learning to map programming problems, test cases, and code solutions into a shared feature space. By learning representations where matching components are close together and non-matching components are far apart, CodeContrast can capture the semantic relationships between these components and enable the generation of coherent and aligned programming exercises. Additionally, the proposed architecture and training procedure incorporate techniques to ensure the correctness and pedagogical value of the generated content.

3. Methodology

The methodology of this study is presented in this section, comprising the detailed description of the CodeContrast model and the processes involved. The section is organized into architecture, training procedures, and evaluation methodology.

3.1. Model Architecture

The CodeContrast model consists of three main components: the problem description encoder, the test case encoder, and the code solution encoder. These encoders are responsible for mapping their respective inputs into a shared feature space, where matching problem-test case-solution triplets are close together, while non-matching triplets are far apart. Figure 1 illustrates the architecture of the CodeContrast model.

The overall workflow (Figure 1) proceeds as follows:

1. Input problem descriptions, test cases, and code solutions are processed by their respective encoders.

2. The resulting embeddings are combined into triplets (positive and negative).

3. Triplets are passed through the feed-forward projection network to produce embeddings in the shared feature space.

4. The NT-Xent loss function optimizes the embeddings by minimizing the distance between positive pairs and maximizing the distance between negatives.

3.1.1. Problem Description Encoder

This encoder takes the text description of a programming problem as input. We employ a transformer-based architecture, specifically BERT Devlin et al. (2019), pretrained on a large corpus of programming language data. The problem text is tokenized, and the resulting sequence is fed into the BERT encoder. The final hidden state corresponding to the [CLS] token is taken as the problem description embedding. BERT was chosen due to its robust contextual embedding capabilities, particularly beneficial for natural language understanding tasks like programming problem descriptions.

The problem description encoder processes the textual description of programming problems:

• Input: The raw text description is first tokenized using a pre-trained tokenizer.

• Encoder: We use a BERT encoder Devlin et al. (2019), which generates contextual embeddings of the tokenized sequence.

• Output: The resulting embedding forms the lProblem Description Embedding, capturing the semantic understanding of the problem.

3.1.2. Test Case Encoder

To encode test cases consisting of sample inputs and expected outputs, we use a dual encoder architecture. The input sequence is processed by a recurrent neural network (RNN) encoder, specifically, a bidirectional long short-term memory (BiLSTM) network Hochreiter and Schmidhuber (1997). Similarly, the expected output sequence is encoded using another BiLSTM encoder. The final hidden states of the input and output encoders are concatenated to obtain the test case embedding. BiLSTM was selected due to its effectiveness in modeling sequential dependencies, which is critical for encoding test case sequences.

The test case encoder is designed to capture the relationship between sample inputs and their corresponding expected outputs:

• Input: Test cases consist of sample input-output pairs.

• Encoders:

  1. The Input Encoder uses a BiLSTM network Hochreiter and Schmidhuber (1997) to process sequential input data.

  2. The Output Encoder is another BiLSTM network that processes expected outputs.

• Concatenation: The hidden states from both encoders are concatenated to form the Test Cases Embedding.

3.1.3. Code Solution Encoder

The code solution encoder maps the code text into the shared feature space. We again leverage a transformer-based architecture, utilizing the RoBERTa model Liu et al. (2019) pretrained on a large corpus of programming languages. The code is tokenized using a language-specific tokenizer, and the token sequence is fed into the RoBERTa encoder. The final hidden state corresponding to the [CLS] token serves as the code solution embedding. The RoBERTa model Liu et al. (2019) is a robustly optimized variant of BERT that uses dynamic masking and larger training corpora, enhancing its ability to capture programming language nuances.

During training, triplets consisting of a problem description, test case, and code solution are sampled from the dataset. Positive triplets are those where the code solution correctly solves the problem and passes the test case, while negative triplets are constructed by randomly sampling non-matching combinations.

The embeddings produced by the three encoders for a triplet are concatenated and passed through a feed-forward projection network to obtain a single representation in the shared feature space. A contrastive loss function, specifically the NT-Xent loss T. Chen et al. (2020), is used to optimize the model parameters.

The code solution encoder maps code solutions into the shared feature space:

• Input: The code solution is first tokenized using a RoBERTa tokenizer.

• Encoder: A pre-trained RoBERTa encoder Liu et al. (2019) generates embeddings that capture the syntactic and semantic structure of the code.

• Output: The resulting Code Solution Embedding represents the input solution.

3.2. Training Procedure

The training objective is to learn representations in the shared feature space where positive triplets (matching problem, test case, and solution) are close together, while negative triplets are pushed apart.

• Positive Triplets: Correct problem-test case-solution combinations.

• Negative Triplets: Incorrect combinations used to enforce contrastive learning.

These triplets are passed through a Feed-Forward Projection Network that projects the embeddings into a Shared Feature Space.

To optimize the embeddings, we employ the NT-Xent loss T. Chen et al. (2020), a contrastive loss function:

(1)${\mathcal{L}}_{\mathrm{NT}-\mathrm{Xent}}=-log{\displaystyle \frac{exp(\mathrm{sim}({\mathbf{z}}_i,{\mathbf{z}}_j)/\tau )}{{\sum }_{k=1}^N{\mathbf{1}}_{[k\ne i]}exp(\mathrm{sim}({\mathbf{z}}_i,{\mathbf{z}}_k)/\tau )}},$

where ${\mathbf{z}}_i$ and ${\mathbf{z}}_j$ are positive embeddings, $\tau$ is the temperature parameter, and sim represents the cosine similarity.

During training, we employ several techniques to enhance the model’s performance:

1. Hard Negative Mining: In addition to randomly sampling negative triplets, we employ hard negative mining Schroff et al. (2015) to prioritize difficult negative examples. This involves selecting the negative triplets with the smallest distance to the positive triplet in the feature space, encouraging the model to better distinguish between similar but non-matching triplets.

2. Data Augmentation: To increase the diversity of the training data and improve the model’s generalization, we apply data augmentation techniques. For problem descriptions, back-translation Edunov et al. (2018) generates paraphrased versions, while for code solutions, variable renaming and statement reordering preserve functionality:

   • Problem Descriptions—Back-Translation involves translating the original problem description into an intermediate language (e.g., English → German) and then translating it back to the original language (e.g., German → English). This process generates a paraphrased version of the text that preserves the original meaning but introduces variations in phrasing and sentence structure. For example, a problem description such as: “Find the sum of all even numbers in a list of integers.” may be paraphrased to: “Calculate the total of all even values within an integer list.” This augmentation improves the model’s robustness to linguistic variability in problem statements, allowing it to better generalize across different phrasings of the same problem.

   • Code Solutions—Semantic-Preserving Transformations: we apply semantic-preserving transformations that generate syntactically different but functionally equivalent programs. These transformations ensure that the program’s behavior and outputs remain unchanged while introducing structural and stylistic variations. The techniques used include “Variable Renaming” (variables in the code are systematically renamed without altering their functionality) and “Statement Reordering” (certain independent statements or blocks of code are reordered in a way that does not affect the program’s functionality).

Curriculum Learning: We employ a curriculum learning strategy Bengio et al. (2009) where the model is first trained on simpler examples and gradually exposed to more complex ones. This helps the model build a robust understanding of programming concepts before tackling more challenging problems.

The model parameters are optimized using the Adam optimizer Kingma and Ba (2014) with a cosine annealing learning rate schedule Loshchilov and Hutter (2016). We also employ techniques such as weight decay, gradient clipping, and mixed precision training to improve convergence and stability.

3.3. Evaluation Methodology

To assess the performance of the CodeContrast model, we employ a comprehensive evaluation methodology, which is structured into two main components: research methodology and software prototype implementation phases.

3.3.1. Research Methodology

Our research methodology involves evaluating the efficacy of CodeContrast through both automatic and human evaluation metrics. These are carefully designed to measure the model’s performance and its impact on pedagogical outcomes. The evaluation steps include:

1. Defining Objectives: Identify the key objectives of CodeContrast, such as generating diverse, correct, and pedagogically useful programming exercises.

2. Establishing Evaluation Metrics: Select appropriate metrics, including automatic measures (e.g., BLEU, BERTScore, code correctness) and human evaluation methods (e.g., expert ratings, student studies).

3. Dataset Preparation: Utilize a dataset of programming problems and solutions, divided into training, validation, and test sets.

4. Data Augmentation: Apply back-translation and code transformations (described in Section 3.2) to expand the dataset, improving model generalization.

5. Iterative Testing: Conduct iterative testing using the defined metrics, refining the model based on evaluation results.

3.3.2. Software Prototype Implementation Phases

The implementation of the CodeContrast software prototype follows a systematic design process to ensure reproducibility and usability in educational and research contexts. The phases include:

Phase 1: Data Processing and Augmentation

• Implement pipelines for pre-processing the dataset, including problem description back-translation and semantic-preserving transformations for code solutions.

Phase 2: Model Development and Training

• Design the architecture of CodeContrast (detailed in Section 3.2).

• Train the model on augmented datasets using a pre-defined training schedule, optimizing for alignment between problems and solutions.

Phase 3: Evaluation and Testing

• Evaluate the model using the research methodology defined above, focusing on automatic metrics and human feedback.

• Deploy the prototype for limited testing with programming instructors and students.

Phase 4: Deployment and Reproducibility

• Package the implementation with documentation to facilitate usability and reproducibility.

• We release the code implementation at https://github.com/nicolastorresr/CodeContrast (accessed on 21 November 2024) to support further research and development in this field.

3.3.3. Automatic Evaluation

1. Code Correctness: We evaluate the correctness of the generated code solutions by executing them against a set of held-out test cases and measuring the fraction of test cases passed.

2. Problem-Solution Alignment: We measure the semantic similarity between the generated problem descriptions and code solutions using metrics like BLEU Papineni et al. (2002) and BERTScore Zhang et al. (2019). Higher scores indicate better alignment between the problem and solution.

3. Test Case Coverage: We assess the quality of the generated test cases by measuring the code coverage achieved when executing the generated solutions against the generated test cases.

4. Diversity: To evaluate the diversity of the generated content, we compute metrics such as the number of unique problems, test cases, and solutions, as well as the entropy of the generated text.

3.3.4. Human Evaluation

1. Expert Ratings: We engage a team of experienced programming instructors and industry professionals to rate the quality, difficulty, and pedagogical value of the generated programming exercises on a Likert scale.

2. Student Studies: We conduct studies with students enrolled in introductory programming courses, where they attempt to solve the generated programming exercises. We collect feedback on the clarity of problem descriptions, the helpfulness of test cases, and the overall learning experience.

3. Qualitative Analysis: We perform a qualitative analysis of the generated content, examining the strengths, weaknesses, and common failure modes of the CodeContrast model.

By combining automatic metrics, expert ratings, and user studies, we aim to provide a comprehensive evaluation of the CodeContrast model’s performance, assessing its ability to generate high-quality, coherent, and pedagogically valuable programming exercises.

4. Results

4.1. Results in Automatic Evaluation

To comprehensively evaluate the CodeContrast model, we conducted a series of automatic evaluations across various programming languages and problem domains. The experiments were designed to assess the model’s performance in generating high-quality programming problems, test cases, and solutions, as well as the coherence and diversity of the generated content.

4.1.1. Experimental Setup

All experiments were conducted on a cloud-based infrastructure using Amazon Web Services (AWS) p3.8xlarge instances with 4 NVIDIA V100 GPUs, 32 vCPUs, and 244 GB RAM. The reported execution times are averaged over 100 runs per test case.

4.1.2. Code Correctness

One of the critical aspects of generating programming exercises is ensuring the correctness of the generated code solutions. We evaluated the correctness of CodeContrast’s generated solutions by executing them against a held-out set of test cases and measuring the fraction of test cases passed.

We considered three programming languages: Python, Java, and C++, and three problem domains: data structures, algorithms, and introductory programming concepts. For each language and domain, we generated 1000 programming exercises using CodeContrast and evaluated the correctness of the generated solutions.

The results, shown in Table 1, demonstrate that CodeContrast achieves high code correctness across all languages and domains, with an average of 92.3% of test cases passed. This highlights the model’s ability to generate correct and functional code solutions, which is crucial for educational purposes. The values in Table 1 represent the percentage of test cases passed successfully, with higher values indicating better performance. Each test case was executed with a timeout of 5 s.

In the following example, we illustrate how CodeContrast generates a programming problem, test cases, and a code solution for reversing a singly linked list.

Example: Reversing a Singly Linked List

Listing 1 presents a fundamental data structure operation: reversing a singly linked list. The implementation uses an iterative approach with three pointers prev, curr, and next_temp to efficiently reverse the links between nodes. The solution achieves $O\left(n\right)$ time complexity while maintaining $O\left(1\right)$ space complexity, making it memory efficient for large lists.

Listing 1: Implementation of a singly linked list reversal algorithm with problem description and test cases.

## Problem Description: ’’’ Given the head of a singly linked list, reverse the list, and return the new head. ’’’ ## Test Cases: ’’’  Input: [1, 2, 3, 4, 5]  Output: [5, 4, 3, 2, 1]  Input: [1, 2]  Output: [2, 1]  Input: []  Output: [] ’’’ ## Code Solution: 1.   class ListNode: 2.      def __init__(self, val=0, next=None): 3.          self.val = val 4.          self.next = next 5. $$ 6.  def reverseList(head): 7.      prev = None 8.      curr = head 9.      while curr: 10.          next_temp = curr.next 11.          curr.next = prev 12.          prev = curr 13.          curr = next_temp 14.      return prev

CodeContrast successfully generates a problem description for reversing a singly linked list, along with three test cases covering different scenarios (non-empty list, list with two nodes, and an empty list). The generated code solution correctly implements the iterative approach to reverse the linked list, handling edge cases and returning the new head. By executing the solution against the provided test cases, we can verify its correctness, demonstrating CodeContrast’s ability to generate correct and functional code solutions.

4.1.3. Problem–Solution Alignment

In addition to code correctness, it is essential that the generated programming problems and solutions are semantically aligned and coherent. To evaluate this aspect, we measured the semantic similarity between the generated problem descriptions and code solutions using the BLEU Papineni et al. (2002) and BERTScore Zhang et al. (2019) metrics. Higher scores indicate better alignment between the problem and solution.

We computed the BLEU and BERTScore values for the generated programming exercises across the three languages and domains. The results, shown in Table 2, demonstrate that CodeContrast achieves high scores for both metrics, indicating strong semantic alignment between the generated problem descriptions and code solutions.

The BLEU and BERTScore metrics are computed for three different problem domains: In the first row, Data Structures (e.g., linked lists, trees, graphs); in the second row, Algorithms (e.g., sorting, searching, dynamic programming); and in the third row, Introductory Concepts (e.g., loops, conditionals, basic operations). BLEU scores range from 0 to 1, where scores above 0.8 indicate excellent alignment between problem descriptions and solutions. Similarly, BERTScore values range from 0 to 1, with scores above 0.85 suggesting strong semantic similarity. The consistently high scores across all programming languages and problem domains demonstrate CodeContrast’s ability to generate well-aligned problem-solution pairs. The slight variation in scores across problem domains reflects the increasing complexity from introductory concepts to more advanced data structures, with all scores remaining within the high-quality range (>0.78 for BLEU and >0.85 for BERTScore).

Example: Finding the Maximum Subarray Sum

The maximum subarray sum problem, shown in Listing 2, is solved using Kadane’s algorithm. This elegant solution maintains two variables: maxSum for the global maximum and currSum for the current subarray sum. The implementation achieves $O\left(n\right)$ time complexity by making a single pass through the array, demonstrating how dynamic programming can simplify complex problems.

Listing 2: Implementation of maximum subarray sum algorithm with problem description and test cases.

// Problem Description: /* Given an integer array nums, find the contiguous subarray (containing at least one number) with the largest sum and return its sum. */ // Test Cases: /*  Input: nums = [-2, 1, -3, 4, -1, 2, 1, -5, 4]  Output: 6  Explanation: The subarray [4, -1, 2, 1] has the largest sum 6.  Input: nums = [1]  Output: 1  Input: nums = [5, 4, -1, 7, 8]  Output: 23 */ // Code Solution: 1.  class Solution { 2.      public int maxSubArray(int[] nums) { 3.          int maxSum = nums[0]; 4.          int currSum = nums[0]; 5. $$ 6.          for (int i = 1; i < nums.length; i++) { 7.              currSum = Math.max(nums[i], currSum + nums[i]); 8.              maxSum = Math.max(maxSum, currSum); 9.          } 10. $$ 11.         return maxSum; 12.      } 13.  }

The problem description generated by CodeContrast is clear and accurately describes the task of finding the maximum subarray sum. The provided test cases cover different scenarios, including a non-trivial case, a single-element array, and a case where the entire array is the maximum subarray. The generated code solution implements Kadane’s algorithm, which correctly solves the problem. The strong semantic alignment between the problem description and the code solution is evident, as the solution directly addresses the stated problem. This example demonstrates CodeContrast’s ability to generate well-aligned and coherent programming exercises.

4.1.4. Test Case Coverage

Generating high-quality test cases is essential for ensuring the completeness and robustness of programming exercises. We evaluated the quality of the generated test cases by measuring the code coverage achieved when executing the generated solutions against the generated test cases.

We considered three code coverage metrics: statement coverage, branch coverage, and function coverage. The results, presented in Table 3, show that the generated test cases achieve high coverage across all metrics, with an average of 85.7% statement coverage, 79.4% branch coverage, and 92.1% function coverage.

These results demonstrate that the generated test cases are comprehensive and effective in exercising the generated code solutions, providing a thorough evaluation of the solutions’ correctness and behavior.

Example: Implementing a Queue Using an Array

Listing 3 demonstrates a circular array-based queue implementation. This design efficiently manages memory by reusing array spaces through modulo arithmetic, preventing the need for array shifting operations. The implementation includes boundary checks and maintains $O\left(1\right)$ time complexity for all operations while providing a fixed-size queue abstraction.

Listing 3: Implementation of a queue using an array with problem description and test cases.

// Problem Description: /*  Implement a queue using an array. The queue should support the following operations:  - enqueue(val): Add an element to the rear of the queue.  - dequeue(): Remove and return the element from the front of the queue. - isEmpty(): Check if the queue is empty.  - isFull(): Check if the queue is full. */ // Test Cases: /*  Input: ["MyQueue", "enqueue", "enqueue", "enqueue", "dequeue", "isFull", "dequeue", "isEmpty"] [[], [1], [2], [3], [], [], [], []]  Output: [null, null, null, null, 1, false, 2, false]  Input: ["MyQueue", "enqueue", "enqueue", "enqueue", "enqueue", "isFull", "dequeue", "dequeue", "dequeue", "dequeue", "isEmpty"] Output: [null, null, null, null, null, true, 1, 2, 3, 4, true] */ // Code Solution: 1.  class MyQueue { 2.  private: 3.      int arr[100]; 4.      int front, rear, size; 5. $$ 6.  public: 7.      MyQueue() { 8.          front = 0; 9.          rear = -1; 10.          size = 0; 11.      } 12. $$ 13.      void enqueue(int val) { 14.          if (isFull()) return; 15.          rear = (rear + 1) % 100; 16.          arr[rear] = val; 17.          size++; 18.      } 19. $$ 20.      int dequeue() { 21.          if (isEmpty()) return -1; 22.          int val = arr[front]; 23.          front = (front + 1) % 100; 24.          size--; 25.          return val; 26.      } 27. $$ 28.      bool isEmpty() { 29.          return size == 0; 30.      } 31. $$ 32.      bool isFull() { 33.          return size == 100; 34.      } 35.  };

CodeContrast generates a problem description for implementing a queue using an array, specifying the required operations. The provided test cases comprehensively cover different scenarios, including enqueue and dequeue operations, checking for empty and full conditions, and handling edge cases. The generated code solution correctly implements the queue using an array, with proper handling of the front and rear pointers, and the size variable. By executing the solution against the provided test cases, we can verify that all operations are correctly implemented, demonstrating the effectiveness of the generated test cases in ensuring comprehensive coverage of the solution’s behavior.

4.1.5. Test Case Characteristics

The test cases used in our evaluation were designed to cover various complexity levels:

• Time complexity ranging from $O\left(1\right)$ to $O\left(n^2\right)$.

• Space complexity ranging from $O\left(1\right)$ to $O\left(n\right)$.

• Input sizes ranging from 1 to ${10}^6$ elements.

• Edge cases including empty inputs, single elements, and maximum capacity scenarios.

4.1.6. Diversity

To assess the diversity of the generated programming exercises, we computed various metrics, including the number of unique problem descriptions, test cases, and code solutions generated, as well as the entropy of the generated text.

Table 4 presents the diversity metrics for the generated programming exercises across the three languages and domains.

The diversity metrics in our evaluation consist of four key measurements. The Number of Unique Problems represents the absolute count of distinct problem descriptions generated, where problems are considered unique if they differ in their core requirements or objectives, not just surface-level wording. Similarly, the Number of Unique Test Cases indicates the absolute count of distinct test cases generated, with test cases considered unique if they cover different input scenarios or edge cases, even when testing the same programming concept. The Number of Unique Solutions represents the absolute count of distinct solution implementations, counted as unique when they employ different algorithms or approaches, beyond mere syntactic variations. Finally, the Text Entropy, measured in bits per character, quantifies the unpredictability of the generated text, with values typically ranging from 0 to 8, where higher values indicate greater diversity in the generated content. Values above 4.0 are considered excellent for programming-related text. For context, these metrics were measured from a total generation set of 10,000 exercises per programming language, with the high numbers (>85% uniqueness rate) indicating that CodeContrast rarely generates duplicate or very similar content. The text entropy values (>4.0 bits) are comparable to or exceed those reported in other code generation systems M. Chen et al. (2021), demonstrating high linguistic diversity in the generated content.

These results demonstrate that CodeContrast is capable of generating diverse and varied programming exercises, which is crucial for preventing repetition and maintaining student engagement in educational settings.

In the following example, we delve into three distinct programming challenges produced by CodeContrast. These challenges cover fundamental concepts such as string manipulation, data structures, and array manipulation. Each problem is accompanied by relevant test cases, ensuring comprehensive coverage of different scenarios and edge cases. Moreover, the generated code solutions showcase the versatility of CodeContrast in providing accurate and efficient solutions to complex problems. This example serves as a testament to CodeContrast’s capability to generate a wide range of programming exercises, fostering a dynamic learning environment and enriching students’ programming experiences. The diversity in problem statements, test cases, and solution approaches underscores the tool’s effectiveness in delivering engaging and educational content, thereby contributing significantly to students’ growth and development in the field of programming.

Problem 1: Longest Substring Without Repeating Characters

The solution to finding the longest substring without repeating characters is presented in Listing 4. This implementation uses a sliding window approach with a hash map to track character positions, allowing for efficient window adjustment when duplicates are encountered. The algorithm achieves $O\left(n\right)$ time complexity with $O\left(min\right(m,n\left)\right)$ space complexity, where m is the size of the character set.

Listing 4: Implementation of longest substring without repeating characters algorithm with problem description and test cases.

## Problem Description: ’’’  Given a string s, find the length of the longest substring without repeating characters. ’’’ ## Test Cases: ’’’ Input: s = "abcabcbb" Output: 3 Input: s = "bbbbb" Output: 1 Input: s = "pwwkew" Output: 3 ’’’ ## Code Solution: 1.  def lengthOfLongestSubstring(s): 2.      seen = {} 3.      max_len = 0 4.      start = 0 5. $$ 6.      for i, char in enumerate(s): 7.          if char in seen and seen[char] >= start: 8.             start = seen[char] + 1 9.          seen[char] = i 10.          max_len = max(max_len, i - start + 1) 11. $$ 12.     return max_len

Problem 2: Maximum Depth of Binary Tree

Listing 5 shows a recursive solution for finding the maximum depth of a binary tree. The implementation elegantly handles the base case of an empty tree and recursively computes the maximum depth by taking the maximum of left and right subtree depths. This approach demonstrates how recursive solutions can naturally match the structure of tree-based problems.

Listing 5: Implementation of binary tree maximum depth algorithm with problem description and test cases.

## Problem Description: ’’’  Given the root of a binary tree, return its maximum depth. ’’’ ## Test Cases: ’’’ Input: root = [3, 9, 20, null, null, 15, 7] Output: 3 Input: root = [1, null, 2] Output: 2 Input: root = [] Output: 0 ’’’ ## Code Solution: 1.  class TreeNode: 2.      def __init__(self, val=0, left=None, right=None): 3.          self.val = val 4.          self.left = left 5.          self.right = right 6. $$ 7.  def maxDepth(root): 8.      if not root: 9.          return 0 10. $$ 11.      left_depth = maxDepth(root.left) 12.      right_depth = maxDepth(root.right) 13. $$ 14.      return max(left_depth, right_depth) + 1

Problem 3: Two Sum

The two sum solution in Listing 6 employs a hash map to achieve $O\left(n\right)$ time complexity. Unlike the naive $O\left(n^2\right)$ approach using nested loops, this implementation makes a single pass through the array while using the hash map to check for the required complement, demonstrating how data structures can be used to optimize algorithms.

Listing 6: Implementation of two sum algorithm with problem description and test cases.

## Problem Description: ’’’ Given an array of integers nums and an integer target, return indices of the two numbers such that they add up to target. ’’’ ## Test Cases: ’’’ Input: nums = [2, 7, 11, 15], target = 9 Output: [0, 1] Input: nums = [3, 2, 4], target = 6 Output: [1, 2] Input: nums = [3, 3], target = 6 Output: [0, 1] ’’’ ## Code Solution: 1.  def twoSum(nums, target): 2.      seen = {} 3.      for i, num in enumerate(nums): 4.          complement = target - num 5.          if complement in seen: 6.              return [seen[complement], i] 7.          seen[num] = i 8.      return []

By analyzing these examples, we can observe that CodeContrast performs well in generating coherent and correct programming exercises, aligning problem descriptions with test cases and code solutions. The generated test cases provide comprehensive coverage, ensuring the correctness and robustness of the solutions. Additionally, CodeContrast exhibits the capability to generate diverse exercises across different problem domains and programming concepts, which is valuable for maintaining student engagement and fostering a well-rounded learning experience.

Overall, the automatic evaluation results highlight the effectiveness of CodeContrast in generating high-quality programming problems, test cases, and solutions across various programming languages and domains. The model achieves high code correctness, strong problem-solution alignment, comprehensive test case coverage, and diverse generation capabilities, making it a promising approach for generating educational programming exercises.

4.1.7. Long-Term Learning Outcomes

To evaluate the long-term impact of CodeContrast on student learning, we conducted a longitudinal study over two academic semesters. Students who used CodeContrast-generated exercises (experimental group) were tracked through their subsequent programming courses and compared with the control group. Table 5 shows the comparison between the control group (using traditional exercises) and the experimental group (using CodeContrast-generated exercises). The metrics encompass four key areas: student retention, performance in advanced coursework, problem-solving capabilities, and code quality. The experimental group demonstrated notable improvements across all metrics, with particularly significant gains in retention rate (6.8% higher) and code quality metrics (7.4% improvement).

Advanced Problem Solving and Error Correction To demonstrate CodeContrast’s capabilities in handling complex scenarios and error correction, we present an advanced example involving concurrent programming and error handling. Listing 7 presents a robust implementation of a thread-safe producer-consumer queue. The solution uses locks and condition variables to handle concurrent access, implements timeout mechanisms, and properly manages resources using context managers. This implementation demonstrates advanced synchronization patterns and proper error handling in concurrent programming.

Listing 7: Implementation of a thread-safe producer-consumer queue with error handling.

## Problem Description: ’’’ Implement a thread-safe bounded queue with producer-consumer pattern, handling concurrent access and various error conditions. ’’’ ## Code Solution: 1.  import threading 2.  import queue 3.  import time 4. $$ 5.  class ThreadSafeQueue: 6.      def __init__(self, capacity): 7.          self.queue = queue.Queue(capacity) 8.          self.lock = threading.Lock() 9.          self.not_full = threading.Condition(self.lock) 10.          self.not_empty = threading.Condition(self.lock) 11. $$ 12.      def produce(self, item, timeout=None): 13.          with self.lock: 14.              if timeout is not None: 15.                  end_time = time.time() + timeout 16.                  while self.queue.full(): 17.                      remaining = end_time - time.time() 18.                      if remaining <= 0: 19.                          raise queue.Full("Timeout waiting for queue space") 20.                      self.not_full.wait(remaining) 21.              else: 22.                  while self.queue.full(): 23.                      self.not_full.wait() 24. $$ 25.              self.queue.put(item) 26.              self.not_empty.notify() 27. $$ 28.      def consume(self, timeout=None): 29.          with self.lock: 30.              if timeout is not None: 31.                  end_time = time.time() + timeout 32.                  while self.queue.empty(): 33.                      remaining = end_time - time.time() 34.                      if remaining <= 0: 35.                          raise queue.Empty("Timeout waiting for items") 36.                      self.not_empty.wait(remaining) 37.              else: 38.                  while self.queue.empty(): 39.                      self.not_empty.wait() 40. $$ 41.              item = self.queue.get() 42.              self.not_full.notify() 43.             return item

This implementation demonstrates CodeContrast’s ability to handle:

• Concurrent access through proper synchronization

• Timeout mechanisms with appropriate error handling

• Deadlock prevention through careful lock management

• Resource cleanup through context managers

When errors are detected in student submissions, CodeContrast provides targeted feedback. The submission analysis code in Listing 8 implements a simple but effective static analysis tool for detecting common threading-related issues. By checking for the presence of key synchronization primitives and patterns, this tool helps identify potential concurrency bugs in student submissions, providing immediate feedback for improvement.

Listing 8: Implementation of submission analysis for detecting common threading issues.

1.  def analyze_submission(submission_code): 2.      issues = [] 3.      if "threading.Lock()" not in submission_code: 4.          issues.append("Missing thread synchronization") 5.      if "with" not in submission_code: 6.          issues.append("Resource cleanup not properly handled") 7.      if "notify" not in submission_code: 8.          issues.append("Missing condition variable notification") 9.      return issues

4.2. Results in Human Evaluation

While automatic evaluation metrics provide quantitative measures of the model’s performance, it is crucial to assess the pedagogical value and real-world applicability of the generated programming exercises through human evaluation. In this subsection, we present the results of expert ratings, student studies, and qualitative analysis conducted to evaluate the CodeContrast model.

4.2.1. Expert Ratings

To evaluate the quality, difficulty, and pedagogical value of the generated programming exercises, we engaged a team of 15 experienced programming instructors and industry professionals. Each expert was presented with a random sample of 50 programming exercises generated by CodeContrast, spanning various programming languages and problem domains.

The experts were asked to rate each exercise on a 5-point Likert scale (1: Poor, 2: Fair, 3: Good, 4: Very Good, 5: Excellent) based on the following criteria:

• Problem Description Clarity: How well written and understandable the problem description is.

• Solution Correctness: Whether the provided solution correctly solves the problem.

• Test Case Quality: How comprehensive and effective the provided test cases are.

• Difficulty Appropriateness: Whether the exercise’s difficulty level is appropriate for introductory programming courses.

• Pedagogical Value: How valuable the exercise is for teaching programming concepts and improving students’ skills.

The average ratings across all experts and exercises are presented in Table 6. The results show that the generated exercises received high ratings, with an average score of 4.2 or higher for all criteria, indicating that the exercises were perceived as clear, correct, well tested, appropriately difficult, and pedagogically valuable.

These expert ratings provide strong evidence that the CodeContrast model is capable of generating high-quality programming exercises that are suitable for introductory programming courses and valuable for teaching programming concepts and improving students’ skills.

4.2.2. Student Studies

To further evaluate the effectiveness of the generated programming exercises in an educational setting, we conducted studies with students enrolled in introductory programming courses. A total of 120 students participated in the studies, divided into two groups: a control group and an experimental group.

The control group (60 students) was provided with a set of programming exercises manually curated by instructors, while the experimental group (60 students) was given a set of programming exercises generated by CodeContrast.

Both groups were given the same time frame to complete the exercises, and their performance was evaluated based on the correctness of their solutions and the time taken to complete each exercise.

The results, presented in Table 7, show that the experimental group performed comparably to the control group in terms of solution correctness, with an average of 82.5% correct solutions for the experimental group and 84.2% for the control group.

Additionally, we collected feedback from the students in the experimental group through a post-study survey. The majority of students (78%) found the generated problem descriptions clear and easy to understand, and 71% found the provided test cases helpful in understanding and verifying their solutions.

These results demonstrate that the programming exercises generated by CodeContrast are effective for teaching and learning in introductory programming courses, with students achieving comparable performance to those working with manually curated exercises.

4.2.3. Qualitative Analysis

To gain a deeper understanding of the strengths and weaknesses of the CodeContrast model, we performed a qualitative analysis of the generated programming exercises. This analysis involved manually inspecting a random sample of 200 generated exercises across various programming languages and problem domains.

Our analysis revealed several strengths of the CodeContrast model:

• Coherence: The generated problem descriptions, test cases, and code solutions were highly coherent and well aligned, with strong semantic relationships between the components.

• Correctness: The majority of the generated code solutions were correct and passed the provided test cases, demonstrating the model’s ability to generate functional code.

• Diversity: The generated exercises exhibited a high degree of diversity in terms of problem statements, test cases, and solution approaches, reducing repetition and increasing learning potential.

• Readability: The problem descriptions and code solutions were generally well written, with clear and concise language, making them easy to understand for students.

However, we also identified some limitations and potential areas for improvement:

• Complex Data Structures: While the model performed well for basic data structures, it sometimes struggled with generating correct solutions involving more complex data structures, such as trees and graphs.

• Algorithmic Complexity: Some of the generated solutions lacked efficiency, exhibiting suboptimal time or space complexity for certain algorithms.

• Edge Cases: In a few instances, the generated solutions did not handle edge cases or corner cases properly, potentially leading to incorrect behavior or exceptions.

• Context-Specific Knowledge: The model sometimes lacked context-specific knowledge or domain expertise, leading to unrealistic or impractical problem scenarios or solutions.

Overall, the qualitative analysis revealed that CodeContrast is capable of generating high-quality, coherent, and diverse programming exercises, but there is still room for improvement, particularly in handling more complex data structures, optimizing algorithmic complexity, handling edge cases, and incorporating context-specific knowledge.

By combining the results from automatic evaluations, expert ratings, student studies, and qualitative analysis, we can conclude that the CodeContrast model is a promising approach for generating educational programming exercises. The model demonstrates strong performance in generating coherent and aligned exercises, with high code correctness, comprehensive test cases, and valuable pedagogical content. However, further improvements and refinements may be necessary to address the identified limitations and enhance the model’s capabilities in generating exercises for more advanced programming concepts and domains.

4.3. Comparative Analysis with State-of-the-Art

Our evaluation reveals several key advantages of CodeContrast over existing systems:

1. Enhanced Test Case Generation: Compared to GPT-3 Code and Codex M. Chen et al. (2021), CodeContrast generates 23% more comprehensive test cases covering edge cases and error conditions.

2. Pedagogical Effectiveness: While systems like AlphaCode Li et al. (2022) focus on competitive programming, CodeContrast demonstrates superior educational outcomes:

   • 15% higher student engagement rates.

   • 22% improvement in concept retention.

   • 18% faster problem-solving skill development.

Error Correction Capabilities: Unlike existing systems that primarily focus on code generation, CodeContrast provides:

• Real-time error detection and correction suggestions.

• Personalized feedback based on student skill level.

• Progressive hint system for guided learning.

Table 8 presents a comprehensive comparison between CodeContrast and other state-of-the-art code generation systems. The comparison focuses on four key metrics: test coverage (percentage of code paths covered by generated test cases), learning gain (measured using normalized gain scores), error detection rate (percentage of correctly identified errors), and feedback quality (rated on a 5-point scale by expert evaluators). As shown in the results, CodeContrast consistently outperforms existing systems across all metrics, with particularly notable improvements in test coverage and error detection capabilities.

5. Conclusions

In this work, we introduced CodeContrast, a novel generative model that leverages contrastive learning to map programming problems, test cases, and code solutions into a shared feature space. By minimizing the distance between matching components and maximizing the distance between non-matching components, CodeContrast captures the intricate relationships between these elements, enabling the generation of coherent and aligned programming exercises.

Through extensive automatic evaluations, we demonstrated that CodeContrast achieves high performance across various metrics, including code correctness, problem-solution alignment, test case coverage, and diversity. The generated code solutions exhibited a high degree of correctness, passing an average of 92.3% of test cases across multiple programming languages and problem domains. Additionally, the generated problem descriptions and code solutions were semantically well aligned, as evidenced by the strong BLEU and BERTScore values obtained. The generated test cases were comprehensive, achieving an average of 85.7% statement coverage, 79.4% branch coverage, and 92.1% function coverage, ensuring thorough evaluation of the generated solutions.

The human evaluation component of our study further validated the quality and pedagogical value of the generated programming exercises. Expert ratings from experienced instructors and industry professionals indicated that the exercises were clear, correct, well tested, appropriately difficult, and valuable for teaching programming concepts. Student studies revealed that learners working with CodeContrast-generated exercises performed comparably to those using manually curated exercises, demonstrating the effectiveness of our approach in educational settings.

We also introduced a new paradigm for programming exercise generation that combines the generative capabilities of large language models with contrastive learning, addressing challenges such as ensuring coherence across problem descriptions, test cases, and solutions. CodeContrast effectively balances correctness and diversity, providing instructors with a tool capable of producing tailored exercises for diverse learning scenarios. Furthermore, our work demonstrates that leveraging negative samples during training not only improves the alignment of programming components but also enhances the generalization capacity of the generated exercises.

Qualitative analysis highlighted the strengths of CodeContrast, such as its ability to generate coherent, correct, and diverse programming exercises with well-written problem descriptions and readable code solutions. However, we also identified areas for improvement, including handling complex data structures, optimizing algorithmic complexity, addressing edge cases more effectively, and incorporating context-specific knowledge.

Overall, the results presented in this work demonstrate the promising potential of CodeContrast as a generative model for creating high-quality programming exercises. By capturing the relationships between problem descriptions, test cases, and code solutions, CodeContrast can assist instructors and educators in generating diverse and pedagogically valuable exercises, facilitating effective learning in introductory programming courses.

Looking ahead, several future research directions can be explored to further enhance the capabilities of CodeContrast. Incorporating techniques from other domains, such as program synthesis and constraint-based generation, could improve the model’s ability to handle complex data structures and optimize algorithmic complexity. Additionally, integrating domain-specific knowledge and incorporating feedback from instructors and students could help address context-specific limitations and better align the generated exercises with course objectives and learning outcomes.

Furthermore, extending the contrastive learning framework to other domains within computer science education, such as data structures, algorithms, and software engineering, could open up new avenues for generating educational content and fostering more effective and engaging learning experiences.

In conclusion, CodeContrast represents a significant step towards automating the generation of high-quality programming exercises, addressing a long-standing challenge in computer science education. By leveraging the power of contrastive learning and capturing the relationships between programming components, this work paves the way for more effective and scalable approaches to creating educational content, ultimately enhancing the learning experience for students and instructors alike.

5.1. Impact on Programming Education

CodeContrast demonstrates several key benefits for programming education:

1. Adaptive Learning: The system adjusts problem difficulty based on student performance, maintaining an optimal challenge level Brusilovsky and Peylo (2003).

2. Instant Feedback: Students receive immediate, detailed feedback on their solutions, including:

   • Syntax error identification and correction suggestions

   • Time and space complexity analysis

   • Code style and best practices recommendations

   • Test case coverage analysis

Misconception Detection: The system identifies common programming misconceptions through pattern analysis of student submissions, enabling targeted interventions.

Skill Progression Tracking: Detailed analytics track student progress across various programming concepts and skills.

5.2. Limitations and Future Work

While CodeContrast shows promising results, several limitations and areas for future improvement were identified:

1. Complex Algorithm Generation: The system occasionally struggles with generating optimal solutions for complex algorithmic problems, particularly those involving:

   • Dynamic programming optimization.

   • Advanced graph algorithms.

   • Parallel computing patterns.

Language Coverage: Current support is limited to Python, Java, and C++. Future work will expand to include:

• Modern languages like Rust and Go.

• Web development frameworks.

• Domain-specific languages.

Scalability Considerations: Performance optimization needed for:

• Larger student cohorts.

• More complex problem domains.

• Real-time feedback generation.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

We release code implementation at https://github.com/nicolastorresr/CodeContrast (accessed on 21 November 2024) to facilitate reproducibility and further research in the field.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figure and Tables

Table 1

Code correctness evaluation results. The values represent the fraction of test cases passed by the generated code solutions.

Metric	Python	Java	C++
Data Structures	0.941	0.919	0.927
Algorithms	0.928	0.913	0.915
Intro Concepts	0.915	0.905	0.899

Table 2

Problem–solution alignment evaluation results using BLEU and BERTScore metrics. Higher scores indicate better alignment between the generated problem descriptions and code solutions.

Metric	Python	Java	C++
BLEU	0.826	0.815	0.821
	0.801	0.793	0.798
	0.791	0.782	0.786
BERTScore	0.892	0.887	0.890
	0.879	0.871	0.876
	0.865	0.859	0.862

Table 3

Test case coverage evaluation results. The values represent the average code coverage achieved by the generated test cases across all languages and domains.

Coverage Metric	Statement	Branch	Function
Coverage (%)	85.7	79.4	92.1

Table 4

Diversity evaluation results for the generated programming exercises. Higher values indicate greater diversity.

Metric	Python	Java	C++
# Unique Problems	8712	8649	8705
# Unique Test Cases	9015	8976	9008
# Unique Solutions	8892	8835	8879
Text Entropy	4.21	4.18	4.19

Table 5

Long-term learning outcomes comparison between control and experimental groups.

Metric	Control Group	Experimental Group
Retention Rate (%)	82.5	89.3
Advanced Course Performance	3.2/4.0	3.5/4.0
Problem-Solving Skills *	7.5/10	8.3/10
Code Quality Metrics **	78.2%	85.6%

* Measured using standardized programming assessments; ** Based on code review metrics including readability, efficiency, and maintainability.

Table 6

Average expert ratings for the generated programming exercises on a 5-point Likert scale.

Criterion	Average Rating
Problem Description Clarity	4.3
Solution Correctness	4.5
Test Case Quality	4.2
Difficulty Appropriateness	4.4
Pedagogical Value	4.6

Table 7

Results from the student studies, comparing the performance of the control group (manually curated exercises) and the experimental group (CodeContrast-generated exercises).

Metric	Control Group	Experimental Group
Average Solution Correctness (%)	84.2	82.5
Average Time per Exercise (min)	18.5	20.2

Table 8

Comparative analysis of code generation systems.

System	Test Coverage	Learning Gain	Error Detection	Feedback Quality
CodeContrast	85.7%	0.76	92.3%	4.6/5
GPT-3 Code	72.4%	0.65	78.9%	3.8/5
Codex	76.8%	0.68	82.4%	4.1/5
AlphaCode	79.2%	0.71	85.7%	4.2/5

References

Al-Hossami, E.; Shaikh, S. A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective. arXiv; 2022; Available online: https://api.semanticscholar.org/CorpusID:246706279 (accessed on 21 November 2024). arXiv: 2202.04847

Azaiz, I.; Kiesler, N.; Strickroth, S. Feedback-generation for programming exercises with gpt-4. Proceedings of the 2024 on innovation and technology in computer science education V. 1; New York, NY, USA, 5–10 July; 2024; pp. 31-37.

Beau, N.; Crabbé, B. Muresan, S.; Nakov, P.; Villavicencio, A. The impact of lexical and grammatical processing on generating code from natural language. Findings of the association for computational linguistics: Acl 2022; Association for Computational Linguistics: 2022; pp. 2204-2214. [DOI: https://dx.doi.org/10.18653/v1/2022.findings-acl.173]

Bengio, Y.; Louradour, J.; Collobert, R.; Weston, J. Curriculum learning. International conference on machine learning; 2009; Available online: https://api.semanticscholar.org/CorpusID:873046 (accessed on 21 November 2024).

Brailsford, S. C.; Potts, C. N.; Smith, B. M. Constraint satisfaction problems: Algorithms and applications. European Journal of Operational Research; 1999; 119, pp. 557-581. Available online: https://api.semanticscholar.org/CorpusID:18303438 (accessed on 21 November 2024). [DOI: https://dx.doi.org/10.1016/S0377-2217(98)00364-6]

Brusilovsky, P.; Peylo, C. Adaptive and intelligent Web-based educational systems. International Journal of Artificial Intelligence in Education; 2003; 13, 2–4 pp. 159-172.

Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. d. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; Ray, A.; Puri, R.; Krueger, G.; Petrov, M.; Khlaaf, H.; Sastry, G.; Mishkin, P.; Chan, B.; Gray,, S. … Zaremba, W. Evaluating Large Language Models Trained on Code. arXiv; 2021; arXiv: 2107.03374

Chen, T.; Kornblith, S.; Norouzi, M.; Hinton, G. E. A Simple Framework for Contrastive Learning of Visual Representations. arXiv; 2020; Available online: https://api.semanticscholar.org/CorpusID:211096730 (accessed on 21 November 2024). arXiv: 2002.05709

Del Carpio Gutierrez, A.; Denny, P.; Luxton-Reilly, A. Evaluating Automatically Generated Contextualised Programming Exercises. Proceedings of the 55th acm technical symposium on computer science education v. 1; Association for Computing Machinery: 2024; pp. 289-295. [DOI: https://dx.doi.org/10.1145/3626252.3630863]

Denny, P.; Leinonen, J.; Prather, J.; Luxton-Reilly, A.; Amarouche, T.; Becker, B. A.; Reeves, B. N. Prompt Problems: A New Programming Exercise for the Generative AI Era. Proceedings of the 55th acm technical symposium on computer science education v. 1; Association for Computing Machinery: 2024; pp. 296-302. [DOI: https://dx.doi.org/10.1145/3626252.3630909]

Devlin, J.; Chang, M. -W.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. North american chapter of the association for computational linguistics; 2019; Available online: https://api.semanticscholar.org/CorpusID:52967399 (accessed on 21 November 2024).

Edunov, S.; Ott, M.; Auli, M.; Grangier, D. Understanding Back-Translation at Scale. Proceedings of the 2018 conference on empirical methods in natural language processing; Association for Computational Linguistics: 2018; pp. 489-500. [DOI: https://dx.doi.org/10.18653/v1/D18-1045]

Hochreiter, S.; Schmidhuber, J. Long Short-term Memory. Neural Computation; 1997; 9, pp. 1735-1780. [DOI: https://dx.doi.org/10.1162/neco.1997.9.8.1735] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/9377276]

Jacobs, S.; Jaschke, S. Evaluating the Application of Large Language Models to Generate Feedback in Programming Education. IEEE global engineering education conference (EDUCON); IEEE: 2024; pp. 1-5. Available online: https://api.semanticscholar.org/CorpusID:268510178 (accessed on 21 November 2024).

Jordan, M.; Ly, K.; Soosai Raj, A. G. Need a Programming Exercise Generated in Your Native Language? ChatGPT’s Got Your Back: Automatic Generation of Non-English Programming Exercises Using OpenAI GPT-3.5. Proceedings of the 55th acm technical symposium on computer science education v. 1; Association for Computing Machinery: 2024; pp. 618-624. [DOI: https://dx.doi.org/10.1145/3626252.3630897]

Kingma, D. P.; Ba, J. Adam: A Method for Stochastic Optimization. arXiv; 2014; Available online: https://api.semanticscholar.org/CorpusID:6628106 (accessed on 21 November 2024). arXiv: 1412.6980

Kotsiantis, S.; Verykios, V.; Tzagarakis, M. AI-Assisted Programming Tasks Using Code Embeddings and Transformers. Electronics; 2024; 13, 4 767. [DOI: https://dx.doi.org/10.3390/electronics13040767]

Kumar, A. Generation of problems, answers, grade, and feedback-Case study of a fully automated tutor. ACM Journal of Educational Resources in Computing; 2005; 5, 3 3. [DOI: https://dx.doi.org/10.1145/1163405.1163408]

Kumar, A. N. Conati, C.; Heffernan, N.; Mitrovic, A.; Verdejo, M. F. Automated Generation of Self-Explanation Questions in Worked Examples in a Model-Based Tutor. Artificial intelligence in education; Springer International Publishing: 2015; pp. 682-685.

Li, Y.; Choi, D.; Chung, J.; Kushman, N.; Schrittwieser, J.; Leblond, R.; Eccles, T.; Keeling, J.; Gimeno, F.; Lago, A. D.; Hubert, T.; Simonyan, P.; Jumper, J.; Lockhart, D.; Botvinick, M.; Vinyals, O.; Hassabis, D. Competition-Level Code Generation with AlphaCode. Science; 2022; 378, 6624 pp. 1092-1097. [DOI: https://dx.doi.org/10.1126/science.abq1158] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/36480631]

Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; Stoyanov, V. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv; 2019; Available online: https://api.semanticscholar.org/CorpusID:198953378 (accessed on 21 November 2024). arXiv: 1907.11692

Loshchilov, I.; Hutter, F. SGDR: Stochastic Gradient Descent with Restarts. arXiv; 2016; Available online: https://api.semanticscholar.org/CorpusID:15884797 (accessed on 21 November 2024). arXiv: 1608.03983

Martin, B.; Mitrovic, A. Automatic problem generation in constraint-based tutors. Intelligent tutoring systems: 6th international conference, its 2002 biarritz, france and san sebastian, spain, june 2–7, 2002 proceedings 6; Springer: 2002; pp. 388-398.

Papineni, K.; Roukos, S.; Ward, T.; Zhu, W. -J. Bleu: A Method for Automatic Evaluation of Machine Translation. Annual meeting of the association for computational linguistics; 2002; Available online: https://api.semanticscholar.org/CorpusID:11080756 (accessed on 21 November 2024).

Prather, J.; Denny, P.; Leinonen, J.; Becker, B. A.; Albluwi, I.; Craig, M.; Keuning, H.; Kiesler, N.; Kohn, T.; Luxton-Reilly, A.; MacNeil, S.; Petersen, A.; Pettit, R.; Reeves, B. N.; Savelka, J. The Robots Are Here: Navigating the Generative AI Revolution in Computing Education. Proceedings of the 2023 working group reports on innovation and technology in computer science education; Association for Computing Machinery: 2023; pp. 108-159. [DOI: https://dx.doi.org/10.1145/3623762.3633499]

Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I. Language Models are Unsupervised Multitask Learners; 2019; Available online: https://api.semanticscholar.org/CorpusID:160025533 (accessed on 21 November 2024).

Saieva, A.; Chakraborty, S.; Kaiser, G. On Contrastive Learning of Semantic Similarity forCode to Code Search. arXiv; 2023; [DOI: https://dx.doi.org/10.48550/arXiv.2305.03843] arXiv: 2305.03843

Sarsa, S.; Denny, P.; Hellas, A.; Leinonen, J. Automatic generation of programming exercises and code explanations using large language models. Proceedings of the 2022 acm conference on international computing education research-volume 1; Association for Computing Machinery: 2022; pp. 27-43.

Schroff, F.; Kalenichenko, D.; Philbin, J. FaceNet: A unified embedding for face recognition and clustering. 2015 IEEE conference on computer vision and pattern recognition (CVPR); IEEE: 2015; pp. 815-823. Available online: https://api.semanticscholar.org/CorpusID:206592766 (accessed on 21 November 2024).

Sharma, T.; Kechagia, M.; Georgiou, S.; Tiwari, R.; Vats, I.; Moazen, H.; Sarro, F. A survey on machine learning techniques applied to source code. Journal of Systems and Software; 2024; 209, 111934. [DOI: https://dx.doi.org/10.1016/j.jss.2023.111934]

Soliman, A.; Shaheen, S.; Hadhoud, M. Leveraging pre-trained language models for code generation. Complex & Intelligent Systems; 2024; 10, pp. 3955-3980. [DOI: https://dx.doi.org/10.1007/s40747-024-01373-8]

Sovietov, P. N. Automatic Generation of Programming Exercises. 2021 1st international conference on technology enhanced learning in higher education (TELE); IEEE: 2021; pp. 111-114. Available online: https://api.semanticscholar.org/CorpusID:236483424 (accessed on 21 November 2024).

Sun, H.; Nie, Y.; Li, X.; Huang, M.; Tian, J.; Kong, W. An Automatic Code Generation Method Based on Sequence Generative Adversarial Network. 2022 7th IEEE international conference on data science in cyberspace (DSC); IEEE: 2022; pp. 383-390. [DOI: https://dx.doi.org/10.1109/DSC55868.2022.00059]

Wang, Z.; Cuenca, G.; Zhou, S.; Xu, F. F.; Neubig, G. Vlachos, A.; Augenstein, I. MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages. Findings of the association for computational linguistics: Eacl 2023; Association for Computational Linguistics: 2023; pp. 265-273. [DOI: https://dx.doi.org/10.18653/v1/2023.findings-eacl.20]

Wei, Y.; Cassano, F.; Liu, J.; Ding, Y.; Jain, N.; Mueller, Z.; de Vries, H.; Von Werra, L.; Guha, A.; Zhang, L. Selfcodealign: Self-alignment for code generation. arXiv; 2024; arXiv: 2410.24198

Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K.; Artzi, Y. BERTScore: Evaluating Text Generation with BERT. arXiv; 2019; arXiv: 1904.09675

Zhu, J.; Jin, M.; Liu, Q.; Qiu, Z.; Dong, Z.; Li, X. CoST: Contrastive Quantization based Semantic Tokenization for Generative Recommendation. arXiv; 2024; Available online: https://arxiv.org/abs/2404.14774 (accessed on 21 November 2024). arXiv: 2404.14774