Comparative Analysis of Prompt Strategies for Large Language Models: Single-Task vs. Multitask Prompts

Abstract

This study investigates the effectiveness of prompt engineering strategies for Large Language Models (LLMs), comparing single-task and multitasking prompts. Specifically, we analyze whether a single prompt handling multiple tasks—such as named entity recognition (NER), sentiment analysis, and JSON output formatting—can achieve performance comparable to dedicated single-task prompts. To substantiate our findings, we employ statistical analyses, including paired Wilcoxon tests, McNemar tests, and Friedman tests, to validate claims of performance similarity or superiority. Experiments were conducted using five open-weight LLMs: LLama3.1 8B, Qwen2 7B, Mistral 7B, Phi3 Medium, and Gemma2 9B. The results indicate that there is no definitive rule favoring single-task prompts over multitask prompts; rather, their relative performance is highly contingent on the specific model’s data and architecture. This study highlights the nuanced interplay between prompt strategies and LLM characteristics, offering insights into optimizing their use for specific NLP tasks. Limitations and future directions, such as expanding task types, are also discussed.

1. Introduction

Large Language Models represent one of the most significant innovations in the field of artificial intelligence in recent years. These models are based on transformer neural architectures, characterized by self-attention mechanisms that allow for the processing of input sequences while maintaining contextual relationships between various elements. Their operation is based on a training process on enormous textual datasets, during which the model learns to predict the next token in a sequence, optimizing billions of parameters through the backpropagation process. The transformer architecture, introduced in the paper “Attention Is All You Need” [1], distinguishes itself through its ability to process input in parallel, overcoming the limitations of traditional recurrent neural networks (RNNs). The attention mechanism allows the model to weigh differently the importance of various parts of the input, creating dynamic contextual representations. This is made possible through three main components: the query, key, and value matrices, which together allow the model to establish complex correlations between different elements of the input. Large Language Models operate through a tokenization process that converts text into numerical sequences, using vocabularies that can contain tens or hundreds of thousands of tokens. Each token is then embedded in a high-dimensional vector space, where semantic and syntactic relationships are captured through vector distances. This distributed representation allows the model to capture complex linguistic nuances and generalize to cases never seen during training. A distinctive characteristic of modern Large Language Models is their few-shot and zero-shot [2] learning capability, meaning they can adapt to new tasks without the need for further training, simply through examples or textual instructions (prompts). This is made possible by their deep architecture and broad exposure to various contexts during training, allowing them to develop a sort of generalized language “understanding”.

Large Language Models are increasingly integrated into everyday life, making it crucial to understand and learn the correct interaction methods to effectively interface with these tools. The countless parameters that define Large Language Models are sufficient to emulate natural conversations. Mastering best practices in prompting is essential, as even minor variations in a prompt can significantly affect the outcome. In light of this, we have undertaken a research study aimed at quantifying the performance of multitask prompts in comparison to single-task prompts. Our initial hypothesis posits that single-task prompts, due to their inherently lower complexity, should yield better results than their multitask counterparts. While this assumption may seem intuitive, quantifying these differences and conducting a comparative analysis across various models is essential for understanding the performance degradation in prompting. Furthermore, despite the intuition that simpler, single-task prompts should produce better results, empirical data suggest that this is not universally true for all Large Language Models. Indeed, while certain models support our initial hypothesis, others exhibit superior performance with multitask prompts. This observation highlights the necessity for a thorough analysis to understand the underlying dynamics of each model. The differences, particularly those seen in models such as LLama 3.1 and Mistral, demonstrate that the interaction between a prompt and a model cannot be reduced to a simple general rule. This makes our study critical for optimizing the use of Large Language Models in various pipelines.

The objective of this study is to systematically and quantitatively analyze the performance differences between single-task and multitask prompts across a variety of large-scale language models. The goal is to provide empirical data that can guide developers and users in selecting the most effective prompting strategy for each model and use case. Our methodology includes defining a series of standardized tasks, which will be presented to various Large Language Models in both single-task and multitask formats. Performance will be measured using a composite metric that combines the results of three metrics applied to specific natural language processing tasks: F1 score for named entity recognition, exact match for sentiment analysis, and Bilingual Evaluation Understudy (BLEU) for review coherence. The findings from this research could have significant practical implications. Firstly, they provide data-driven guidelines to optimize interaction with Large Language Models in various application contexts. Additionally, any discrepancies observed among the models may offer valuable insights into the architectural and training differences that influence how Large Language Models respond to prompt complexity. One of the most well-known challenges of Large Language Models is the interpretability of their output. This study highlights this issue by revealing “non-standard” results when using what would be considered a standard approach.

2. Related Work

Recent research has explored various prompt engineering techniques for enhancing Large Language Models’ (Large Language Models) performance in natural language processing (NLP) tasks. Studies have investigated different types of prompts, including discrete, continuous, few-shot, and zero-shot approaches [3]. Discrete prompts are formulated using natural language, making them interpretable and easier to design, whereas continuous prompts leverage embeddings that are optimized through gradient-based methods, offering more flexibility but requiring specialized tuning techniques. Few-shot and zero-shot prompting techniques allow Large Language Models to perform tasks with minimal or no task-specific training examples, significantly reducing the need for extensive labeled data [3]. These strategies have shown promise in various contexts, highlighting the adaptability of Large Language Models to new tasks with limited supervision.

Researchers have also developed a catalog of prompt patterns to solve common problems when interacting with Large Language Models [4]. These patterns, which include guidelines for crafting effective prompts, address issues such as prompt ambiguity, model misinterpretations, and response consistency [4]. They provide a taxonomy of prompt patterns that categorize different approaches based on task types, such as classification, generation, and summarization, thus offering a structured framework for prompt design that can be easily applied across different NLP tasks. The effectiveness of different prompting strategies, such as simple prefix, cloze, chain-of-thought, and anticipatory prompts, has been empirically evaluated for clinical NLP tasks [5]. For instance, the chain-of-thought prompting technique encourages Large Language Models to break down complex tasks into intermediate reasoning steps, improving performance on tasks that require logical progression and detailed explanation. Anticipatory prompts, on the other hand, guide models to predict subsequent responses by leveraging prior knowledge of likely outcomes, enhancing the coherence and relevance of generated text, particularly in domains requiring domain-specific reasoning like clinical NLP [5].

Additionally, novel prompting techniques like heuristic and ensemble prompting have been introduced [5]. Heuristic prompting involves using domain-specific rules or prior knowledge to construct prompts that better align with the task at hand, while ensemble prompting combines multiple prompts to aggregate the strengths of different strategies, enhancing overall model performance and reducing variability in responses. These approaches provide robust alternatives to traditional single-prompt methods, demonstrating the potential of combining multiple prompt types for more reliable outputs. While most studies focus on single-task prompts, some research has explored multitask prompting approaches across various applications, from question–answering to commonsense reasoning [6]. Multitask prompting aims to create a single prompt that can handle multiple related tasks, thus improving the efficiency of Large Language Models by reducing the need to design task-specific prompts. This approach has been particularly beneficial in scenarios where models must adapt quickly to a wide range of questions or tasks without retraining, underscoring the versatility of prompt engineering as a technique for broadening the applicability of Large Language Models [6]. These advancements in prompt engineering contribute to improving Large Language Models’ performance across diverse NLP tasks without modifying core model parameters. By optimizing the way models interact with input prompts, researchers are able to enhance Large Language Models’ capabilities in handling complex, varied, and specialized tasks, driving forward the field of NLP and expanding the utility of these powerful models in real-world applications.

The Relation Between Prompt Architecture and Large Language Model Size

Recent research has delved into the interplay between the size of Large Language Models (LLMs) and the architecture of prompts used to guide them. The consensus emerging from these studies is that, while larger models generally benefit from prompts containing a greater number of examples, enhancing their overall accuracy and reliability [7], this is not an absolute rule. In particular contexts, smaller, more specialized models have been shown to outperform their larger counterparts. This phenomenon is especially evident in highly specific domains where smaller LLMs, fine-tuned on domain-specific data, demonstrate superior accuracy and efficiency compared to generalized, larger models [8]. These findings underscore the importance of model specialization and the targeted use of data, suggesting that model size alone is not a definitive predictor of performance. Moreover, the structure and complexity of prompts themselves play a critical role in determining LLM performance. Research has demonstrated that simple, straightforward prompt structures are often more effective for knowledge retrieval tasks compared to prompts that employ complex grammatical constructions. For instance, Linzbach et al. [9] found that prompts with less syntactic complexity allow models to retrieve relevant information more consistently, minimizing the risk of misinterpretation or erroneous outputs. This insight has significant implications for the design of user interactions with LLMs, suggesting that clarity and simplicity should be prioritized when crafting prompts, especially when the goal is to extract factual or technical information.

However, the sensitivity of LLMs to minor variations in prompt formatting remains a substantial challenge. Even subtle changes—such as the reordering of examples, shifts in tone, or slight alterations in wording—can have a pronounced impact on performance. In few-shot learning contexts, where LLMs are provided with only a handful of examples to guide their responses, Sclar et al. [10] observed discrepancies in accuracy reaching as high as 76 percentage points due to minimal prompt adjustments. Notably, this sensitivity is not confined to any specific model size or level of training; it persists across a wide spectrum of LLMs, independent of the number of examples given or the extent of instruction tuning applied. These findings highlight an inherent instability in LLM responses that complicates their evaluation, making it difficult to establish reliable benchmarks. To mitigate this variability, it has been suggested that assessments of LLMs should not rely on a single-prompt format. Instead, evaluations should encompass a diverse set of prompt configurations to capture a more accurate picture of model performance. Such an approach acknowledges the influence that prompt design can exert on results and seeks to account for this factor in comparative analyses. In response to these challenges, Sclar et al. [10] introduced the concept of “FormatSpread”, a methodology designed to systematically examine how variations in prompt format impact LLM outputs. FormatSpread aims to provide a more nuanced evaluation framework that includes an array of prompt formats, thereby enabling a more comprehensive understanding of model behavior and capabilities. The implications of these studies are far-reaching, as they call into question some common assumptions about LLM development and evaluation. The findings suggest that model size, while important, should not overshadow considerations of prompt design, specialization, and sensitivity. Furthermore, they highlight the need for a more sophisticated evaluation paradigm that acknowledges the complexities of language prompts and their effects on model performance. Researchers are increasingly advocating for evaluation protocols that reflect the diverse ways in which LLMs might be engaged in practical applications, thereby ensuring that performance metrics are both robust and representative of real-world usage scenarios.

To further build upon the discussion, our empirical study seeks to highlight the inherent complexity in determining the optimal use of single-task versus multitask prompts within a fixed-prompt architecture. Existing research often contrasts these approaches in specific contexts, suggesting that single-task prompts may offer precision and focus when handling distinct tasks, while multitask prompts provide efficiency and flexibility, especially in environments where models need to generalize across a wide array of related tasks without additional tuning. However, there is no definitive guideline to suggest that one method consistently outperforms the other across all scenarios. Our study aims to empirically demonstrate that the decision between single-task and multitask prompts cannot be universally dictated by a single criterion or strategy. In fixed-prompt architectures, the effectiveness of these approaches is highly dependent on factors such as the complexity of the tasks, the domain specificity, and the nature of the input data. We contend that instead of searching for a one-size-fits-all rule, researchers and practitioners should adopt a context-sensitive strategy, evaluating the trade-offs between precision and generalization based on the specific requirements of the task at hand. This nuanced perspective aligns with the broader trend in prompt engineering research, which emphasizes the importance of tailoring prompts to the unique characteristics of the target application, thus challenging the notion that a universal solution exists for optimizing prompt performance within Large Language Models.

3. The Experiment

The objective of our experiment is to generate a JSON output for each observation in the dataset, which will be directly compared to the ground truth JSON. The dataset holds movies reviews written by users of the web. The ground truth JSON includes the sentiment label, named entities extracted from the review, and the original review text extracted from the IMDB dataset (as shown in Listing A1). To achieve this, we employed two distinct approaches: a single-task approach and a multitask approach, as shown in the Figure 1, below. These methodologies were designed to test the efficiency and accuracy of different prompt strategies for Large Language Models in handling multiple tasks simultaneously versus individually.

The two experimental workflows provide a comprehensive framework for evaluating the efficiency and precision of single-task versus multitask prompts. The outputs from both approaches were systematically compared against the ground truth JSON to determine the following key metrics: accuracy, which involves the correctness of sentiment classification and named entity recognition in both single-task and multitask scenarios; and consistency, which refers to the consistency of JSON formatting and structure across different prompt strategies.

3.1. The Dataset

In this study, we constructed our dataset using the IMDB review dataset available on Kaggle [11], a well-established dataset commonly employed in sentiment analysis research. The IMDB dataset contains movie reviews, each labeled with binary sentiment values indicating whether the sentiment is positive or negative. This dataset forms the basis of our comparative analysis, which focuses on evaluating the performance of prompt strategies for Large Language Models (LLMs) in both sentiment analysis and named entity recognition (NER) tasks. Given the large size of the original IMDB dataset, we performed a random sampling to select 1000 reviews, ensuring a manageable yet representative subset for our experiments. To enrich the dataset with named entity information, we utilized the SpaCy library, a leading tool in NLP for English language tasks [12]. We selected SpaCy’s transformer-based model, en_core_web_trf, known for its ability to capture nuanced contextual information and accurately identify entities. For each review, we applied SpaCy’s NER function to extract named entities, specifically targeting categories such as persons (PER), locations (LOC), and organizations (ORG). This extraction process involved tokenizing the review text, identifying potential entities using the transformer architecture, and classifying them into relevant categories. Following NER processing, we augmented the dataset by introducing an additional column labeled “entities”. This column contains a structured list of the extracted entities for each review, organized in a dictionary format with the entity type (e.g., PER, LOC, ORG) and the corresponding text identified by the SpaCy model. This structured representation facilitates further analysis and allows us to incorporate entity information seamlessly into our experimental framework.

The final dataset is composed of three primary components: the original review text, the sentiment label provided by the IMDB dataset, and the named entities extracted through SpaCy. To better clarify the dataset structure, we show an example in the Table 1 below. We organized this information into a JSON format, where each review is represented by three keys: “review” for the original text, “sentiment” for the sentiment label, and “entities” for the list of named entities. This JSON structure supports a comprehensive examination of LLM performance across both single-task and multitask prompts, ensuring a rigorous and systematic evaluation of the models’ accuracy and effectiveness. By leveraging the IMDB dataset’s extensive sentiment annotations and supplementing it with high-quality named entity data, we created a multifaceted testing ground to assess the efficacy of different prompting strategies for LLMs. This dual-task setup not only provides a clear benchmark for comparative performance but also highlights the practical applications of LLMs in real-world scenarios that require the integration of multiple NLP tasks.

Justification for IMDB Dataset Selection

Our choice to employ the IMDB dataset is grounded in its suitability for studying the capabilities of LLMs in realistic and diverse linguistic environments. The dataset is composed of user-generated movie reviews, which inherently vary in writing style, linguistic complexity, and formal accuracy. This variety introduces challenges that are essential for evaluating the robustness of LLMs, as user-generated content often includes informal expressions, grammatical errors, and unconventional syntactic structures—features that are less prevalent in professionally curated texts. The IMDB dataset’s broad adoption in the NLP research community, particularly in sentiment analysis studies, provides a well-defined benchmark for experimental comparison. Its widespread use enables us to position our findings within the context of the existing literature, facilitating a reliable assessment of the efficacy of different prompting strategies. Furthermore, the dataset’s focus on movie reviews—texts that often combine objective descriptions with subjective opinions—makes it an excellent resource for tasks requiring both sentiment analysis and NER. By selecting a dataset that closely mirrors the types of language encountered in everyday user interactions, we aim to ensure the relevance and applicability of our study. This choice allows us to evaluate the performance of LLMs in handling complex, real-world linguistic inputs, contributing to a deeper understanding of how these models manage the intricacies of unstructured and diverse language.

3.2. The Prompt Architecture

The experiment workflow is meant to compare the single-task prompt approach with the multitask prompt approach. We employed a fixed-prompt architecture with the aim to provide a sort of model-agnostic usage. The use of a fixed-prompt architecture in prompt engineering is driven by several factors that enhance the rigor and efficiency of comparative analysis. A fixed set of prompts allows for a fair comparison across different models, as it maintains a consistent input structure. This consistency ensures that the effects observed are due to the models’ architecture and capabilities, rather than variations in prompt formulation. In doing so, potential biases from tailored prompts for individual tasks or models are avoided, leading to a more accurate attribution of performance differences. Another advantage of the fixed-prompt approach is its ability to reduce both complexity and computational costs. By removing the need to optimize prompts for each specific task, the development and testing process becomes more straightforward. This not only saves time but also conserves computational resources, as prompt tuning is often an iterative and resource-intensive endeavor. Furthermore, a fixed-prompt strategy facilitates the assessment of a model’s generalization and robustness. Using a standardized set of prompts across tasks allows for a clear evaluation of how well a model generalizes, making comparisons between multitask and single-task approaches under uniform conditions possible. This serves as a robust test of a model’s ability to handle generic, non-optimized inputs. Scalability and ease of maintenance are also supported by this architecture. A uniform set of prompts makes system updates or modifications less complex, since changes do not require re-optimizing prompts for individual tasks. This scalability is particularly advantageous in large-scale applications, where the number of tasks may increase over time, allowing systems to expand without significant increases in maintenance overhead. The choice of a fixed-prompt architecture also enhances reproducibility, which is crucial for academic research. A consistent input context increases the likelihood that results are replicable, providing a stable baseline for evaluating the benefits of adaptive or optimized prompts. This stability allows for a quantitative assessment of prompt optimization by comparing results to a standardized reference point.

The implications of adopting a fixed-prompt architecture extend to both research and practical applications. In research, it enables more reliable benchmarking across different models, making comparisons more meaningful. Researchers can identify genuine performance differences attributable to the models themselves, rather than adjustments in prompt design, leading to a clearer understanding of the strengths and limitations of various architectures, particularly in multitask versus single-task settings. This approach also places a strong emphasis on generalization. Evaluating models under a consistent set of prompts highlights their ability to handle diverse tasks, pushing the field towards the development of architectures that perform well across a wide range of contexts without extensive adaptation. This could encourage the creation of more versatile and generalizable AI systems, as opposed to those relying on domain-specific prompt optimization. In practical applications, the fixed-prompt strategy significantly impacts scalability. In scenarios where the number of tasks or domains grows over time, using a single set of prompts allows systems to expand efficiently, making large-scale deployment more feasible and cost-effective. This is particularly valuable in commercial settings, where operational efficiency and scalability are key considerations. From a research methodology perspective, a fixed-prompt architecture establishes a solid baseline, enhancing the reproducibility of studies by providing a clear and invariant prompt set. This promotes rigorous scientific discourse, allowing new methods to be evaluated against well-defined standards. However, this approach places significant importance on the initial prompt selection. The chosen prompts must be representative and carefully designed to ensure fair assessment across a variety of tasks. If the prompts are poorly selected, they might either obscure a model’s strengths or exaggerate its weaknesses, potentially leading to misleading conclusions.

3.2.1. The Single-Task Approach

In the single-task prompt approach, we decoupled the tasks of sentiment classification, named entity recognition (NER), and JSON formatting into separate, distinct operations. The goal was to isolate each task to determine how effectively Large Language Models can perform when given a dedicated prompt for each task. The process began with sentiment classification. We first used a single-task prompt to classify the binary sentiment of each review. The Large Language Models were prompted to read the review text and determine whether the sentiment was positive or negative. All 1000 elements in our dataset were processed independently, with each review being fed into the LLM, and a sentiment label (either “positive” or “negative”) was generated for each review. This operation produced an initial output consisting solely of sentiment classifications. Following sentiment classification, a separate prompt was employed to extract named entities from the reviews. The entities included people (PER), locations (LOC), and organizations (ORG). Each review was again processed individually through the Large Language Models, this time with a focus on identifying and classifying named entities. The output of this step was a collection of lists containing the extracted entities for each review. The final step involved formatting the results into a structured JSON format using the outputs from the first two tasks. The sentiment labels and named entities were combined with the original review text to create a JSON object for each review. Each JSON object included keys for “review”, “sentiment”, and “entities”, matching the structure of the ground truth JSON. This step ensured that the outputs were aligned with the expected format for comparison. A clarifying UML activity diagram of the approach is displayed below in Figure 2.

The used single-task prompts are shown in Appendix Listing A3. Single-task prompts were designed to create a chain of invocations where the outputs of two calls are used as input context for the final prompt. The first two prompts performed named entity recognition (NER) and sentiment analysis classification tasks. The output of these two tasks then merged and was used as input for the third prompt, whose goal is to reorganize the obtained information, producing a comprehensive JSON output.

3.2.2. The Multitask Approach

The multitask prompt approach was designed to evaluate the performance of Large Language Models when tasked with performing multiple tasks simultaneously. In this method, a single prompt was used to instruct the Large Language Models to carry out sentiment classification, named entity recognition, and JSON formatting in one unified operation. The detailed prompt is shown in Appendix Listing A2.

In this approach, we used a unified prompting strategy where the Large Language Models were provided with a single, comprehensive prompt for each review. To better clarify the workflow, we provide an UML activity diagram shown in Figure 3. This prompt instructed them to analyze the sentiment, extract named entities, and format the results into a JSON object. All 1000 reviews were processed in a batch manner, with each review being fed into the LLM with a multitask prompt designed to handle all three tasks at once. The output for each review was a complete JSON object containing the sentiment classification, extracted entities, and the review text itself. By handling sentiment analysis, named entity recognition, and JSON formatting in parallel, the multitask approach leverages the Large Language Models’ ability to process complex, integrated prompts. The final output for each review was a JSON object directly generated by the LLM, structured similarly to the ground truth JSON. This direct approach allows for the assessment of the Large Language Models’ capability to multitask effectively and efficiently.

3.3. Execution Environment

The experiment was executed in Jupyter notebooks, which are publicly available on our GitHub repository [13]. This decision aligns with our commitment to transparency and reproducibility, allowing other researchers to access and replicate our findings with ease. The Jupyter notebooks provide a detailed step-by-step account of the entire experimental workflow, including data preprocessing, prompt formulation, and LLM interactions. By using Jupyter notebooks, we ensure that each experiment is documented in a manner that captures the nuances of our methodology, from data ingestion to output generation. These notebooks not only contain the code used for executing each task but also include commentary and insights into the decisions made throughout the experiment. This transparency is crucial for fostering collaboration and innovation within the research community. In addition to the code, the dataset utilized in our study is also hosted on the same GitHub repository. The dataset includes the processed IMDB reviews, along with the generated JSON outputs for both single-task and multitask approaches. By providing both the data and the code, we facilitate an open-access environment where researchers can easily validate and build upon our work. Our GitHub repository serves as a comprehensive resource for those interested in exploring the intricacies of LLM prompt strategies. It invites further exploration and experimentation, offering a platform for continuous improvement and shared learning within the field of natural language processing.

The Temperature

In our study, we deliberately set a fixed temperature of 0.8 for all models. This decision was grounded in the need to maintain a balance between precision and creativity in the outputs. A temperature setting of 0.8 was chosen because it avoids the extremes of determinism and randomness. Lower temperatures, closer to zero, tend to produce highly deterministic responses, reducing variability and creativity, which may inhibit the model’s ability to explore diverse but relevant outputs. Conversely, higher temperatures might lead to overly creative or tangential responses, increasing the likelihood of generating irrelevant or imaginative content that could obscure the evaluation of the models’ true capabilities. By fixing the temperature at this intermediate level, our goal was to create conditions that would allow the models to demonstrate their generalization abilities more effectively. The selected temperature encourages the generation of outputs that are varied enough to showcase the models’ adaptability across tasks, yet sufficiently constrained to remain grounded in relevant contexts. This configuration emphasizes the models’ inherent capacity to manage diverse scenarios without relying on extreme prompt tuning or specialized adjustments.

The decision to keep a consistent temperature across all experiments ensures a fair and controlled environment for comparative analysis. It eliminates the variability that might arise from fluctuating temperature settings, thereby allowing us to attribute differences in performance directly to the models’ architectures and underlying mechanisms, rather than to variations in sampling strategies. This standardized approach not only enhances the reliability of our findings but also facilitates a deeper exploration of each model’s potential to generalize effectively across multiple tasks without excessive prompting or guidance. Thus, the fixed temperature of 0.8 was not merely a technical choice, but a strategic decision aimed at fostering a thorough assessment of each model’s robustness and flexibility, while maintaining a consistent baseline that allows for equitable comparisons. This enabled us to investigate how different architectures handle the inherent complexity and diversity of the prompts, offering insights into their adaptability and generalization capabilities under a balanced but challenging setting.

3.4. The Employed Large Language Models

In this study, we focused on evaluating the performance of five different open Large Language Models (Large Language Models) for sentiment analysis and named entity recognition tasks.

The term “open” when referring to Large Language Models, as highlighted by Liesenfeld and Dingemanse [14], encompasses significant differences across various levels of “openness” in AI models, which can be categorized as follows:

• Open-source: The model’s source code is fully accessible and modifiable.

• Open-weights: The trained model weights are available, but possibly with licensing restrictions.

• Open-access: The model can be accessed via API, but without access to the code or weights.

• Open-science: The research methodology and results are publicly documented.

In the context of our study, the models used are primarily classified as “open-weights”, as their weights are publicly available and redistributable, though with some restrictions specified in their licenses. It is important to note that this level of openness, while significant for research reproducibility, does not necessarily equate to full freedom of use and modification, as would be found in a fully open-source project. Furthermore, as Bielefeld and Dingemanse emphasized, the simple label of “open” can obscure various degrees of restrictions and limitations, which are crucial to understand for an accurate assessment of accessibility and reproducibility in AI research.

The primary motivation behind selecting open-weight models was to enhance the reproducibility of our research. By choosing models that are accessible to the public, we ensure that other researchers can replicate our experiments, verify our findings, and build upon our work without the constraints often associated with proprietary models. While state-of-the-art Large Language Models such as OpenAI’s GPT, Anthropic’s Claude, and Google’s Gemini have demonstrated remarkable performance across a wide range of language tasks, their proprietary nature poses challenges for academic research in terms of accessibility and transparency. These models often come with usage restrictions, limited customization options, and require considerable computational resources, which can hinder reproducibility and broader scientific exploration. Therefore, to foster an open and collaborative research environment, we selected a set of high-performing open-weight models that provide a balance between accessibility and capability. The specific Large Language Models we employed in our study are as follows:

1. LLama 3.1 8B (8b-instruct-q4_0).

2. Phi3 Medium (14b-medium-128k-instruct-q4_0).

3. Qwen2 7B (7b-instruct-q4_0).

4. Gemma2 9B (9b-instruct-q4_0).

5. Mistral 7B (7b-instruct-v0.3-q4_0).

3.4.1. Justification for Employed Large Language Models

The selection of these specific open-weight models was guided by several factors. Using open-weighted models, we enhanced the reproducibility of our research, enabling other researchers to replicate our experiments and validate our findings without proprietary restrictions. Open-weight models are generally more accessible, allowing a wider range of researchers and practitioners to engage with the research, regardless of their institutional or financial resources. Although not in absolute state of the art, the chosen models still offer competitive performance in sentiment analysis and named entity recognition tasks, providing meaningful insight into LLM capabilities. Open-weight models benefit from active community participation, which leads to continuous improvements and innovations. This collaborative environment fosters the development of robust models that evolve in response to community needs and feedback. The selected models have varying parameter sizes that allow for experimentation on different computational platforms, from local machines to more extensive cloud-based infrastructures, facilitating scalable research approaches. In order to determine the five candidates, we observed the “Open LLM Leaderboard 2” hosted on Huggingface that lists the open-weight LLM performances [15,16,17,18,19,20,21,22,23]. We accessed this leaderboard in June 2024. By focusing on open-weight Large Language Models, this study not only provides valuable insights into the effectiveness of prompt strategies, but also contributes to a body of work that prioritizes transparency and accessibility in AI research. This approach aligns with the broader goals of promoting open science and collaborative innovation within the machine learning community.

In this study, the decision to utilize a smaller model, rather than opting for larger and potentially more capable alternatives, was driven by several key considerations. Firstly, smaller models offer substantial advantages in terms of resource efficiency. They require less computational power, memory, and storage, leading to reduced costs and time investment during the experimental phase. This is particularly pertinent when a high volume of experiments must be conducted or when computational resources are constrained. Additionally, the complexity of the tasks under investigation, which focused on single-classification operations, did not necessitate the deployment of the most sophisticated or extensive models. The relative simplicity of single-classification tasks can often be effectively managed by smaller models without significant losses in accuracy, thereby making their use more practical. Furthermore, the primary objective of this study was to explore the impact of different prompt strategies, specifically comparing single-task prompts with multitask prompts, rather than to assess the raw capabilities of large-scale models. Employing smaller models allowed for the isolation of prompt strategy effects, minimizing the potential influence of a model’s inherent complexity on the observed outcomes.

The study’s exclusive focus on single-classification tasks was similarly motivated by the need for a controlled and precise evaluation framework. Single-task classification offers a streamlined environment for analyzing how prompt modifications influence model performance, eliminating the complexities and additional variables that multitask scenarios would introduce. This simplicity enhances the reproducibility and comparability of results across various models and prompt strategies, allowing for clearer and more reliable insights. Moreover, single-classification tasks often serve as foundational elements within more complex natural language processing tasks. By initially concentrating on these simpler operations, the study establishes a robust baseline for understanding the dynamics of prompt strategies, which could potentially be extended to more intricate multitasking assessments in future research. In sum, the decision to employ smaller models and to focus on single-task classification facilitated a research environment in which the effects of prompt strategies could be isolated and scrutinized with greater clarity.

3.4.2. The Adoption of Ollama

Ollama is an open-source framework developed to facilitate the local deployment and execution of Large Language Models (LLMs). In our experiments, Ollama played a pivotal role, serving as the platform for deploying the five chosen LLMs: LLama 3.1 7B, Phi3 Medium, Qwen2 7B, Gemma2 7B, and Mistral 7B. Its selection was driven by its user-friendly interface, ease of use, and strong alignment with our commitment to reproducibility, which is central to our research objectives. By employing Ollama, we ensured that our experimental setup remains accessible and easily replicable, fostering transparency and collaboration within the research community.

Ollama’s architecture is built upon a REST API, which simplifies model interactions through automated management processes. The framework automatically checks the local availability of requested models; if a model is not found, Ollama alerts the user and facilitates a direct download. After acquisition, the framework autonomously manages the configuration process, optimizing the model according to the available hardware. These steps encompass the extraction of model files, the fine-tuning of memory resources through quantization, and the balancing of computational loads across the RAM, CPU, and GPU. Additionally, Ollama integrates robust error management, verifying the integrity of downloaded files and recording any discrepancies to aid debugging. This architectural design ensures compatibility across major operating systems, making Ollama a versatile and accessible choice for both researchers and practitioners. In our study, this cross-platform compatibility was crucial, as it enabled a consistent and reliable setup regardless of the specific hardware used by different members of the research community.

A core feature of Ollama is its implementation of quantization, a technique that significantly reduces the numerical precision of model parameters while maintaining acceptable levels of accuracy. This reduction involves converting high-precision data types like float32 (32-bit) to more compact formats such as int8 (8-bit) or int4 (4-bit). Quantization drastically lowers memory consumption, a vital advantage for the local execution of large models. For example, a seven billion parameter model in float32 format may require up to 28 GB of RAM, whereas the int8 version requires only about 7 GB. This efficiency was particularly advantageous in our experiments, as it allowed us to run multiple high-parameter models locally without compromising performance, facilitating efficient and reproducible testing.

Ollama utilizes a standardized inference pipeline, structured into four primary phases, to optimize the execution of LLMs as shown in Figure 4. This structured approach was instrumental in our study, ensuring consistent and high-quality output across various models. The first phase, preprocessing, involves the preparation of input data. Text is tokenized into discrete units that the model can process, a context is constructed to guide comprehension, and memory resources are dynamically allocated. This ensures that each model receives input in a format conducive to accurate processing. In the execution phase, the framework loads the quantized model into memory and sequentially processes tokens. Effective cache management during this phase minimizes latency by reusing previously computed results, optimizing repeated tasks. The generation phase is dedicated to transforming processed tokens back into human-readable text. Advanced sampling techniques, such as beam search and nucleus sampling, are employed to enhance the quality of generated output, while incremental generation enables real-time responses. This step-by-step text generation was particularly useful in our research, as it allowed us to monitor model outputs in real time, facilitating adjustments where necessary. Finally, the postprocessing phase involves refining the generated output. Text is formatted into a coherent structure, memory resources are released, and the final output is prepared in JSON format for exposure via the API. This systematic approach, coupled with Ollama’s user-friendly interface, allowed us to maintain a reproducible and transparent workflow throughout our experiments.

Our choice of Ollama was not merely technical but also philosophical. The platform’s straightforward installation and configuration allowed us to maintain a high level of accessibility, ensuring that others can easily replicate our setup without extensive technical expertise. Its active support for a broad range of open-weight models, including those integral to our research, was another decisive factor. By enabling a seamless integration of diverse LLMs, Ollama ensured consistent execution across different experimental scenarios, contributing to the robustness of our findings. By leveraging Ollama, we adhered to our commitment to open science. The platform’s widespread usability means that any researcher interested in our work can replicate our environment and methodology with minimal effort. This accessibility not only bolsters the credibility of our results but also encourages further validation and exploration, thereby fostering a collaborative research ecosystem. Our decision to use Ollama aligns with a broader movement towards transparent and reproducible science, contributing to a culture of open inquiry and shared discovery.

3.5. Evaluation

In order to rigorously assess the performance of the Large Language Models (Large Language Models) employed in our study, we designed a comprehensive evaluation metric that integrates three critical components: sentiment exact match, named entity recognition (NER) performance, and review text fidelity. Our evaluation metric is designed to provide a holistic measure of how effectively the Large Language Models handle the tasks of sentiment classification, named entity extraction, and text reproduction. The evaluation metric is defined as follows:

$f(x,y,z)={\displaystyle \frac{x+y+z}{3}}$

where x is the exact match of sentiment detection, defined as follows:

$x=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\mathrm{the}\mathrm{detected}\mathrm{sentiment}\mathrm{is}\mathrm{correct}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$

Then, y represents the F1 score of the named entity recognition (NER). True positives are entities present in both the ground truth and the output. False negatives refer to those missing from the output but included in the ground truth, while false positives are found in the output but absent from the ground truth.

This score reflects the precision and recall balance achieved by the Large Language Models in identifying and classifying named entities within the review text. The “exact match” approach has been used here. z is the BLEU (Bilingual Evaluation Understudy) score of the review, which measures how closely the generated review matches the original text. This component of the metric evaluates the LLM’s ability to reproduce the review text accurately. The function $f(x,y,z)$ computes the arithmetic mean of x, y, and z, providing an overall performance score for each review processed by the Large Language Models. By averaging these three metrics, we achieve a balanced evaluation that considers both classification accuracy and text generation quality. The rationale behind this metric is to ensure that each aspect of the task is equally weighted, acknowledging the importance of sentiment exact match, entity recognition precision, and fidelity to the original review text. This holistic approach allows us to capture the multifaceted nature of the task, offering a robust framework for evaluating the effectiveness of different prompting strategies in Large Language Models.

The LLM’s output is postprocessed in order to extract the right JSON data. This is due to the fact that Large Language Models often respond discoursively without providing strict JSON. Two distincts regular expressions are used in this scenario. The former one, mentioned in Appendix Listing A4 is used to remove JSON comments. Sometimes, Large Language Models follow the JSONC format for their answers instead of JSON, so we need to remove all the comments in order to parse the data correctly. Then, the second one, mentioned in Appendix Listing A5, is a regular expression used to extract the JSON string. Ultimately, it extracts the group starting from the first left curly bracket and ending with the last right curly bracket. Moreover, Large Language Models tend to respond with a JSON that uses single apices instead of double quotes. So, in a pythonic manner, the algorithm works as shown in Appendix Listing A6, assuming that the two regular expressions have already been applied. When the parsing fails, we attribute a score of 0% to that case.

4. Results

The analysis of our experimental results provides a nuanced perspective on the efficacy of multitask versus atomic single-task prompts. Contrary to our initial hypothesis, the data reveal that the atomic single-task prompt approach does not uniformly outperform a multitask prompt across all contexts. Our study highlights significant variability in prompt effectiveness depending on the specific model used. This observation suggests that the interaction between prompt type and model architecture is complex and warrants careful consideration. Specifically, the performance of a given prompt can be highly sensitive to the underlying model’s characteristics, indicating that model-specific factors play a crucial role in determining the relative success of prompting strategies. In detail, our experiments yielded mixed outcomes. Out of the five distinct experimental setups, three demonstrated that atomic single-task prompts were more effective than their multitask counterparts. These results suggest that for certain tasks, simpler and more specialized prompts may offer advantages in terms of accuracy or efficiency. Conversely, two experiments showed that multitask prompts provided superior performance, challenging the assumption that simplicity inherently leads to better outcomes. This variability underscores the importance of tailoring the prompting approach to the specific task and model, rather than relying on a one-size-fits-all strategy.

Furthermore, the unexpected nature of our findings is worth noting. Despite the theoretical benefits of atomic single-task prompts such as the potential for improved efficiency and generalization, the empirical evidence from our study does not consistently support these advantages. We had anticipated that the low complexity associated with single-task prompts would correlate with enhanced performance. However, the results indicate that this expectation does not always hold true in practice. The complexity of single-task prompts did not translate into universally superior outcomes compared to the relatively straightforward multitask prompts. Additionally, our investigation included a range of models with varying sizes, from 2 billion to 14 billion parameters. The results from these experiments did not reveal a clear relationship between model size and prompt effectiveness. This finding suggests that the performance of prompting strategies is not solely dependent on the scale of the model but is influenced by other factors, such as task characteristics and prompt design. The Table 2 below shows the mean scores for each model and approach, where “scores” mean the metric explained in Section 3.

4.1. Stastical Significance

In this section, we present a series of statistical tests conducted to provide additional evidence supporting our findings. First, we performed the Shapiro–Wilk test to assess whether the distributions of our data were normal. The results clearly indicate that all distributions deviate significantly from normality, as reflected by the extremely low p-values reported in Table 3. These p-values, consistently below conventional thresholds (e.g., 0.05), strongly suggest that none of the distributions for the evaluated metrics follow a normal pattern.

Given the non-normal nature of the data, we employed the Wilcoxon signed-rank test for continuous metrics such as F1, BLEU, and the overall score. For the categorical metric, specifically the Exact-Match score for Sentiment (which can take a binary value of 0 or 1 depending on whether the output matches the ground truth), we opted for the McNemar test. The decision to use these non-parametric tests ensures that our analysis remains statistically valid despite the deviations from normality. The results are summarized in Table 4.

Expanding upon the results, the findings provide valuable insights into how different models react to single-task versus multitask prompting strategies across a variety of evaluation metrics. Each metric presents a distinct pattern of sensitivity, indicating that the effectiveness of the prompting approaches is not uniform across tasks or models, highlighting the nuanced behavior of Large Language Models.

For the NER F1 metric, the Wilcoxon test identified significant differences in most models, suggesting that the choice between single-task and multitask prompts can substantially influence the model’s ability to recognize named entities. However, the exception of LLama 3.1 8B, with a p-value just above the conventional threshold of 0.05, indicates a borderline case where the prompting approach may have an impact, but not to a statistically significant extent. This borderline result suggests that LLama 3.1 8B might be relatively resilient to changes in prompt structure for NER tasks, or that its response to such changes lies within a range of performance variability where differences do not reach statistical significance. This observation could warrant further investigation, possibly involving larger sample sizes or additional metrics, to clarify if this lack of significance is inherent to the model or a result of sample-specific variability.

In the case of the BLEU score, which is a critical measure for assessing the quality of generated text in review scenarios, the Wilcoxon test demonstrated overwhelmingly significant results for most models, with Phi3 Medium and Qwen 2 7B exhibiting particularly strong effects. The extremely low p-values suggest that these models are highly sensitive to the structure of the prompt when tasked with generating coherent and accurate text. This sensitivity implies that for text generation tasks, selecting the appropriate prompting strategy is crucial, as it can lead to substantial differences in the quality of the generated output. The stark significance across multiple models also suggests that multitask prompting could provide a structured context that enhances generative quality, particularly for complex or multi-dimensional output like reviews.

The results for the Sentiment Score, analyzed using the McNemar test, present a more heterogeneous picture. Only Mistral 7B exhibited a strong statistically significant result, suggesting that it is particularly responsive to the choice of the prompting strategy when it comes to binary sentiment classification. In contrast, the other models showed p-values around or above the 0.05 threshold, indicating a weaker or non-significant differentiation between single-task and multitask prompts in sentiment tasks. This outcome implies that for many models, sentiment classification may be relatively stable across prompting strategies, or that the gains from multitask prompting are not as pronounced in this binary classification context. It is possible that the simplicity of binary sentiment decisions, compared to more nuanced tasks like named entity recognition or text generation, reduces the sensitivity to prompt changes, particularly in models that are already well tuned for such binary classification tasks.

Lastly, the overall Score metric, which synthesizes multiple aspects of performance, reinforces the impact of prompting strategies, as indicated by the statistically significant differences observed for the majority of models. These results suggest that across a broad spectrum of metrics, the choice of prompt affects model performance, underscoring the importance of prompt engineering in optimizing LLM outputs. However, the lack of significance for Mistral 7B, as demonstrated by a p-value of 0.60, suggests a robustness in its performance, regardless of the prompting strategy. This consistency may indicate that Mistral 7B has a more uniform handling of information across different prompts, potentially due to architectural factors or training data characteristics that enable it to maintain stable performance. It could also reflect an inherent stability in how Mistral 7B processes diverse inputs, making it less prone to fluctuations in performance based on prompt structuring.

Statistical Tests to Compare Large Language Models

To compare the performance of the various models across different metrics and prompting conditions, we employed the Friedman test, a non-parametric statistical test suitable for analyzing data of repeated measures. This choice was motivated by the need to assess multiple models under the same conditions while avoiding assumptions of normality, which do not hold for our data distributions. The Friedman test allowed us to determine whether there were statistically significant differences between the models’ performances in both single-task and multitask scenarios, providing a robust comparison framework that accounts for the inherent dependencies in our experimental design.

Based on the Friedman test results shown in Table 5, it is clear that significant differences exist between the performance of the evaluated models across all metrics, under both single-task and multitask prompting conditions. Each metric consistently demonstrates p-values that are orders of magnitude below typical thresholds for statistical significance (e.g., 0.05), indicating that the models exhibit distinct behaviors depending on the prompting strategy used. This consistent pattern of significance underscores the critical role that prompt structuring plays in shaping the output quality of Large Language Models.

The F1 score for named entity recognition (NER) reveals substantial model variability, both under single-task and multitask conditions. The Friedman statistic for single-task prompting is 1030.82, with a corresponding p-value of $7.49\times {10}^{-222}$. This already indicates a strong differentiation among the models. In the multitask setup, the statistic slightly increases to 1050.92, with an even lower p-value of $3.29\times {10}^{-226}$, reinforcing that the multitask strategy introduces subtle, yet statistically detectable, shifts in model performance.

The BLEU score, used for evaluating the quality of text generation in review scenarios, shows a notable significance across the models. In the single-task scenario, the Friedman statistic is 1002.16, with a p-value of $1.21\times {10}^{-215}$, suggesting strong differences in text generation abilities when the models are prompted individually. The multitask configuration amplifies this distinction dramatically, with the statistic reaching 1970.88 and a p-value effectively equal to 0, indicating extremely robust model variability under multitask prompts.

Sentiment classification, measured by the Exact-Match score, also displays clear differences, although the statistics are relatively lower compared to F1 and BLEU scores. The Friedman test for single-task conditions yields a statistic of 655.21 and a p-value of $1.73\times {10}^{-140}$, which is still well within the range of significance. Under multitask conditions, the statistic increases to 792.74, with a corresponding p-value of $2.87\times {10}^{-170}$, suggesting that multitask prompting potentially offers a more nuanced differentiation between the models, even in simpler binary classification tasks.

The overall Score metric, synthesizing various aspects of model performance, presents the most pronounced differences. The Friedman statistic under single-task conditions is 1310.58, with an extremely low p-value of $1.69\times {10}^{-282}$, indicating that single-task prompts already cause marked differences in overall performance. This differentiation becomes even more pronounced in the multitask scenario, where the statistic soars to 1808.26 with a p-value of 0, highlighting a profound impact of multitask prompts on overall model effectiveness.

The statistical evidence gathered from this study confirms that different prompting strategies have a tangible impact on model behavior across various evaluation metrics. This underscores the importance of carefully selecting prompting techniques based on the desired outcome. While single-task prompts provide a baseline for evaluating individual task performance, multitask prompting introduces additional complexity that can either enhance or differentiate model outputs more significantly, depending on the metric and task at hand. Future research should explore why certain models react more sensitively to prompt changes, potentially investigating architectural factors or training data characteristics that influence prompt responsiveness. Additionally, expanding the evaluation to include more diverse tasks and model types would offer further insights into the generalizability of these findings.

4.2. Descriptive Statistics Related to Model Performance

4.2.1. Gemma 2 9B

Gemma 2 9B outperformed all other models in this study. Although the assumption that single-task prompts yield better results compared to the dual, multitask approach still holds true, it is worth noting that the difference in performance is not particularly pronounced. Shifting the focus to individual tasks, the table below provides a detailed breakdown of the results.

The data presented in the Table 6 highlight the comparative performance of the multitask and single-task approaches for the Gemma 2 9B model across various metrics. The single-task approach demonstrates a slight edge in certain areas, such as Exact-Match accuracy on sentiment analysis, achieving 91.50% compared to 90.00% for the multitask approach, and in the NER F1 score, with a small improvement from 54.75% to 55.75%.

However, the multitask strategy shows advantages in NER Precision, scoring 60.99% versus 59.86% for the single-task prompts, suggesting that the multitask model may be more accurate in identifying named entities, though it comes at the cost of a slightly lower NER Recall (54.11% for multitask versus 56.87% for single-task).

Additionally, the formatting error rate is similar across both approaches, with a marginal difference of 1 per thousand (9.00‰for multitask versus 8.00‰for single-task), indicating that the complexity of the prompt does not significantly impact the model’s ability to maintain proper formatting. These data suggest that while single-task prompts may offer a slight performance benefit in certain aspects, the differences are generally minor, and multitask prompting retains certain advantages, particularly in precision.

4.2.2. Qwen 2 7B

For Qwen 2 7B, the assumption that single-task prompts yield better results compared to the dual, multitask approach remains valid. However, in contrast to Gemma 2, the observed pattern is more erratic, and the performance gap between the two approaches becomes more pronounced in this case, as shown in Table 7.

Table 7 highlights a more significant divergence between the multitask and single-task approaches for Qwen 2 7B compared to Gemma 2 9B. Single-task prompts outperform multitask prompts across most metrics, particularly in the Mean BLEU score on review tasks (73.26% vs. 56.09%), indicating a substantial advantage in generating coherent and accurate text for single-task prompts. Similarly, the Exact-Match score for sentiment analysis is higher for the single-task approach (82.70% vs. 80.80%).

However, an interesting deviation can be observed in NER Precision, where multitask prompts demonstrate better performance (32.20% vs. 27.97%), suggesting that Qwen 2 7B’s ability to precisely recognize named entities benefits from the complexity of the multitask setup. Despite this, single-task prompts yield higher NER Recall (30.79% vs. 24.32%), reflecting a trade-off between precision and recall similar to what was observed in the previous model. Additionally, the formatting error rate is notably higher for single-task prompts (88.00‰vs. 34.00‰), suggesting that while single-task prompts may improve content accuracy, they introduce a greater risk of formatting errors, a factor worth considering in practical applications.

4.2.3. LLama 3.1 8B

LLama 3.1 8B is the first model to deviate from the expected pattern. In contrast to previous models, the multitask approach outperforms the single-task approach, demonstrating a clear reversal of the trends observed earlier. LLama 3.1 8B performances are described in Table 8.

Table 8 reveals a distinctive performance pattern for LLama 3.1 8B, where the multitask approach shows superior results compared to the single-task approach across most metrics. Notably, the Mean BLEU score on review tasks is significantly higher for multitask prompts (88.94% vs. 76.55%), indicating that LLama 3.1 8B generates more coherent and contextually accurate responses when handling multiple tasks simultaneously. Similarly, in sentiment analysis, the multitask approach outperforms the single-task approach with a higher Exact-Match score (83.70% vs. 81.00%). However, the NER metrics present a more nuanced picture. While the single-task approach achieves a slightly higher F1 score (44.10% vs. 43.00%) and NER Recall (46.01% vs. 42.05%), multitask prompts excel in NER Precision (50.25% vs. 47.98%). This suggests that LLama 3.1 8B is more precise but slightly less comprehensive in recognizing named entities when dealing with multitask prompts. Additionally, the formatting error rate is notably lower in the multitask setting (69.00‰vs. 94.00‰), indicating that multitask prompts not only yield better content accuracy but also lead to fewer formatting errors. These results underscore the model’s capacity to handle multitask scenarios effectively, challenging the conventional assumption that single-task prompting is inherently superior.

4.2.4. Phi 3 Medium

Regarding Phi3 Medium 14B, based on its performances listed in Table 9, it can be unequivocally stated that its performance was the worst among all models in the experimental set. The difference in performance between the two prompting approaches is particularly stark, with the single-task approach significantly outperforming the multitask approach.

Table 9 illustrates that Phi3 Medium 14B exhibits the weakest overall performance across all evaluated models. The results clearly demonstrate a substantial gap between the multitask and single-task approaches, with the latter consistently outperforming the former. For instance, the Mean BLEU score on review tasks is notably higher for single-task prompts (57.63% vs. 16.98%), indicating that Phi3 Medium 14B struggles significantly with generating coherent text in multitask scenarios. Similarly, the Exact-Match score for sentiment analysis shows a slight but consistent improvement in single-task settings (50.80% vs. 48.30%).

The disparity is even more pronounced in NER tasks, where the single-task approach nearly doubles the F1 score (22.62% vs. 11.68%) and achieves higher Precision (25.45% vs. 14.49%) and Recall (23.78% vs. 11.06%). These results suggest that Phi3 Medium 14B’s ability to recognize and categorize named entities is severely hindered in multitask settings.

Moreover, both approaches exhibit high formatting error rates, with the single-task method slightly worse (307.00‰vs. 253.00‰). This suggests that, although the single-task approach improves performance in content-related tasks, both prompting methods struggle with formatting precision. These results position Phi3 Medium 14B as the least capable model in handling complex or multitask scenarios, emphasizing the limitations of this particular architecture in the context of large-scale language models.

4.2.5. Mistral 7B

The following Table 10 reflects the mixed performance of Mistral 7B across various tasks, showcasing both strengths and weaknesses depending on the task and prompting approach.

In terms of review generation, the single-task approach achieves a higher Mean BLEU score (76.41% vs. 70.84%), indicating better text generation performance for single-task prompts. However, for sentiment analysis, the multitask approach far outperforms the single-task method with a notably higher Exact-Match score (87.20% vs. 71.80%). In named entity recognition (NER), the results are more nuanced. While the single-task approach yields a slightly higher F1 score (32.56% vs. 30.60%) and Recall (33.50% vs. 28.04%), the multitask approach achieves better Precision (39.28% vs. 37.05%), highlighting a trade-off between completeness and accuracy in entity recognition. One of the most significant differences is observed in the formatting error rate, where the multitask approach significantly outperforms the single-task approach, with a much lower error rate (29.00‰vs. 155.00‰). This suggests that multitask prompts not only excel in specific tasks, such as sentiment analysis, but also lead to more reliable formatting outputs. The overall distribution of scores confirms that Mistral 7B performs better under multitask settings in certain scenarios, although single-task prompts still offer advantages in specific metrics like BLEU and NER Recall.

The experiments conducted on the five models—Gemma 2 9B, Qwen 2 7B, LLama 3.1 8B, Phi3 Medium 14B, and Mistral 7B—present a diverse and complex landscape of performance across single-task and multitask prompting approaches. While the general assumption that single-task prompts yield superior results holds true for most models, the nuances of performance reveal significant variations depending on the architecture and task type. Gemma 2 9B and Qwen 2 7B conform to expectations, with single-task prompts outperforming multitask prompts across the majority of metrics, though the difference is more pronounced in Qwen 2 7B. In contrast, LLama 3.1 8B challenges this assumption, demonstrating superior results with multitask prompts, particularly in generating coherent text and sentiment analysis, signaling that not all models adhere to a uniform performance pattern. Phi3 Medium 14B exhibited the weakest overall performance, with both single-task and multitask approaches underperforming compared to the other models, but with the single-task approach consistently outperforming the multitask one. This highlights potential limitations in the model’s architecture when handling both simple and complex tasks. Mistral 7B presents a mixed profile, with performance fluctuating between the two prompting approaches depending on the task. While single-task prompts show an advantage in text generation and NER Recall, multitask prompts excel in sentiment analysis and NER Precision, with a notably lower formatting error rate, suggesting that Mistral 7B is more versatile but less predictable. Overall, these experiments underscore the importance of selecting the appropriate prompting strategy based on the specific model and task at hand. While single-task prompts generally offer better performance, certain models like LLama 3.1 8B and Mistral 7B demonstrate the potential of multitask prompts to exceed single-task results in specific contexts. The diverse outcomes across these models suggest that optimizing prompting strategies for Large Language Models should be model-specific and task-aware, rather than guided by a one-size-fits-all approach.

5. Discussion

This study provides an in-depth comparison of single-task and multitask prompting strategies for Large Language Models (LLMs) across five models: Gemma 2 9B, Qwen 2 7B, LLama 3.1 8B, Phi3 Medium 14B, and Mistral 7B. Contrary to the initial hypothesis, the results indicate that single-task prompts do not consistently outperform multitask prompts across all scenarios. The analysis revealed that, while single-task prompts often excel in structured tasks like sentiment analysis and generate higher BLEU scores for review generation, multitask prompts have their own advantages. These include higher precision in named entity recognition (NER) tasks for certain models and a reduced frequency of formatting errors, highlighting the potential of multitask prompts for integrated tasks. This variability underscores the significant impact of model architecture on the effectiveness of prompt strategies.

The findings demonstrate that prompt strategy effectiveness is model-dependent. For instance, Gemma 2 9B performed well with both prompt strategies, showing only minor differences between single-task and multitask approaches. This suggests that more capable models, like Gemma 2 9B, are relatively unaffected by prompt complexity, maintaining stable performance regardless of prompt type. However, for models like Qwen 2 7B and Phi3 Medium 14B, single-task prompts clearly outperformed multitask prompts, particularly in review generation, where BLEU scores were notably higher with single-task prompts. This suggests that models with lower capacity or different architectures might struggle with the complexity introduced by multitask prompts; on the other hand, models such as LLama 3.1 8B and Mistral 7B showed unexpected strengths with multitask prompts. LLama 3.1 8B, in particular, achieved higher BLEU scores and exhibited better sentiment classification accuracy in multitask scenarios compared to single-task ones, suggesting that its architecture might be better suited to handling integrated prompts that require simultaneous consideration of multiple tasks. This is an indication that multitask prompts might enhance contextual understanding in models designed to manage complex dependencies within the text.

Different tasks yielded varying results depending on the prompting strategy. In sentiment analysis, single-task prompts generally offered better performance across most models, leading to higher accuracy scores. This supports the notion that simpler and more focused prompts can enhance performance in tasks that require clear-cut, binary decisions. However, in NER, multitask prompts occasionally demonstrated higher precision, particularly in LLama 3.1 8B and Mistral 7B, suggesting that integrated prompts may help in extracting contextual information relevant to entity recognition. Additionally, the multitask prompt strategy resulted in a lower error rate for JSON formatting across several models, indicating that multitask prompts might provide advantages in tasks requiring consistent output structures. This insight is critical when choosing between single-task and multitask approaches, as the task requirements and desired outcomes must be carefully matched to the strengths of each prompting strategy.

The study’s findings challenge the common assumption that single-task prompts are universally superior due to their simplicity. While they frequently lead to higher precision in certain contexts, multitask prompts have demonstrated advantages in integrating multiple outputs and maintaining structural consistency. This suggests that prompt engineering must be flexible, adapting to both the task’s complexity and the specific architecture of the LLM. Moreover, the results showed no straightforward correlation between model size and prompt effectiveness. Larger models like Phi3 Medium did not always benefit from multitask prompts, while smaller ones like LLama 3.1 8B performed unexpectedly well with integrated tasks. These observations suggest that elements beyond size, such as attention mechanisms and the training dataset’s nature, significantly influence how LLMs respond to various prompt strategies.

This study’s scope is constrained by several factors. The selected models, while diverse, represent a subset of open-weight LLMs, limiting the generalizability of findings to proprietary or cutting-edge models. Additionally, the focus was on a narrow range of NLP tasks—sentiment analysis, NER, and JSON formatting—leaving room for further exploration in more diverse and challenging scenarios like machine translation or complex reasoning tasks. Future research should aim to broaden the scope by incorporating a wider variety of tasks and models, potentially examining how specific architectural elements, such as attention head count or layer depth, influence prompt performance. It would also be valuable to investigate automated prompt optimization methods, such as reinforcement learning, to dynamically adapt prompts to maximize model efficiency. Exploring domain-specific challenges, like legal or medical texts, could provide deeper insights into the applicability of single-task versus multitask strategies in specialized contexts.

From a practical standpoint, the study’s findings provide actionable guidance for developers aiming to optimize LLM interactions. The data underscore the need for a context-sensitive approach to prompt engineering, where prompt choice is not driven by simplicity alone but by a careful assessment of the task’s requirements and the model’s characteristics. Theoretically, this study challenges the dominance of single-task prompts in the field, revealing that effective prompt strategies are highly dependent on the LLM’s architecture and the nature of the task. These findings advocate for a more nuanced approach to prompt design, moving away from one-size-fits-all solutions and towards strategies tailored to the specific context.

Statistical tests corroborated the complexity of LLM responses to different prompt types. For example, non-parametric tests like the Wilcoxon signed-rank test highlighted significant differences in BLEU and NER F1 scores between single-task and multitask prompts. Such results point to deeper, architecture-specific factors influencing prompt effectiveness, suggesting that the interplay between task complexity and model design is more intricate than previously thought. To address this, future research must delve deeper into understanding how LLM architectures process prompts. Greater clarity on the internal workings of LLMs, particularly in how they handle varied prompt structures, will be crucial for developing more reliable AI systems. This is especially relevant in high-stakes domains like healthcare or finance, where transparency and consistency in AI behavior are paramount.

6. Conclusions

This comparative study on the effectiveness of multitask versus single-task prompts for Large Language Models (LLMs) has produced nuanced and unexpected findings, challenging the initial assumption that single-task prompts would consistently outperform multitask ones.

Among the tested models, Gemma 2 9B exhibited the strongest overall performance, showing only a slight preference for single-task prompts. The difference between the two prompting strategies was marginal, with nearly equivalent results in sentiment exact match (91.50% for single-task vs. 90.00% for multitask) and F1 score for entity recognition (55.75% vs. 54.75%). Qwen 2 7B demonstrated a clearer advantage for single-task prompts, particularly in text generation, as evidenced by a higher BLEU score (73.26% vs. 56.09%). However, multitask prompts outperformed in NER precision (32.20% vs. 27.97%), revealing a notable trade-off between precision and recall. LLama 3.1 8B defied expectations by favoring multitask prompts, which yielded higher BLEU scores (88.94% vs. 76.55%) and better sentiment exact match (83.70% vs. 81.00%), suggesting that this model handles multitasking with greater consistency and precision in text generation and sentiment analysis. Conversely, Phi3 Medium 14B showed the weakest overall performance, with a distinct preference for single-task prompts. The disparity was particularly significant in the BLEU score (57.63% for single-task vs. 16.98% for multitask) and in the F1 score for NER (22.62% vs. 11.68%), indicating significant challenges in processing multitask scenarios. Mistral 7B displayed the most variable results. While single-task prompts yielded a better BLEU score (76.41% vs. 70.84%), multitask prompts excelled in sentiment exact match (87.20% vs. 71.80%) and resulted in a notably lower formatting error rate (29.00‰ vs. 155.00‰), indicating its capacity to handle multitasking effectively in some contexts.

These findings highlight the fact that the effectiveness of prompts varies significantly across models. In some cases, such as with LLama 3.1 8B and Mistral 7B, multitask prompts proved to be more effective, challenging the assumption that simplicity in prompting always leads to better outcomes. The analysis also revealed complex trade-offs, particularly in tasks requiring a balance between precision and recall, as observed in NER. This suggests that the choice between single-task and multitask prompts can significantly impact not only overall accuracy but also how well a model handles the nuanced aspects of performance. The practical implications of this study are substantial. It provides concrete guidance on selecting effective prompts for different LLM architectures and applications, emphasizing the need for a nuanced, model-specific approach to prompt design. Furthermore, the results underscore the importance of understanding the underlying factors that drive performance variations. Future research should focus on unpacking these mechanisms to gain a deeper understanding of how model architecture, task characteristics, and prompt complexity interact. This would lay the groundwork for optimizing prompts and refining LLM performance in real-world contexts.

Author Contributions

Conceptualization, M.G. and F.D.M.; Methodology, M.G.; Software, M.G. and F.D.M.; Validation, M.G. and F.D.M.; Data curation, M.G. and F.D.M.; Writing–original draft, M.G. and F.D.M.; Writing–review and editing, M.G. and F.D.M.; Supervision, M.G. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

The data supporting the results reported in this study are available online in a publicly accessible GitHub repository. All datasets analyzed or generated during the research can be found at the following link: https://github.com/gozus19p/Comparative-Analysis-of-Prompt-Strategies-for-LLMs (accessed on 26 November 2024). We encourage further exploration and verification of our findings using these data.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Flow chart that describes the experiment workflow.

Figure 2. UML activity diagram representing the single-task prompts’ execution workflow.

Figure 3. UML activity diagram representing the multitask prompt’s execution workflow.

Figure 4. Flow chart that describes the Ollama inference pipeline.

Table 1

A single row of the dataset (review text has been truncated to enhance readability).

Review	Sentiment	Entities	Json
1st watched 2/9/2008, 4 out of 10 (Dir-J.S. Cardone [...])	negative	[‘label’: ‘ORG’, ‘value’: ‘qwest‘, ‘label’: ‘ORG’, ‘value’: ‘qwest’]	{“review”: “1st watched 2/9/2008, 4 out of 10 (Dir-J.S. Cardone [...]”, “sentiment”: “negative”, ”entities”: [{“label”: “ORG”, “value”: “qwest”}, {“label”: “ORG”, “value”: “qwest”}]}

Table 2

Performance scores of different models with single-task and multitask approaches across metrics.

Model	F1-Score NER		Sentiment Score		Review BLEU		Score
	Single	Multi	Single	Multi	Single	Multi	Single	Multi
Gemma2 9B	55.75%	54.74%	91.50%	90.00%	96.70%	97.49%	81.32%	80.74%
Qwen2 7B	26.98%	25.13%	82.70%	80.80%	73.26%	56.09%	60.98%	54.01%
LLama 3.1 8B	44.10%	43.00%	81.00%	83.70%	76.55%	88.94%	67.21%	71.88%
Phi3 Medium	22.62%	11.68%	50.80%	48.30%	57.63%	16.98%	43.68%	25.65%
Mistral 7B	32.56%	30.60%	71.80%	87.20%	76.41%	70.84%	60.26%	62.88%

Table 3

Shapiro–Wilk test p-values were measured for each model to test if the distributions are normal.

Model	F1-Score NER		Review BLEU		Sentiment Score		Score
	Single	Multi	Single	Multi	Single	Multi	Single	Multi
Gemma2 9B	$5.66\times {10}^{-26}$	$3.85\times {10}^{-26}$	$5.90\times {10}^{-52}$	$3.87\times {10}^{-53}$	$1.96\times {10}^{-51}$	$1.08\times {10}^{-50}$	$1.11\times {10}^{-27}$	$2.63\times {10}^{-27}$
Qwen2 7B	$1.56\times {10}^{-31}$	$2.12\times {10}^{-34}$	$3.02\times {10}^{-41}$	$3.52\times {10}^{-36}$	$1.25\times {10}^{-47}$	$5.96\times {10}^{-47}$	$1.99\times {10}^{-26}$	$5.47\times {10}^{-15}$
LLama 3.1 8B	$4.72\times {10}^{-26}$	$3.91\times {10}^{-26}$	$3.09\times {10}^{-44}$	$7.98\times {10}^{-49}$	$5.08\times {10}^{-47}$	$5.29\times {10}^{-48}$	$3.78\times {10}^{-30}$	$7.45\times {10}^{-32}$
Phi3 Medium	$1.07\times {10}^{-36}$	$6.03\times {10}^{-47}$	$9.12\times {10}^{-39}$	$4.25\times {10}^{-46}$	$1.36\times {10}^{-41}$	$1.32\times {10}^{-41}$	$5.22\times {10}^{-28}$	$8.91\times {10}^{-33}$
Mistral 7B	$8.20\times {10}^{-30}$	$7.08\times {10}^{-36}$	$1.06\times {10}^{-42}$	$6.29\times {10}^{-38}$	$2.66\times {10}^{-44}$	$2.02\times {10}^{-49}$	$9.74\times {10}^{-30}$	$3.22\times {10}^{-18}$

Table 4

The p-values for different models using various evaluation metrics.

Metric (p-Value)	Gemma 2 9B	Qwen 2 7B	LLama 3.1 8B	Phi 3 Medium	Mistral 7B
NER F1 Wilcoxon	0.02	0.01	0.06	$6.96\times {10}^{-16}$	0.02
Review BLEU Wilcoxon	0.01	$1.58\times {10}^{-28}$	$9.06\times {10}^{-10}$	$6.98\times {10}^{-63}$	$2.45\times {10}^{-7}$
Sentiment Score McNemar	0.03	0.13	0.06	0.29	$8.93\times {10}^{-27}$
Score Wilcoxon	0.03	$1.14\times {10}^{-18}$	$1.54\times {10}^{-5}$	$1.30\times {10}^{-29}$	0.60

Table 5

Summary of Friedman test results for different metrics and prompting conditions.

Metric	Single-Task	Multitask
F1-Score (NER)	Statistic = $1.03\times {10}^3$	Statistic = $1.05\times {10}^3$
	p-value = $7.49\times {10}^{-222}$	p-value = $3.29\times {10}^{-226}$
BLEU Score	Statistic = $1.00\times {10}^3$	Statistic = $1.97\times {10}^3$
	p-value = $1.21\times {10}^{-215}$	p-value = $0.0$
Sentiment Score	Statistic = $6.55\times {10}^2$	Statistic = $7.93\times {10}^2$
	p-value = $1.73\times {10}^{-140}$	p-value = $2.87\times {10}^{-170}$
Overall Score	Statistic = $1.31\times {10}^3$	Statistic = $1.81\times {10}^3$
	p-value = $1.69\times {10}^{-282}$	p-value = $0.0$

Table 6

Specific-task performances on Gemma 2 9B.

Metric	Multitask	Single-Task
Mean BLEU on review	97.49%	96.70%
Mean Exact-Match on sentiment	90.00%	91.50%
Mean NER F1	54.75%	55.75%
Mean NER Precision	60.99%	59.86%
Mean NER Recall	54.11%	56.87%
Formatting error rate	9.00‰	8.00‰

Table 7

Specific-task performances on Qwen 2 7B.

Metric	Multitask	Single-Task
Mean BLEU on review	56.09%	73.26%
Mean Exact-Match on sentiment	80.80%	82.70%
Mean NER F1	25.13%	26.98%
Mean NER Precision	32.20%	27.97%
Mean NER Recall	24.32%	30.79%
Formatting error rate	34.00‰	88.00‰

Table 8

Specific-task performances on LLama 3.1 8B.

Metric	Multitask	Single-Task
Mean BLEU on review	88.94%	76.55%
Mean Exact-Match on sentiment	83.70%	81.00%
Mean NER F1	43.00%	44.10%
Mean NER Precision	50.25%	47.98%
Mean NER Recall	42.05%	46.01%
Formatting error rate	69.00‰	94.00‰

Table 9

Specific-task performances on Phi 3 Medium.

Metric	Multitask	Single-Task
Mean BLEU on review	16.98%	57.63%
Mean Exact-Match on sentiment	48.30%	50.80%
Mean NER F1	11.68%	22.62%
Mean NER Precision	14.49%	25.45%
Mean NER Recall	11.06%	23.78%
Formatting error rate	253.00‰	307.00‰

Table 10

Specific-task performances on Mistral 7B.

Metric	Multitask	Single-Task
Mean BLEU on review	70.84%	76.41%
Mean Exact-Match on sentiment	87.20%	71.80%
Mean NER F1	30.60%	32.56%
Mean NER Precision	39.28%	37.05%
Mean NER Recall	28.04%	33.50%
Formatting error rate	29.00‰	155.00‰

Appendix A

Listing A1. Example of a ground truth JSON data.
1	{
2	"review": "But at least this movie got what it deserved - to be sent to the
	↪ Satellite of Love to be ridiculed on by Mike, Tom Servo, and~Crow T.
	↪ Robot from Pearl Forrester on \"Mystery Science Theater 3000!\" \"
	↪ Soultaker\" is one of those long lost, forgotten movies that are so
	↪ bad you’ll be guaranteed to have nightmares or depression later on in
	↪ life. Even though the movie is not that old, it’s still a very
	↪ forgotten type of movie. If~it had never been for the intelligent
	↪ minds at \"Mystery Science Theater 3000,\" the movie would not only
	↪ seem like it was never made, but~the movie wouldn’t be very enjoyable
	↪ by us moviegoers.<br /><br />In real life: this movie is really bad.
	↪ In~the Satellite of Love: this movie is excellent!",
3	"sentiment": "negative",
4	"entities": [
5	{
6	"label": "PERSON",
7	"value": "Mike"
8	},
9	{
10	"label": "PERSON",
11	"value": "Tom Servo"
12	},
13	{
14	"label": "PERSON",
15	"value": "Crow T. Robot"
16	},
17	{
18	"label": "ORG",
19	"value": "Pearl Forrester"
20	}
21	]
22	}

Appendix B

Listing A2. Multitask prompts.

PROMPT_TEMPLATE = """Instruction: Analyze the following review text

↪ and provide one distinct outputs formatted in JSON:

1.  **Sentiment Classification:** Indicate whether the sentiment of the

↪ review is "positive" or "negative".

2.  **Named Entity Extraction:** List all named entities present in the

↪ text, categorizing them by label (PERSON, ORG, LOC).

3.  **Required JSON Format:** Ensure the response is formatted in JSON

↪ according to the following schema:

{{

"sentiment": "<sentiment>",

"review": "<review>",

"entities": [

{{

"label": "<label>",

"value": "<value>"

}}

]

}}

example:

"I recently visited the restaurant ’La Dolce Vita’ in Rome and was

↪ thrilled with the service and food. The~waiter, Marco, was

↪ exceptionally friendly and the truffle risotto was simply divine.

↪ I can’t wait to return and recommend this place to my~friends."

‘‘‘json

{{

"sentiment": "positive",

"review": "I recently visited the restaurant ’La Dolce Vita’ in Rome

↪ and was thrilled with the service and food. The~waiter, Marco,

↪ was exceptionally friendly and the truffle risotto was simply

↪ divine. I can’t wait to return and recommend this place to my

↪ friends.",

"entities": [

{{

"label": "ORG",

"value": "La Dolce Vita"

}},

{{

"label": "LOC",

"value": "Rome"

}},

{{

"label": "PERSON",

"value": "Marco"

}}

]

}}

‘‘‘

{content}"""

Listing A3. Single-task prompts.

NER_PROMPT = """Analyze the following review text listing all named

↪ entities present in the text, categorizing them by label.

↪ Consider only PERSON, ORG, and~LOC categories.

Ensure the response is formatted in JSON according to the following

↪ schema:

[

{{

"label": "<label>",

"value": "<value>"

}}

]

Example:

"I recently visited the restaurant ’La Dolce Vita’ in Rome and was

↪ thrilled with the service and food. The~waiter, Marco, was

↪ exceptionally friendly and the truffle risotto was simply divine.

↪ I can’t wait to return and recommend this place to my~friends."

[

{{

"label": "ORG",

"value": "La Dolce Vita"

}},

{{

"label": "LOC",

"value": "Rome"

}},

{{

"label": "PERSON",

"value": "Marco"

}}

]

{content}"""

SENTIMENT_PROMPT = """Analyze the following review text indicating

↪ whether the sentiment of the review is "positive" or~"negative".

Example:

"I recently visited the restaurant ’La Dolce Vita’ in Rome and was

↪ thrilled with the service and food. The~waiter, Marco, was

↪ exceptionally friendly and the truffle risotto was simply divine.

↪ I can’t wait to return and recommend this place to my~friends."

"positive"

{content}"""

FORMATTING_OUTPUT = """You are given three informations: sentiment,

↪ review and entities. Generate a JSON representation using the

↪ following schema. Use just the data you receive:

{{

"sentiment": "<sentiment>",

"review": "<review>",

"entities": [

{{

"label": "<label>",

"value": "<value>"

}}

]

}}

example:

"Sentiment: positive

Review: I recently visited the restaurant ’La Dolce Vita’ in Rome and was

↪ thrilled with the service and food. The~waiter, Marco, was

↪ exceptionally friendly and the truffle risotto was simply divine.

↪ I can’t wait to return and recommend this place to my friends.

Entities: [

{{

"label": "ORG",

"value": "La Dolce Vita"

}},

{{

"label": "LOC",

"value": "Rome"

}},

{{

"label": "PERSON",

"value": "Marco"

}}

]"

‘‘‘json

{{

"sentiment": "positive",

"review": "I recently visited the restaurant ’La Dolce Vita’ in Rome

↪ and was thrilled with the service and food. The~waiter, Marco,

↪ was exceptionally friendly and the truffle risotto was simply

↪ divine. I can’t wait to return and recommend this place to my

↪ friends.",

"entities": [

{{

"label": "ORG",

"value": "La Dolce Vita"

}},

{{

"label": "LOC",

"value": "Rome"

}},

{{

"label": "PERSON",

"value": "Marco"

}}

]

}}

‘‘‘

Sentiment: {sentiment}

Review: {review}

Entities: {entities}"""

Appendix C

Listing A4. Regular expression to clean JSON-like LLM output.

clean_regex: str = r"(?<!\S)//.*?$"

Listing A5. Regular expression to extract JSON-like LLM output.

extract_regex: str = r"(\{.*\})"

Listing A6. JSON output parsing algorithm.

def parse(string: str) -> dict:

dictionary: dict = None

try:

dictionary = json.loads(string)

except Exception:

pass

try:

dictionary = eval(string)

except Exception:

pass

return dictionary

References

1. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.; Polosukhin, I. Attention Is All You Need. arXiv; 2017; [DOI: https://dx.doi.org/10.48550/arXiv.1706.03762] arXiv: 1706.03762

2. Kojima, T.; Gu, S.; Reid, M.; Matsuo, Y.; Iwasawa, Y. Large Language Models Are Zero-Shot Reasoners. 2022; Available online: https://www.semanticscholar.org/paper/Large-Language-Models-are-Zero-Shot-Reasoners-Kojima-Gu/e7ad08848d5d7c5c47673ffe0da06af443643bda (accessed on 26 November 2024).

3. Li, Y. A Practical Survey on Zero-Shot Prompt Design for In-Context Learning. arXiv; 2023; [DOI: https://dx.doi.org/10.26615/978-954-452-092-2_069] arXiv: 2309.13205

4. White, J.; Fu, Q.; Hays, S.; Sandborn, M.; Olea, C.; Gilbert, H.; Elnashar, A.; Spencer-Smith, J.; Schmidt, D.C. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv; 2023; [DOI: https://dx.doi.org/10.48550/arXiv.2302.11382] arXiv: 2302.11382

5. Sivarajkumar, S.; Kelley, M.; Samolyk-Mazzanti, A.; Visweswaran, S.; Wang, Y. An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing. arXiv; 2023; [DOI: https://dx.doi.org/10.48550/arXiv.2309.08008] arXiv: 2309.08008

6. Sahoo, P.; Singh, A.K.; Saha, S.; Jain, V.; Mondal, S.; Chadha, A. A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications. arXiv; 2024; [DOI: https://dx.doi.org/10.48550/arXiv.2402.07927] arXiv: 2402.07927

7. Joaquin, A.S.; Haroen, A. Understanding How Model Size Affects Few-shot Instruction Prompting. arXiv; 2022; [DOI: https://dx.doi.org/10.48550/arXiv.2212.01907] arXiv: 2212.01907

8. Yousri, R.; Safwat, S. How Big Can It Get? A comparative analysis of LLMs in architecture and scaling. Proceedings of the 2023 International Conference on Computer and Applications; Cairo, Egypt, 28–30 November 2023; [DOI: https://dx.doi.org/10.1109/ICCA59364.2023.10401818]

9. Linzbach, S.; Tressel, T.; Kallmeyer, L.; Dietze, S.; Jabeen, H. Decoding Prompt Syntax: Analysing its Impact on Knowledge Retrieval in Large Language Models. Proceedings of the Companion Proceedings of the ACM Web Conference; Austin, TX, USA, 30 April–4 May 2023; [DOI: https://dx.doi.org/10.1145/3543873.3587655]

10. Sclar, M.; Choi, Y.; Tsvetkov, Y.; Suhr, A. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. arXiv; 2023; [DOI: https://dx.doi.org/10.48550/arXiv.2310.11324] arXiv: 2310.11324

11. Lakshmipathi, N. IMDB Dataset of 50K Movie Reviews. 2019; Available online: https://www.kaggle.com/datasets/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews (accessed on 26 November 2024).

12. Shelar, H.; Kaur, G.; Heda, N.; Agrawal, P. Named Entity Recognition Approaches and Their Comparison for Custom NER Model. Sci. Technol. Libr.; 2020; 39, pp. 324-337. [DOI: https://dx.doi.org/10.1080/0194262X.2020.1759479]

13. Gozzi, M.; Maio, F.D. Comparative Analysis of Prompt Strategies for LLMs: Single-Task vs. Multitasking Prompts. 2024; Available online: https://github.com/gozus19p/llm-benchmark (accessed on 26 November 2024).

14. Liesenfeld, A.; Dingemanse, M. Rethink. Open Source Gener. AI: Open Washing EU AI Act. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency; Rio de Janeiro, Brazil, 3–6 June 2024; pp. 1774-1787. [DOI: https://dx.doi.org/10.1145/3630106.3659005]

15. Fourrier, C.; Habib, N.; Lozovskaya, A.; Szafer, K.; Wolf, T. Open LLM Leaderboard v2. Published by Hugging Face. 2024; Available online: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard (accessed on 15 June 2024).

16. Gao, L.; Tow, J.; Biderman, S.; Black, S.; DiPofi, A.; Foster, C.; Golding, L.; Hsu, J.; McDonell, K.; Muennighoff, N. et al. A Framework for Few-Shot Language Model Evaluation. Zenodo, 2021, v0.0.1. Available online: https://zenodo.org/records/14216804 (accessed on 26 November 2024).

17. Zhou, J.; Lu, T.; Mishra, S.; Brahma, S.; Basu, S.; Luan, Y.; Zhou, D.; Hou, L. Instruction-Following Evaluation for Large Language Models. arXiv; 2023; [DOI: https://dx.doi.org/10.48550/arXiv.2311.07911] arXiv: 2311.07911

18. Suzgun, M.; Scales, N.; Schärli, N.; Gehrmann, S.; Tay, Y.; Chung, H.W.; Chowdhery, A.; Le, Q.V.; Chi, E.H.; Zhou, D. et al. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. arXiv; 2022; [DOI: https://dx.doi.org/10.48550/arXiv.2210.09261] arXiv: 2210.09261

19. Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; Steinhardt, J. Measuring Mathematical Problem Solving With the MATH Dataset. arXiv; 2021; [DOI: https://dx.doi.org/10.48550/arXiv.2103.03874] arXiv: 2103.03874

20. Rein, D.; Hou, B.L.; Stickland, A.C.; Petty, J.; Pang, R.Y.; Dirani, J.; Michael, J.; Bowman, S.R. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv; 2023; [DOI: https://dx.doi.org/10.48550/arXiv.2311.12022] arXiv: 2311.12022

21. Sprague, Z.; Ye, X.; Bostrom, K.; Chaudhuri, S.; Durrett, G. MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning. arXiv; 2024; [DOI: https://dx.doi.org/10.48550/arXiv.2310.16049] arXiv: 2310.16049

22. Wang, Y.; Ma, X.; Zhang, G.; Ni, Y.; Chandra, A.; Guo, S.; Ren, W.; Arulraj, A.; He, X.; Jiang, Z. et al. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. arXiv; 2024; [DOI: https://dx.doi.org/10.48550/arXiv.2406.01574] arXiv: 2406.01574

23. Beeching, E.; Fourrier, C.; Habib, N.; Han, S.; Lambert, N.; Rajani, N.; Sanseviero, O.; Tunstall, L.; Wolf, T. Open LLM Leaderboard (2023–2024). Published by Hugging Face. 2023; Available online: https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard (accessed on 15 June 2024).