Revisiting Problem-Solution Co-Evolution in the Context of Team Conceptual Design Activity

Abstract

The conventional prescriptive and descriptive models of design typically decompose the overall design process into elementary processes, such as analysis, synthesis, and evaluation. This study revisits some of the assumptions established by these models and investigates whether they can also be applied for modelling of problem-solution co-evolution patterns that appear during team conceptual design activities. The first set of assumptions concerns the relationship between performing analysis, synthesis, and evaluation and exploring the problem and solution space. The second set concerns the dominant sequences of analysis, synthesis, and evaluation, whereas the third set concerns the nature of transitions between the problem and solution space. The assumptions were empirically tested as part of a protocol analysis study of team ideation and concept review activities. Besides revealing inconsistencies in how analysis, synthesis, and evaluation are defined and interpreted across the literature, the study demonstrates co-evolution patterns, which cannot be described by the conventional models. It highlights the important role of analysis-synthesis cycles during both divergent and convergent activities, which is co-evolution and refinement, respectively. The findings are summarised in the form of a model of the increase in the number of new problem and solution entities as the conceptual design phase progresses, with implications for both design research and design education.

1. Introduction

Ever since the 1960s, the design process has been dominantly prescribed within the context of the problem-solving paradigm. The initial efforts of the design methods movement [1] utilised linear and high-level depictions of the design process, the most common being the analysis-synthesis-evaluation (ASE) sequence as part of a general problem-solving cycle (see, e.g., [2,3,4] for the initial developments of ASE-based models). The work of Herbert Simon [5] further strengthened the paradigm by laying down the scientific foundations of designing as a problem-solving activity [6,7]. As such, the conventional “analyse-synthesise-evaluate” approach has been seen as a recurring and dominant model of design, used primarily in design education, but also as one that is reported to work well in design practice, especially in the engineering design domain [8].

A typical engineering design textbook (see, e.g., [9,10]) will thus define designing as a process in which information in the form of requirements, constraints, and needs (the problem) are systematically being converted into information describing technical systems (the solution). The systematic approach prescribes designers with a simple and linear process that can be applied at various points of the engineering design process: To collect a sufficient amount of information, to confront the problem by formulating the objectives and main constraints, to create solution alternatives, to evaluate them, and, finally, to make a decision on the final solution [11,12,13,14,15]. Simply put, the systematic design implies gradual movement from the design problem to the design solution, as well as from analytic over synthetic towards evaluative processes. Hence, the prescriptive engineering design literature instructs designers to always start with the analysis of design requirements, constraints, and needs in order to apprehend the overall problem, before they can start synthesising and evaluating solution alternatives that address the problem.

Nevertheless, the descriptive research streams (which have been focused on investigating how designers really work) revealed that this is not what designers tend to do when they come across new, unique, or ill-defined problems [16,17] and that the reality of design projects is in no way a gradual movement from problem to solution, nor from analysis to evaluation [18,19]. The descriptive approaches rather utilise a co-evolution perspective, where the design problem and solutions to the problem are developed in parallel, through a constant iteration of ASE [20]. According to the co-evolution perspective, new problem entities result from exploring possible design solutions, as well as the other way around [21,22]. It hence proposes a separation of the design space into two notional spaces: the problem and solution space. The first constitutes from problem space entities, such as requirements, constraints, and needs, whereas the latter consists of solution space entities, such as solution ideas, concepts, working principles, etc. Designers explore both spaces and switch between them, and as a result, the spaces co-evolve. The switching in-between the spaces appears causal in nature, while activity in one space associates to the activity in the other space in a highly interactive manner [23,24]. Due to its focus on the evolution of both the design problem and the design solutions, the co-evolution perspective has often been utilised to study design creativity [20,21,22,23,24].

Despite the different viewpoints on the nature of the design process, both the prescriptive (e.g., conventional problem-solving) and descriptive (e.g., problem-solution co-evolution) design research streams agree that the most intensive design space exploration is likely to appear within the conceptual design phase of product development. Namely, when designers conceptualise, they transform the initial and often ill-defined design problem into a clear description of a concept solution, thus ensuring a more certain design work in the subsequent stages [25,26]. The conceptual design stage demands substantial exploration and offers the most scope for striking improvements in return [27]. This is probably the reason why most of the design literature encourages team activity whenever there is a need for idea generation, or solution finding, evaluation, and refining [14,15,28,29,30]. Consequently, team activity is most desirable during tasks, such as defining product specification, searching for working principles, concept generation and evaluation, selection and refinement of concept solutions, and design reviews; that is, throughout the conceptual design phase. Indeed, the descriptive research studies confirm that not only does the majority of engineering designers in modern industrial practice work as part of a team [31,32], but the creative conceptual design tasks, such as idea generation or concept selection, are often performed exclusively as team activities [33,34]. Such a collaborative approach to framing design problems and developing solutions to these problems is believed to be the driver of creativity and innovativeness in the early product development stages [24]. It is thus no surprise that a large portion of experimental design research related to team activity has been related directly to the conceptual design stage [35].

Taking into account the state-of-the-art in design research and the introduced context of designing in teams, one can provoke the following question: Are the conventional prescriptive models of design and their descriptive counterparts accurate in describing the fine-grain problem-solving patterns appearing during team conceptual design activities? Namely, as shown hereafter, there exists a gap in the literature when it comes to fine-grain process modelling of team design activities. The presented study aims to address the proposed research question and bridge the research gap by revisiting some of the well-established assumptions that regard ASE as drivers of problem-solving and problem-solution co-evolution in the context of team design activity. As such, the study aims to investigate the validity of both the conventional problem-solving as an underlying process of designing, as well as the complementary viewpoint of problems and solutions evolving together. The new insights could potentially help design researchers to align the conceptualisation of ASE within the design models as well as to adapt the design methods to better suit the needs of collaborative design. Three sets of hypotheses have been proposed hereafter, in order to address the relevance of current models for fine-grain investigation of team design activity.

The first hypothesis set concerns the relationship between performing ASE and teams’ focus on exploring either the design problem or the design solutions. Namely, the prescriptive design research approaches, which inherit the problem-solving interpretation of ASE, have related the notion of analysis predominately to the problem space, where it envelops understanding, decomposition, and formulation of problem-related entities, primarily the design requirements (see, e.g., [36,37]). A few of the descriptive approaches accept the problem-related interpretation of analysis (e.g., [38,39]), whereas the others challenge it by favouring the relation of performing analysis and exploring the solution space instead (e.g., [40,41,42]). Therefore, the assumption that analysis is primarily related to the problem space exploration, as it is often instructed by the prescriptive design literature, should be revisited in the context of team designing. Namely, this assumption goes beyond simple counting of instances where teams perform problem analysis versus instances where they perform solution analysis (this has already been addressed in one of the previous studies; please consult [43]). It instead correlates analysis (no matter if performed in the problem or solution space) with problem space exploration, meaning that analysis-intensive activity is also likely to be more intensive in terms of exploring the problem space (to the degree it can be observed in the design team’s dialogue). On the other hand, synthesis has been primarily related to performing solution space exploration [39,44], although it has been demonstrated to play an important role in the evolution of both design problems and solutions [45]. One can, therefore, assume an interplay between performing synthesis and exploring solution space, such as that synthesising intensity correlates with the intensity of exploring the solution space. Similarly, the term evaluation has almost exclusively been used to describe performing of the assessment of solution entities rather than problem entities (see, e.g., [41,46,47]), thus triggering additional efforts of solution refinement. Hence, there is an assumption of correlation between performing evaluation and exploring the solution.

The following three hypotheses were formulated in order to investigate the first set of literature-based assumptions:

Hypothesis 1a (H1a).

Performing analysis during team conceptual design activity is positively correlated with performing problem space exploration.

Hypothesis 1b (H1b).

Performing synthesis during team conceptual design activity is positively correlated with performing solution space exploration.

Hypothesis 1c (H1c).

Performing evaluation during team conceptual design activity is positively correlated with performing solution space exploration.

The second hypothesis set concerns the dominant sequences of ASE performed in order to explore the design space during team design activity. As mentioned previously, the early systematic approaches tended to prescribe the analysis-synthesis-evaluation (A-S-E) sequence [2,3,4], implying that problem analysis must be followed by synthesis and evaluation of solutions. Different portrayals of the A-S-E sequence can still be found across design textbooks (e.g., [14,28,38,48]) but also within the scientific literature [37,39,49,50], where this specific sequence has occasionally been considered as an underlying conceptual design process [51].

Mc Neil et al. [39] addressed the A-S-E sequence in their study of individual electronic designers. They found out that, indeed, the analysis was most probable to be followed by synthesis, and synthesis was most probable to be followed by evaluation. Interestingly, they found out that evaluation will most probably lead back to synthesis, rather than analysis. Nevertheless, it is worth revisiting the overall ASE sequence in the context of team designing, particularly considering the descriptive design models that tend to favour contrasting sequences of ASE [51], particularly when describing observations of a moment to moment creative design thinking [52]. Examples include the often-cited thinking processes in design teams by Stempfle and Badke-Schaub [41] and Gero’s FBS framework [53], which both utilise the synthesis-analysis-evaluation (S-A-E) sequence as a vital part of solution-related design cognition. Nevertheless, given that cognition cannot be directly observed, the study will revisit assumptions related to the well-established A-S-E sequence, and test the following set of hypotheses in the context of team design activity:

Hypothesis 2a (H2a).

Analysis is most probable to be followed by synthesis during team conceptual design activity.

Hypothesis 2b (H2b).

Synthesis is most probable to be followed by evaluation during team conceptual design activity.

Hypothesis 2c (H2c).

Evaluation is most probable to be followed by synthesis during team conceptual design activity.

In addition to the assumptions stemming from the prescriptive viewpoint of design as solving well-defined problems, two additional hypotheses can be posed based on the descriptive perspective of design as co-evolution of the ill-defined problem and the design solutions. According to Maher et al. [22,23], the conventional problem-solving process can be characterised as a design search, where the set of requirements is well-defined and fixed prior to the search for solutions. Such a process typically follows the prescribed A-S-E sequence, as described in the above-listed assumptions. The search becomes exploration when different parts of the solution space are searched, or space itself is expanded due to changes in the set of problem entities. The result of exploration is not only a solution but also a well-defined problem [22]. In co-evolutionary design, the exploration also spreads to the problem space and implies mutual interaction between the design problem and the solution alternatives, instead of just the continuous refinement of solutions [21,23]. Consequently, the co-evolution perspective proposes that, in the case of ill-defined design problems, it is the interaction between the two spaces that introduces new problem and solution entities, such as requirements and concept ideas. This so-called co-evolutionary reasoning may arise when an existing problem directly connects to a generated solution, or when a new problem is triggered by a solution [54]. In other words, it can be hypothesised that the emergence of new problem entities during team design activity will most probably follow after exploration of the solution space.

An analogous hypothesis can be made for moving from the problem to solution space; that is, the emergence of new solution entities during team design activity will most probably result from problem space exploration. According to Wiltschnig et al. [24], the dominant type of co-evolution episodes in the collaborative design are the ones where problem analysis leads to solution attempts rather than the other way around. Hence, the third set of hypotheses can be formulated as follows:

Hypothesis 3a (H3a).

When solving ill-defined design problems, the synthesis of new problem entities is more probable to be a result of solution space exploration.

Hypothesis 3b (H3b).

When solving ill-defined design problems, the synthesis of new solution entities is more probable to be a result of problem space exploration.

In order to revisit how these particular assumptions about the problem-solution co-evolution work in the context of designing in teams, the formulated hypotheses were tested as part of a protocol analysis study of team conceptual design activity. Protocol analysis, pioneered by Ericsson and Simon [55], is a frequently employed process-oriented analysis method which utilises observation and recording of subjects’ actions to gain an understanding of ways and approaches to designing [56]. The resulting protocols represent designers’ step-by-step acts concerning the specific aspect of the investigation and can be used for various quantitative and qualitative analyses of the design process [23,57]. The establishment of the method has been supported by the ever-increasing capability, efficiency, and affordability of data capturing and analysis tools needed to perform process-oriented studies in design research (audio-video hardware and software) [58]. Protocol analysis was hence selected as the method of choice for capturing the fine-grain problem-solving patterns that would enable testing of the proposed hypotheses. The model and the metrics that were used for capturing and analysing the protocol data represent an extension of the authors’ previous work on investigating team conceptual design activity [43]. Namely, the previously proposed theoretical framework utilises the conceptualisation of designing as a state-transition process and enables the measurement of occurrences of ASE, the intensity of problem vs. solution space exploration, as well as the occurrences of transitions in between the spaces.

The paper is structured as follows. Section 2 describes the protocol analysis study that was conducted to collect data on the exploration of the problem and solution space during team conceptual design activity, including the theoretical foundations of the model used to capture the data and the experimental procedure. Section 3 reports the results of the protocol analysis and compares them against the study hypotheses. A detailed discussion of the results, the main research implications, and study limitations are provided in Section 4, whereas the conclusions are drawn in Section 5.

2. Materials and Methods

Three major prerequisites were identified prior to testing the proposed hypotheses. First, providing support in the form of a model that would enable capturing of fine-grain ASE instances during team design activity. Second, developing adequate measures that would allow quantification of the processes captured using the model. Finally, obtaining the experimental dataset of team design activity as a source of the investigated phenomena. Hence, the study methodology consisted of three corresponding steps: Modelling team design activity, developing process measures, and conducting a protocol analysis study. These three methodological steps are described hereafter.

2.1. Theoretical Foundations

Given the introduced diversity of interpretations, there exists a need for developing clear and unambiguous definitions of analysis, synthesis, and evaluation, and their role in design space exploration. In order to tackle this need, the authors synthesised various notions across the literature and conceptualised ASE as three fundamental mechanisms for the exploration of design space, independent of its problem and solution portions. On the fine-grain level of the design process, which is characteristic to the design activity, the instances of such mechanisms appear numerous times, thus continuously changing the state of the product being designed. This transition-based depiction of the design process aligns with the domain-independent descriptive model of design proposed by Reymen et al. [59], where the evolution of design space (problem and solution) is represented as a set of states, whereas the act of designing transforms one state into another. By merging the perspective of change in the state of the product being designed and the fine-grain decomposition of the design process into ASE, the team design activity was framed in the form of an ASE state-transition model.

In the state-transition model, designing is decomposed into fine-grain transitions that team members perform in order to explore, that is, to change the state of the design space [43]. These transitions imply the creation and assessment of design entities, such as ideas, concepts, and requirements, as well as increasing the understanding of these entities. More precisely, the transitions correspond to the analysis, synthesis, and evaluation of both problem- and solution-related design entities. Consequently, the sequences of transitions represent the evolution of the explored design space; that is, the change of the state of the product being designed, and the state of the design process while teams approach the goal of the design activity (as proposed by Reymen et al. [59]).

By presuming similar mechanisms for the exploration of both the problem and solution space, the ASE transitions were defined as follows (based on [43]):

• Analysis is a state transition resulting in a change in the level of understanding of a particular design entity within the explored design space. Analysis performed in the problem space (problem analysis) clarifies different aspects of the design problem (needs, requirements, constraints, etc.). Analysis performed in the solution space (solution analysis) increases the understanding of the proposed solutions (ideas, concepts, alternatives, etc.). Solution analysis can be performed by determining or learning the behaviour of a solution (how the proposed solution works/behaves) or by clarifying the structure (building shared understanding) of a solution entity.

• Synthesis is a state transition resulting in the appearance of a new design entity within the explored design space. Solution synthesis encompasses both the creation of new solutions and the improving, refining, and combining of existing solution entities since the original design entities (the ones being improved/refined/ combined) remain in the solution space, and new derivatives appear. Problem synthesis corresponds to operations aimed at identifying new problem entities and decomposing the given design problem.

• Evaluation is a state transition resulting in the assessed appropriacy of a particular design entity within the explored design space and the context of a given design problem. Evaluation of a design entity (in problem or solution space) is performed by addressing a criterion, which is the relevant requirement or constraint entity in the problem space. Problem evaluation is considered as a means of assessing the appropriacy of the problem entities, such as requirements, constraints, or subgoals, whereas solution evaluation corresponds to the assessment of solution entities, such as concept ideas, by focusing on their value and feasibility in the context of the given design problem.

An illustration of the mechanisms associated with the three types of transitions is given in Figure 1. The illustration assumes that the explored design space is populated with a number of different design entities, some related to the design problem (circles) and others related to design solutions (squares). The higher the level of understanding of a design entity, the darker it appears within the state illustrations. Similarly, the size of the design entities corresponds to their appropriacy in the context of the given design problem. The illustrated sequence of three transitions represents a random excerpt that might appear during team design activity. Synthesis is depicted as a transition from state i towards state i + 1, where it introduces a new solution entity (depicted as a square) into the explored design space. The transition from state i + 1 to state i + 2 represents analysis, which increases the understanding of the newly introduced design entity. Finally, the transition from state i + 2 to state i + 3 represents evaluation. The evaluation assesses the appropriacy of a design entity in relation to the relevant criteria entity. For this reason, the evaluative transition encompasses the problem-space entity (e.g., a requirement or a constraint) when assessing the appropriacy of the analysed solution entity.

The state-transition model can be further elaborated if the problem and solution entities are divided into two associated spaces: problem and solution space (see Figure 2). In this case, the transitions within the spaces reflect the evolution of a single space (problem or solution), whereas the transitions in between the spaces reflect the co-evolution of design problems and solutions. The change in the state of the product being designed is expressed either with the change of the design problem or with the change of the design solution.

Formulation of fundamental differences between analysis, synthesis, and evaluation and their effect on changing the state of the explored design space allows mapping of different fine-grain design process notions available throughout the literature (see Table 1). The ability to map the process-related notions appearing in other studies is essential for inter-study comparison and discussion of insights resulting from the protocol analysis study. The mapping was performed based on the definitions given in the literature and relating them to ASE as described above. More precisely, it was inspected whether an entity is created as a result of that fine-grain activity (synthesis), whether an understanding of an entity is increased (analysis), or whether the appropriacy of an entity is assessed (evaluation).

2.2. Measures and Visualisations

Fine-grain investigation of team design activity captures the process in the form of a segmented protocol, where each segment corresponds to ASE state transitions performed within or in between the problem and the solution space. Hence, the protocol does not describe the state of the explored design space but rather the transitions between the states (as shown in Figure 2). Each transition poses an incremental change to the explored design space, either in its problem (circles) or solution (squares) portion. Sequences of transitions within either the problem or the solution space represent the evolution of the respective space, whereas sequences of transitions in between the two spaces represent the co-evolution of the design problem and solutions.

In order to simplify the capturing of activity protocols, the ASE transitions can be further categorised based on the space they pose change to. For example, ASE taking place within problem space were labelled as “problem analysis” (PA), “problem synthesis” (PS), and “problem evaluation” (PE). Likewise, ASE within the solution space were labelled as “solution analysis” (SA), “solution synthesis” (SS), and “solution evaluation” (SE). These transitions were, from here on, conceptualised as six basic state transitions.

An example of an activity protocol excerpt that could be sampled from the overall activity process is shown in Figure 2. Of particular interest are the transitions that introduce new entities in one space following the transitions within the other; that is, when problem space exploration results in solution synthesis or when solution space exploration results in problem synthesis. These transitions can be treated as potential co-evolution episodes, as highlighted in Figure 2.

The state-transition model was intended for capturing ASE in the problem and solution space by means of experimental studies (as demonstrated in [43]) as well as a support for predicting the most probable sequences of state transitions during team design activities. Both purposes require identifying and formulating the variables of interest, as well as their measures and a reliable and valid manner of measurement [69]. In terms of fine-grain investigation of the design process, the variables and measures are typically based on the design protocols. Namely, the majority of fine-grain studies of design activity utilise protocol analysis to decompose the process into small chunks (process elements) [35], and the resulting protocols (instances of process elements) are usually analysed in terms of their duration, frequency, proportions, sequences, and probabilities of moving from one process element to another (for the most relevant examples, please consult [24,41,42,46,50,66,67,70,71,72,73,74]). Proportions and probabilities are particularly useful as they enable straightforward inter-study comparisons. Hence, a similar approach was adopted here, and three dependent sets of variables were defined as follows:

• Proportions of state transitions: Instances of ASE transitions within the problem and the solution space are counted and normalised (divided by their total number) in order to calculate the proportion of each type of transition in the time span of the team design activity (or fragment of the activity). Proportions of state transitions (measured in percentages) provide insight into the general state-transition nature of the investigated activity, in terms of the team’s orientation towards analysing, synthesising, or evaluating problem and solution entities. When analysing team design activity, it is necessary to capture the appearance of six basic transitions. These six types can, of course, be aggregated into ASE or problem- and solution-related design operations.

• Proportions of state-transition sequences: Instances of two or more (depending on the specific aspect of analysis) consecutive state transitions are counted, and the overall distribution is normalised to calculate the proportions of different combinations of sequences of two or more state transitions. Proportions of state-transition sequences, which are also measured in percentages, enable the identification of most common state-transition patterns exhibited when designing in teams. The total number of possible sequence combinations between the six basic transitions (ASE in problem and solution space) is 36. These 36 sequence combinations between two transitions can be aggregated into nine combinations of moves between analysis, synthesis, and evaluation and four combinations of moves in between the problem- and the solution-related transitions, thus providing higher-level process measures.

• Probabilities of state-transition sequences: The counted instances of two consecutive state transitions can simply be transformed into probabilities of one particular state transition following another. These probabilities, again measured in percentages, are useful for both comparing and simulating experimental datasets, particularly when interpreted as probability matrices in Markov processes [75].

Variables related to the proportions of state transitions can be utilised for describing the general nature of team design activity or its fragments, whereas the variables related to proportions and probabilities of state-transition sequences are intended for modelling the patterns of state transitions during team designing. The measurements associated with the proposed variables can be sampled from the team activity protocols that must previously be captured using the state-transition model. Simply put, the proportion measures sampled from the experimental datasets enable quantification of the design process and statistical testing of hypotheses. The reliability of the used measures [69] must be ensured as part of the data collection process. In the case of conducting a protocol analysis study, the level of reliability must be determined by calculating the inter-rater (inter-coder) reliability [73,76].

The following notation is used for the proportion variables: p_A for the proportion of analysis, p_S for the proportion of synthesis, and p_E for the proportion of evaluation. Since the added-up proportions of ASE always make up 100% (p_A + p_S + p_E = 100%), only two measures are needed to characterise an activity (the third measure can be deducted, i.e., p_E = 100% − p_A + p_S). If the triangular proportion visualisation is utilised, the two measures are embedded in the vector, which is anchored in the centre of the triangle (Figure 3). The measures correspond to the vector’s endpoint distance from the triangle centre of gravity (vector length r) and the vector’s direction (angle δ), as shown in Figure 3. If the distance R from the centre of gravity to the corners of the triangle is conceptualised as equal to 1 (or 100%), then the vector length r ranges from 0 to a maximum of 1 in the triangle corners, whereas the direction angle δ can be any angle.

Furthermore, suppose the angle δ is defined clockwise from the vertical axis, as shown in Figure 3. In this case, the relations between the triangular visualisation variables and the proportions of ASE design operations can be defined as shown in Equations (1)–(3). The equations reveal that, in the case where the vector length is zero, the proportion of all three design operations is equal to 1/3:

p_A = 1/3 − (2/3) · r · sin (δ – 30°),(1)

p_S = 1/3 + (2/3) · r · cos (60° – δ),(2)

p_E = 1/3 – (2/3) · r · cos (δ).(3)

The proposed visualisation does not only enable intuitive and straightforward characterisation of the activities’ nature in terms of ASE but can also be used to describe the change in ASE proportions as a function of time passed during the activity (e.g., by sampling different parts of the activity). Moreover, additional layers of information can be added by colour coding the proportion instances within the triangle in order to display, for example, the problem/solution ratio, as well as the probabilities of particular state-transition sequences. Different use cases for the proposed triangular proportion visualisation are presented across the results section.

2.3. Experimental Study

Guided by the fine-grain process-oriented studies of design activity reported in the literature, the experimental part of the research was conducted in the form of verbal protocol analysis. Team communication in conceptual design is primarily verbal [25,77], and the concern regarding the validity of verbalisations in design teams as an authentic reflection of real-time thinking in design teams is insignificant [78]. Other observational methods, such as the analysis of designers’ facial or postural gesturing, are less practical [79] and prone to lower levels of reliability when compared to verbal protocol analysis (e.g., [80]). In the end, these studies often rely also on the verbalisations in order to complement the analysis (e.g., [81]). Hence, instead of being separately coded, the gestures, mimics, gaze, sketching, and other observable aspects can be used for better interpretation of the verbalised state transitions. Indeed, as a “third-party observing”, protocol analysis can be scientific, independent, and relatively objective if it is used to detect the observable aspects of designing [12,82].

2.3.1. Experimental Dataset

The experimental dataset includes video recordings of four teams taking part in two types of conceptual design activities: ideation and concept review. The video recordings were obtained from the previously conducted and rigorously designed experimental studies by Cash et al. [83]. Their experimental structure combines individual and team design activity across four sequential sessions. Such structure is in line with the general recommendation that team members should spend at least some of their concept generation time working alone, whereas team activities are critical for building consensus, communicating information, and refining concepts [29].

The participants were all mechanical engineering students selected from a final-year product design and development course. A total of 12 participants took part in the experiment. Each had 10 months of industrial experience and 4 years of academic training background. The participants communicated in English (their native language). For more information on the teams and the experiment, please consult Cash et al. [83] and Cash and Maier [81].

Participants would first individually work on a task that required seeking of feasibility-level technical information on camera-mounting devices. The second task implied a 50-min team ideation activity, where the participants collaborated in groups of three in order to come up with a number of concept ideas for mounting a camera on a remotely operated balloon. The participants would then again work individually on a concept elaboration task with the goal of developing a single detailed concept solution. In the final task, the teams would meet again to review the concepts collaboratively. They again had 50 min to agree on a single concept or a combination of concepts and refine them into a final concept solution.

Given the introduced research aims, the presented analysis focused solely on the two team design activities: ideation and concept review.

2.3.2. Protocol Coding

Protocol coding was conducted using the ELAN software [84] for video annotation. The dataset that was imported within the annotation software consists of 3 separate video recording files per session. Two cameras were oriented towards the experiment participants (team members), and the third one was oriented towards the whiteboard. Furthermore, a coding scheme was developed through several iterations of familiarisation with the video recording data set. The familiarisation implies the examination of the recordings in order to understand better the required protocol granularity and list all codes needed for annotating the entire sessions. Once finalised, the coding scheme was imported to the annotation software. The final version of the coding scheme is presented in Table 2. It includes the six basic transitions (PA, PS, PE, SA, SS, SE) as well as two additional codes: “process” (PROC) and “other” (O). The two additional codes were not included in further design process analyses, as they are intended for capturing verbalisations that are not related to design space exploration; however, they were coded for the convenience of future research using the same experimental data set.

Codes related to problem space exploration (PA, PS, and PE) are intended for verbalisations concerning requirements, constraints, specification, user needs, use scenarios, criteria, or functions. Codes related to solution space exploration (SA, SS, and SE) are intended for verbalisations regarding ideas and concept solutions; that is, determining/learning solution entity behaviour and clarifying/building share understanding on the structure of solution entities. Process codes concern aligning the team’s design process (e.g., where to start, how to proceed, etc.) and other codes concern verbalisations that cannot be annotated with the above codes (unrelated facts, joking, off-topic discussion, etc.).

As part of the coding process, the recorded team conversations were transcribed and parsed into coded segments, which were then treated as units of analysis [23,85]. The transcription process helped to familiarise with the data and to develop and refine the instructions for coding, before performing the final segmentation and coding step. The coding was performed by the primary researcher and a trained coder. The trained coder did not take part in the development of the model and coded random 10-min intervals of each session (20% of session duration) in order to ensure inter-rater reliability [86]. Since the aim was to investigate proportions and sequences of transitions, the outputs of the coding were depicted as strings of codes, which may vary in length based on who coded the data. A procedure proposed by Quera et al. [87] and GSEQ software [86] were utilised to align the protocols and calculate the event-based interpretation of Cohen’s kappa: the event alignment kappa (ĸ_a). Both the overall event alignment kappa value (ĸ_a = 0.71) and the event alignment kappa values for particular codes indicate substantial agreement when compared to the established values in experimental design research (e.g., [24,72,88]). After assessing the inter-rater reliability (see Table 2), the two coders resolved all identified conflicts and agreed on the final event sequences.

3. Results

Different types of analyses were performed on the resulting protocol data sets in order to test the hypotheses proposed in the introductory section. The overall distribution of coded segments, the comparisons between the teams, and the information on the general nature of ideation and concept review activities can be found in the authors’ previous work on the development and testing of the state-transition model [43]. In short, 333 codes per team were on average coded for the ideation activity and 313 for concept review activity. If the “process” and “other” codes are deducted, the averages are 293 during ideation and 280 during the concept review. The most frequent ASE state transition during ideation was synthesis (on average 49%), followed by analysis (32%) and evaluation (19%). On average, the teams spent a significant amount of time exploring both the problem (37%) and the solution space (63%). The order of the most frequent state transitions was different during concept review: Analysis was the most frequent (44%), followed by synthesis (31%) and evaluation (25%). Concept review was also more solution focused, as only 12% of state transitions were related to problem-space entities and 88% to solution-space entities. Analysis, synthesis, and evaluation were shown to play an important role in the exploration of both the problem and the solution space since all three state transitions appeared within and in-between both spaces.

The presented analysis approach builds on the previous results in the sense that it focuses on the dynamic aspects of the design activity. It samples different fragments of the activity protocols and analyses if regularities can be found both across different teams and at different points in team design activity. Namely, the initial number of data points was relatively small: One point per observed experiment session (eight data points in total), which corresponds to the average proportions of design operations during the whole activity (as reported in [43]). In order to increase the number of data points as well as to provide better insight into process dynamics, the overall protocol strings were split into three equal fragments; that is, subsets containing the same number of protocol strings. The rationale for splitting the protocol strings lies in the assumption that the hypothesised relationships should be consistent not only on the activity level but also for different fragments of the activity. Such splitting resulted in a total of 24 data points, i.e., three fragments per session: The first representing the beginning (first third), the second the middle (second third), and the third the end of the activity (final third). The resulting protocol fragments vary in length from 73 to 114 segments, thus providing a sufficient number of state-transition instances for calculating the previously introduced process measures (proportions and probabilities of state transitions and their sequences).

Splitting of protocols into fragments also allows the triangular visualisation of the teams’ processes in terms of ASE transitions and problem/solution-focus, as shown in Figure 4. The proportions of ASE for each activity fragment were transformed into the position within the triangle, whereas the proportions of problem-related verbalisations were depicted using the markers’ colour. The colours correspond to a gradient from full red (0% of problem-related verbalisation) over yellow (about 30% of problem-related verbalisation) to full green (60% of problem-related verbalisation). Note that for the convenience of visualisation, the scale was limited to 0–60%; however, in general, it can be expanded to any range between 0% and 100%. The resolution of these depictions can be increased if the session protocols are split into more fragments. Nevertheless, this would result in smaller protocol fragments that cannot be analysed using the measures that rely on proportions. Hence, the three-fragment protocols were assessed as the most appropriate for the following analyses.

Figure 4 (based on data from Table A1 in Appendix A) reveals that the ideation processes of all teams slightly gravitated towards the synthesis when compared to the concept review process, which gravitated towards analysis and evaluation. Additionally, ideation steps were more problem focused when compared to the concept review steps. Nonetheless, the proportions of ASE and problem/solution focus alternated in no small degree during the activities, which is best depicted in the change of the marker position and colour.

The teams took different approaches during the two activities. Team 1 exhibited a clear distinction between ideation and concept review. During ideation, they progressively moved towards the synthesis corner, with an intensive exploration of the problem space throughout the session. During concept review, the team exhibited less alternation in terms of ASE and focused primarily on solution space exploration. Team 2 focused on problem space exploration at the beginning of ideation and then gradually moved towards the solution space as the session progressed. In terms of ASE, they altered more during concept review, which was primarily solution focused with a slight increase in problem-related verbalisation early and late in the session. The processes of team 3 were less distinguishable in terms of ASE when compared to teams 1 and 2. Nevertheless, one can again notice a small shift towards synthesis during ideation and analysis during the concept review. The somewhat higher relative proportions of problem-related discussion can be seen in the middle of the ideation, and the beginning of concept review activity. Finally, in terms of ASE, team 4 exhibited more uniform behaviour during both ideation and concept review. Ideation was more synthesis and problem focused, particularly at the beginning of the session, whereas concept review was focused mostly on analysis and evaluation within the solution space.

The following subsections utilise the described data points in order to investigate further the assumed relationships between ASE and design space exploration, the analysis-synthesis-evaluation state-transition sequence, as well as the state transitions in between the problem and the solution space.

3.1. Relationship between ASE and Problem/Solution Exploration

The first set of hypotheses concerns the correlation between performing ASE state-transitions and exploring the problem and the solution space. It was assumed that the intensity of performing analysis is positively correlated with the intensity of problem space exploration, whereas the intensities of performing synthesis and evaluation are positively correlated with the intensity of solution space exploration. For this reason, simply counting the instances of ASE within the problem space vs. ASE in the solution space is not sufficient for testing the hypotheses. Namely, it is evident in the data (as reported in [43]) that the total number of all three types of state transitions was higher within the solution space than in the problem space. However, this data does not provide insight into the relationship that might exist between a particular type of state transition and problem vs. solution space exploration. Hence, additional analyses were made as described hereafter.

First, the data points sampled from all fragments and teams were mapped within a single triangular visualisation of ASE proportions across the activities and the associated proportions of problem-related verbalisations (Figure 5). Such representation gives an overview of the problem space focus that was associated with different proportions of ASE state transitions. Ideation activity fragments (diamonds in Figure 5) are associated with higher proportions of problem-related verbalisations and spread towards the synthesis corner of the triangle when compared to the concept review fragments (circles in Figure 5), which gravitate towards the analysis/evaluation side and exhibit higher proportions of solution-related verbalisations.

Given the hypothesis H1a, one would assume that the closer the markers are to the analysis corner, the more problem-focused transitions they should exhibit. Moreover, according to the hypotheses H1b and H1c, the closer the markers are to the synthesis/evaluation side, the more solution-focused transitions they should exhibit. This initial, qualitative analysis shows that this is not necessarily the case. In fact, markers closer to synthesis describe higher proportions of problem-related verbalisations than those closer to analysis and vice versa. Hence, in order to further investigate the assumed relationships, the hypothesised correlations were tested using Spearman’s rank-order correlation coefficient as a measure of correlation.

The reason for using Spearman’s instead of Pearson’s correlation was that it does not assume the normality, the homoscedasticity, or the linear relationship of the measurement data. Namely, the data were first tested for normality using the Shapiro–Wilk test [89], and the measures of problem and solution proportions were rejected concerning the normality assumption at the significance level of 0.05. Moreover, the scatter plots of the proportions (see Table 3) revealed issues concerning the homoscedasticity and linear relationship. Hence, Spearman’s rank-order correlation coefficient was selected as the most appropriate measure of the correlation between the proportions of ASE state transitions and the associated proportions of problem/solution-related transitions.

One important note is that the following analysis assumes that the observations (measured proportions) are independent. This is an important and debatable assumption, given that the data for each team is sampled from a series of subsequent fragments of two activities. Hence, one could argue that each observed fragment depends on the prior one. However, given the Markov processes perspective underlined within the state-transition model, the state transitions, as well as their sequences and proportions, are here considered independent of the previous ones.

It must also be noted that the correlations only exist when no distinction between ideation and concept review as two parts of the same team design process is made. If ideation and concept review are analysed separately, neither activity exhibits a significant correlation between the frequencies of instances within the problem and the solution space, and the frequencies of ASE state transitions. Hence, the correlation can be found on the level of the overall design process rather than within the team design activity. With this said, the Spearman’s rank correlation coefficients and the corresponding scatter plots are shown in Table 3.

No empirical support was found for hypotheses H1a and H1b based on both the triangular visualisation of team activity (Figure 5) and the correlation testing (Table 3). The correlation tests show a significant opposite trend in the data, meaning that there is a strong negative correlation between performing analysis and problem-related state transitions, and a strong negative correlation between performing synthesis and solution-related state transitions. There is, however, support for H1c, as the Spearman’s correlation test shows a significant and strong positive correlation between performing evaluation and solution-related state transitions.

3.2. The Analysis-synthesis-evaluation Sequence

The second set of hypotheses concerns the dominant sequences of ASE state transitions in design space exploration. The sequences were investigated by determining the probabilities of one state transition following another; that is, what is the probability of either analysis, synthesis, or evaluation to follow the previous type of state transition (Markov analysis [75]). First, the different combinations of adjacent pairs of state transitions in the protocol fragments of all teams were counted (e.g., A→A: analysis followed by analysis, A→S: analysis followed by synthesis, A→E: analysis followed by evaluation, etc.). These counts were normalised into percentages by dividing the count of a particular sequence with the total number of sequences that share the same starting state transition. The normalised results (see Table A2 in Appendix A) represent the probabilities of performing analysis, synthesis, and evaluation, given the previous state transition. The probabilities across all activity fragments and all teams were summarised using boxplots in Figure 6.

The boxplots show that the most probable state transition following analysis is synthesis and that the most probable state transition following synthesis is analysis. It is, however, not clear whether evaluation is most probable to be followed by analysis or synthesis. Additional statistical tests were performed to investigate the significance of difference between these probabilities.

In their investigation of the analysis-synthesis-evaluation sequence, Mc Neil et al. [39] utilised the t-test to compare the pairs of sequence probabilities. The here-presented statistical analysis differs in two ways. First, it also takes into account the probabilities of performing a sequence of two same state-transitions (analysis-analysis, synthesis-synthesis, and evaluation-evaluation). Second, it utilises a non-parametric statistical test, due to the violated assumption of normally distributed data for some of the sequence proportions and probabilities. Namely, while the Shapiro–Wilk test did not reject the assumption of normally distributed data regarding the ASE sequences, the Levene’s test rejected the assumption of homogeneity of variance for all three analysis cases shown in Figure 5. For that reason, the Friedman non-parametric test was used to test whether there exists a significant difference between the probabilities sampled across the team design activities, and the Wilcoxon signed-rank test was used in the post hoc analysis in order to determine whether a significant difference exists between the specific pairs of sequence probabilities.

The Friedman test indicates significant differences between probabilities within all three groups: state transitions following analysis (χ² = 20.891, p < 0.001), state transitions following synthesis (χ² = 24.333, p < 0.001), and state transitions following evaluation (χ² = 32.109, p < 0.001). The Wilcoxon signed-rank test (paired per the same starting state transition in a sequence) was then conducted to identify which pairs of sequence probabilities differ significantly within the three groups. In order to reduce the likelihood of Type I error, the Bonferroni correction was applied to the results of the Wilcoxon test. Significant differences at the α = 0.05 level were found between the following probability pairs: A→A and A→S (p = 0.002), A→S and A→E (p < 0.001), S→A and S→S (p = 0.006), S→A and S→E (p < 0.001), E→A and E→E (p < 0.001), and E→S and E→E (p < 0.001). On the other hand, no significant difference was found between the following pairs: A→A and A→E (p = 0.588), S→S and S→E (p = 1), and E→A and E→S (p = 1).

Hence, there is empirical support for the hypothesis H2a, since the data shows that analysis is most probable to be followed by synthesis. However, no support was found for the hypotheses H2b and H2c. Namely, the test results combined with the boxplots on Figure 5 reveal that synthesis is most probable to be followed by analysis, rather than evaluation. Moreover, there is no support that evaluation is most probable to be followed by synthesis, as no significant difference was found between the likelihood of analysis and synthesis state transitions appearing after evaluation.

3.3. Transitions Between the Problem and Solution Space

The final set of hypotheses concerns the interaction between the problem and the solution space; that is, whether the synthesis of new problem and solution entities directly follows problem or solution space exploration. The procedure for testing the hypotheses again relies on the probabilities of specific state transition sequences appearing. First, all instances of sequences that result in the synthesis of design entities were singled out (e.g., the two transitions highlighted in the example illustrated in Figure 2). Second, these separated sequences were split into two groups: sequences that result with new problem entities and sequences that result with new solution entities. Third, the two groups were again divided into sequences that start within the problem space, and sequences that start within the solution space. Fourth, the counted instances of state-transition sequences were normalised across the groups to calculate their probabilities. Thus, one can analyse whether it is more probable that new problem entities (requirements, needs, or constraints) are a direct result of problem space exploration (PRO→PS) or solution space exploration (SOL→PS). The same analysis can also be performed for the new solution entities (PRO→SS vs. SOL→SS). The probabilities across all activity fragments and all teams are summarised using the boxplots in Figure 7, whereas all normalised results are shown in Table A3 in Appendix A.

The boxplots show that both the new problem entities and new solution entities are more probable to follow after state transitions related to solution space exploration. This likelihood difference is more pronounced in the case of new solutions. Nevertheless, both differences were shown as statistically significant. The Wilcoxon signed-rank test was again used to compare the sequence pairs plotted in Figure 6 since the Shapiro–Wilk test rejected the normality assumption for the probabilities of sequences resulting in problem synthesis. The Wilcoxon test (paired per protocol fragment) revealed a significant difference at the α = 0.05 level between both the PRO→PS and SOL→PS (p < 0.05) probability pair, and the PRO→SS and SOL→SS (p < 0.001) probability pair.

It must, however, be noted that the significant difference in the case of problem synthesis very much depends on the type of activity. Namely, during ideation activity, it was more probable that problem entities will follow after state transitions related to problem space exploration. In contrast, during concept review, the results are the same as on the overall level. Hence, while there exists empirical support for hypothesis H3a, that the identification of new problem space entities (e.g., new requirements) is more probable to be a result of solution space exploration, it is valid solely on the overall conceptual design process level, where no distinction is made between ideation and concept review. In addition, the results show the identification of solution entities was more probable to be a result of the solution space, rather than the problem space exploration (both overall and for each activity separately). Hence, no empirical support was found for the hypothesis H3b.

4. Discussion

The results are discussed in two parts. In the first part, the results of the hypothesis testing are explained and compared to the insights available within the design research literature. Based on this, the newly obtained insights are consolidated within a simplistic model of problem-solution co-evolution and refinement during team design activities. In the second part, the main research implications and limitations are pointed out and discussed, and further developments are proposed.

4.1. Revisiting Co-Evolution

The intention of the presented research was not to question the validity of existing prescriptive and descriptive design research. Instead, the goal was to complement the literature with insights obtained within the team conceptual design activity context. According to the conducted hypotheses tests, there exist deviations from the assumptions based on the well-established design process models:

• Analysis-intensive fragments of team design activities are likely to exhibit higher proportions of state transitions related to solution space exploration, whereas the synthesis-intensive fragments associate to higher proportions of transitions related to the exploration of the problem space.

• Neither the analysis-synthesis-evaluation nor the synthesis-analysis-evaluation sequences dominate on a fine-grain process level of team design activity.

• Co-evolution has a significantly higher effect on synthesising new problem entities than it does on generating new solution entities.

These three findings can be further elaborated and discussed by laying out an additional perspective of the experimental sessions, that is the temporal analysis of the number of problem and solution entities being generated when designing in teams. Figure 8 shows the change in the cumulative number of both problem and solution entities generated by the four teams throughout the deciles of ideation and concept review experimental sessions. Such a breakdown of design activity protocols into 10 parts is often used to depict the process dynamics during the activity [46,90]. Namely, the splitting of sessions into 10 parts provides an even more detailed description of the dynamics in which new design entities are generated. The cumulative number of design entities includes only the entities generated during the two team activities, although it can be assumed that the participants generated additional entities when working individually. Nevertheless, the here-presented discussion focuses solely on the team activity aspect.

The visualised change in the number of design entities reveals interesting trends in how teams generate new problem and solution entities. During most of the ideation, which has been characterised as a synthesis-intensive activity [43], the number of both the new problem and solution entities increases in a constant pace, with a small decrease in the number of new problems towards the end of the activity. Moreover, the number of new problems practically stagnates throughout concept review. In contrast, the pace of generating new solutions extends also to concept review, with a small decrease towards the end of the activity for three of the teams. These trends are in line with studies that attempted to quantify the focus on problem versus solution space exploration. For example, the studies that measured the problem-solution indicator for team designing [46,90,91] have shown that the proportion of problem-related issues tends to decrease with the progress of the conceptual design activity. A study focused on the phenomena of identifying requirements has shown similar results: The rate at which the requirements are identified and evolve decreases with the progress of the design process [92,93]. Of course, the exact number of new problem entities may vary depending on the methods or tools used (e.g., in case of different tools and techniques for requirements’ formulation [94,95]); however, the described trends tend to persist. The conceptual design process thus moves from divergent to convergent activity [85], that is from the problem-focused task clarification and idea generation towards the solution-focused design reviews [43,96]. In addition, a study of freshman and senior students’ conceptual design process [97] revealed that the focus on problem scoping, that is problem definition and information gathering, has been most persistent from the beginning up until the end of the first half of the conceptual design stage. A similar pattern can be discerned in the protocols of Stempfle and Badke-Schaub [41], who analysed the entire process of teams executing a conceptual design task.

This move from problems to solutions is accompanied by a gradual but significant decrease in synthesis and increase in analysis and evaluation of design entities [43]. More precisely, the initially higher proportions of synthesis cycles (divergent process) get gradually substituted by the alternation of analysis and evaluation design operations (convergent process). Commitments made in the conceptual design stage are mainly functional, and designers typically specify the realisation of the solution as they approach the latter stages of conceptual design [98]. By developing the information on how the design works, not only is the problem reduced, but it is also easier for teams to determine “what can go wrong” [98] and conduct solution evaluation. Studies also show that new requirements during conceptual design result mainly from the analysis of proposed solutions and that the process of requirement identification is closely related to solution synthesis [92]. As design problem formulations get more precise, the increase of solution evaluation is crucial for successful concept development [99].

Interestingly, the alternation of analysis and synthesis is fairly persistent throughout all of the activity protocols. Besides, the sequence analysis has shown that analysis is most probable to be followed by synthesis and vice versa. These probabilities again point out the critical role of analysis-synthesis cycle for concept generation, as proposed by some studies [51,63,74,100,101]. It must, however, be noted that some studies also include evaluation as part of the analysis. Nevertheless, it can be argued that fractions of the design process where sequences of analysis and synthesis design operations alternate [43] appear consistently throughout team conceptual design activities. This is because in design teamwork, the analysis is essential for the better understanding of team members, and leads to progress in design activity, whether it is used as clarification [42,102] or questioning [74]. Hence, achieving common ground (understanding) appears to be a vital ingredient of a team’s creative process during conceptual design activities [103]. As such, the analysis plays an important role both during the divergent activity fragments, where design problems and solution evolve together at a high pace, but also during the convergent fragments, where solutions are evaluated and refined. During the former, problem space exploration often results in new solutions, whereas solution space exploration results in new problems. During the latter, however, the co-evolution episodes occur less often, and new solution entities are mainly a result of convergent refinement.

The above-explained phenomena were summarised in the form of a hypothesised model shown in Figure 9. The model depicts the increase in the number of new problem and solution entities generated during team activities as the conceptual design phase progresses. In the beginning, the rate at which team members generate new entities is high for both design problems and solutions. This is the divergent part of the process, where the pace of design space exploration is fast, and the two spaces co-evolve. Most of the synthesised problem entities follow problem space exploration, and most of the synthesised solution entities follow solution space exploration. However, what characterises this part as dominantly co-evolution (see Figure 8) is that teams switch between the problem and solution space very often, with most of these inter-space transitions being the synthesis of design entities.

As the number of new problem entities starts to converge, teams tend to spend more time exploring the solution space, and the dominant effect of co-evolution diminishes (dotted spiral in Figure 8). At this point, the divergent activity gradually shifts into the convergent refinement of solutions. Here, the process involves solution refinement predominantly. The rate at which new solutions are generated keeps increasing until team members come up and agree on the final concept solution. Unlike co-evolution, the refinement process favours the generation of new solutions primarily as a result of solution space exploration. Co-evolution episodes during the convergent activity appear predominantly as identification of new problem entities based on solution space exploration.

In addition, the divergent co-evolution and the convergent solution refinement are intertwined with the ASE state transitions. As shown in Figure 4, the proportions of ASE fluctuate throughout the time span of team design activities. However, there exist patterns of dominant synthesis transitions during the divergent part of the process and the increase in analysis and evaluation as conceptual design converges. The alternation of analysis and synthesis persists throughout the entire phase, as teams develop a shared understanding in parallel to new entities being introduced within the design space, as demonstrated in the previous study [43].

4.2. Implications, Limitations, and Future Work

Two main areas of implication emerge based on the presented research: design theory and design education. Implications for design theory stem primarily from the newly obtained insights on the nature of designing in teams and the existing models’ limited ability to capture what Cross and Dorst describe as “constant iteration of analysis, synthesis and evaluation processes between the two notional design ‘spaces’ - problem space and solution space” [20] in the context of team design activity. It has been demonstrated that analysis cannot be observed solely through the lens of the design problem, nor can synthesis and evaluation be perceived as processes exclusive to the solution space. The three fundamental state transitions were shown to take part in the exploration of both the problems and solutions. The main reason why the conventional prescriptive models and their descriptive counterparts cannot completely describe the constant iteration of ASE between the problem and the solution space lies primarily in the different definitions and scales (granularity levels) they apply to the three transitions. The definition and scale differences hinder not only modelling of the shared understanding through collective analysis of solution ideas, as it is the case with some of the ASE-based prescriptive models (e.g., [38,39]), but also prevents descriptive models of team design thinking to model the creation and assessment of new problem entities, such as requirements, needs, or constraints (e.g., [40,41,42]). Moreover, different definitions hinder straightforward inter-study comparison, study replication, model reuse, and comparing of notes, since the results always must be viewed in the context of how the authors interpreted analysis, synthesis, and evaluation in their study. The definition of ASE as state transitions applicable to both the problem and the solution space, together with their mapping to different fine-grain process notions from design literature as proposed in this study, may facilitate result comparison and alignment of various design models.

Implications for design education relate mainly to the conventional literature problem-solving processes that should be adjusted to support also solving ill-defined design problems as a collective endeavour. As mentioned in the introduction, the applicability of the conventionally taught procedure of problem analysis followed by solution synthesis and evaluation for solving of ill-defined problems has already been questioned by some researchers [16,17,18,19,20]. Moreover, the high amount of creative stimulation that results from collaborative analysis during team design activities [63] results in an alternation of analysis and synthesis transitions that are typically not envisioned by the prescriptive textbook models. On the other hand, the co-evolution perspective is not much discussed in design education [104], and a study has shown that students are typically unaware of the co-evolution processes [105]. Nevertheless, design ability gained through education should not focus solely on conventional problem-solving but include also the resolving of ill-defined problems and adopting of solution-focused cognitive strategies [106]. This ability to frame problems and solutions through co-evolution is critical for the high level of performance in creative design [107]. It is here argued that teams, in particular, can harness the benefits of co-evolution, expressed in decomposition, exploration, and understanding of the design problems and solutions, due to the analysis-intensive dynamics of team design activity.

Research limitations are largely related to the quantity and quality of data collected through the protocol analysis study. Although the statistical analysis provided insight into significant correlations between ASE and problem/solution space exploration, as well as the differences between the probabilities of particular sequences of state transitions to appear, larger sample sizes and different design activity scenarios are preferable to validate the research findings. Moreover, some of the employed analyses rely on dividing the experiment protocols into multiple fragments, with an assumption of their independency. A better approach would be to conduct experiments with several subsequent conceptual design tasks, where each task would represent an individual datapoint for modelling the progress of the conceptual design phase. Additional studies are thus required to capture data sets sufficient for further in-depth analyses of team design activity. Besides, the presented research examined only the proportions and sequences of verbalised state transitions, neglecting both the possibly significant effects of non-verbalised acts and without investigating the rationale for the transitions to appear. By not taking into account the non-verbalised (e.g., [81]) as well as the non-observable processes, the research grasps only a single layer of a multi-layered phenomenon, such as team designing. Another missing layer is that of contextualisation; that is, the sequences of protocol strings were in no way linked to the design entities they refer to, nor the team members that performed the state transitions. It must also be noted the synthesis of entities in one space following the exploration of the other space has been interpreted as a proxy for the study of co-evolution. However, these are only potential co-evolution episodes [54], since it is not necessarily the case that all of these transitions concern the same context. Experimental studies encompassing more comprehensive observational approaches and focused strictly on the reasoning for particular state transitions could provide a further understanding of the identified patterns. Still, design research that is based on observation, such as the presented protocol study, is often limited in terms of analysis resources, since it typically involves manual data coding. Hence, future efforts should also take into account the possibilities of full or partial automatization of the coding process (see, e.g., [54,108,109]).

Besides the additional work required to address the research limitations, there also exist several possible directions for further developments and research extensions. Given the quantifiable process patterns identified using the state-transition model, future research may focus on predictive modelling of team design activity. The resulting models could, for example, be used to suggest particular state transitions and space exploration foci to maximise the co-evolution rate, based on sampled fragments of the team’s design activity. In addition, future studies might test and extend the state-transition model beyond the conceptual phase and the engineering design domain. The model could also be used to investigate the effects of different types of prominent phenomena that interact with co-evolution during team design activities, such as uncertainty, fixation, framing, analogies, etc.

5. Conclusions

By revisiting some of the well-established assumptions on the nature of the design process in the context of designing in teams, the study provides a new perspective on ASE processes and their role in problem-solution co-evolution. The study has revealed inconsistencies in how analysis, synthesis, and evaluation are defined and interpreted across the literature and demonstrated that these three fundamental processes are critical for the exploration of both the problem and the solution space. Namely, the process sampling by means of protocol analysis and quantification using the proposed state-transition model provided empirical evidence of process regularities in regard to teams performing ASE within and between the problem and the solution space. These findings contradict primarily the prescriptive textbook procedures for problem solving, which teach design students to approach design problems by first analysing the problem thoroughly and formulate criteria, before synthesising a number of solutions and selecting the most appropriate one. While such a procedure is normally adequate for well-defined problems (such as the typical engineering problems, where all requirements can be defined at the start), it does not work well for ill-defined problems, especially when they are solved by teams. The study has shown that the way design teams approach ill-defined problems is by exploring the problem and the solution space in parallel, with the constant iteration of ASE transitions between the spaces. The study also revealed an important role of analysis-synthesis cycles during both divergent and convergent activities, that is co-evolution and refinement, respectively.

This new perspective calls for the standardisation of ASE definitions and for adaptation of prescriptive approaches in a way they could also support solving ill-defined problems and the problem-solution co-evolution in team design activities. The support of design teamwork in the conceptual design phase must facilitate team-level analysis as a stimulus for co-evolution during the divergent activities and as a driver of refinement during the convergent activities. Teams rely on a shared understanding, and achieving common ground is an important ingredient of a team’s creative process. The process of developing a shared understanding of the design problem may, in fact, stimulate both the synthesis of new solutions ideas and problem decomposition (co-evolution), whereas developing an understanding of solution ideas may trigger their evaluation and synthesis (refinement). The study was hence concluded with a hypothesised model of new problem and solution entities appearing within the design space, as a result of a gradual transition between these two process patterns. Once further explored and tested, the new variations of the model can be used to develop appropriate support for design teamwork.

Author Contributions

Conceptualization, T.M.; methodology, T.M., S.Š. M.M.P. and M.Š.; software, T.M.; validation, S.Š. and M.Š.; formal analysis, T.M. and M.M.P.; investigation, T.M., S.Š. and M.Š.; resources, T.M., S.Š., M.M.P. and M.Š.; data curation, T.M.; writing—original draft preparation, T.M.; writing—review and editing, T.M., S.Š., M.M.P. and M.Š.; visualization, T.M.; supervision, S.Š. and M.Š.; project administration, M.Š.; funding acquisition, M.Š. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by Croatian Science Foundation project IP-2018-01-7269: Team Adaptability for Innovation-Oriented Product Development - TAIDE (http://www.taide.org).

Acknowledgments

This paper reports on work funded by the Croatian Science Foundation TAIDE project. The authors would like to thank Philip Cash (TU Denmark) for providing the multimedia data of the experiment sessions.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table A1

Distribution of state transitions throughout all sessions and session fragments, based on the proportion of ASE and problem/solution-related verbalisations.

Team/Session/Part			A	S	E	PRO	SOL
T1	I	1	0.416	0.393	0.191	0.472	0.528
		2	0.573	0.270	0.157	0.607	0.393
		3	0.652	0.270	0.079	0.584	0.416
	CR	1	0.238	0.495	0.267	0.152	0.848
		2	0.288	0.452	0.260	0.048	0.952
		3	0.248	0.476	0.276	0.114	0.886
T2	I	1	0.451	0.402	0.147	0.353	0.647
		2	0.447	0.320	0.233	0.311	0.689
		3	0.510	0.284	0.206	0.245	0.755
	CR	1	0.337	0.361	0.301	0.157	0.843
		2	0.417	0.369	0.214	0.071	0.929
		3	0.217	0.566	0.217	0.169	0.831
T3	I	1	0.500	0.298	0.202	0.228	0.772
		2	0.451	0.283	0.265	0.513	0.487
		3	0.333	0.386	0.281	0.237	0.763
	CR	1	0.378	0.387	0.234	0.171	0.829
		2	0.304	0.429	0.268	0.045	0.955
		3	0.243	0.495	0.261	0.108	0.892
T4	I	1	0.535	0.302	0.163	0.326	0.674
		2	0.523	0.267	0.209	0.314	0.686
		3	0.488	0.314	0.198	0.256	0.744
	CR	1	0.338	0.378	0.284	0.081	0.919
		2	0.370	0.397	0.233	0.164	0.836
		3	0.297	0.459	0.243	0.149	0.851

Legend: A—analysis, S—synthesis, E—evaluation, PRO—problem-related, SOL—solution-related.

Table A2

Distribution of ASE state-transition sequence probabilities throughout all sessions and session fragments.

Team/Session/Part			A→A	A→S	A→E	S→A	S→S	S→E	E→A	E→S	E→E
T1	I	1	0.257	0.514	0.229	0.432	0.324	0.243	0.529	0.471	0.000
		2	0.125	0.750	0.125	0.333	0.471	0.196	0.357	0.571	0.071
		3	0.042	0.875	0.083	0.351	0.561	0.088	0.286	0.714	0.000
	CR	1	0.308	0.346	0.346	0.600	0.040	0.360	0.714	0.214	0.071
		2	0.277	0.383	0.340	0.600	0.133	0.267	0.593	0.296	0.111
		3	0.360	0.360	0.280	0.577	0.038	0.385	0.607	0.250	0.143
T2	I	1	0.317	0.463	0.220	0.457	0.413	0.130	0.400	0.533	0.067
		2	0.182	0.576	0.242	0.457	0.304	0.239	0.250	0.542	0.208
		3	0.069	0.690	0.241	0.346	0.481	0.173	0.450	0.350	0.200
	CR	1	0.200	0.433	0.367	0.500	0.286	0.214	0.360	0.320	0.320
		2	0.161	0.548	0.290	0.400	0.371	0.229	0.667	0.222	0.111
		3	0.522	0.261	0.217	0.611	0.222	0.167	0.667	0.111	0.222
T3	I	1	0.118	0.588	0.294	0.474	0.333	0.193	0.174	0.739	0.087
		2	0.219	0.375	0.406	0.333	0.412	0.255	0.267	0.600	0.133
		3	0.186	0.395	0.419	0.500	0.316	0.184	0.500	0.281	0.219
	CR	1	0.186	0.535	0.279	0.548	0.238	0.214	0.462	0.346	0.192
		2	0.313	0.417	0.271	0.471	0.147	0.382	0.567	0.300	0.133
		3	0.418	0.309	0.273	0.519	0.037	0.444	0.607	0.321	0.071
T4	I	1	0.154	0.769	0.077	0.326	0.435	0.239	0.500	0.429	0.071
		2	0.174	0.609	0.217	0.422	0.311	0.267	0.056	0.889	0.056
		3	0.111	0.519	0.370	0.415	0.439	0.146	0.353	0.588	0.059
	CR	1	0.286	0.357	0.357	0.520	0.200	0.280	0.333	0.476	0.190
		2	0.172	0.517	0.310	0.593	0.148	0.259	0.471	0.471	0.059
		3	0.471	0.294	0.235	0.476	0.190	0.333	0.444	0.444	0.111

Legend: A—analysis, S—synthesis, E—evaluation, → from-to sequence.

Table A3

Distribution of probabilities of state-transition sequences resulting in the synthesis of problem and solution entities, based on the space of the first transition in the sequence.

Team/Session/Part			PRO→PS	SOL→PS	PRO→SS	SOL→SS
T1	I	1	0.667	0.333	0.348	0.652
		2	0.852	0.148	0.261	0.739
		3	0.781	0.219	0.385	0.615
	CR	1	0.200	0.800	0.250	0.750
		2	0.500	0.500	0.071	0.929
		3	0.000	1.000	0.208	0.792
T2	I	1	0.800	0.200	0.194	0.806
		2	0.467	0.533	0.387	0.613
		3	0.588	0.412	0.229	0.771
	CR	1	0.000	1.000	0.125	0.875
		2	0.000	1.000	0.065	0.935
		3	0.500	0.500	0.063	0.938
T3	I	1	0.357	0.643	0.262	0.738
		2	0.762	0.238	0.300	0.700
		3	0.667	0.333	0.250	0.750
	CR	1	0.200	0.800	0.189	0.811
		2	0.000	1.000	0.067	0.933
		3	0.000	1.000	0.136	0.864
T4	I	1	0.500	0.500	0.250	0.750
		2	0.571	0.429	0.267	0.733
		3	0.333	0.667	0.250	0.750
	CR	1	0.000	1.000	0.130	0.870
		2	0.000	1.000	0.227	0.773
		3	0.000	1.000	0.263	0.737

Legend: PRO—problem-related, SOL—solution-related, PS—problem synthesis, SS—solution synthesis.

Figures and Tables

Figure 1. Illustration of a state-transition sequence. Adapted with permission from [43], Springer, 2019. The circles represent problem-related entities, whereas the squares represent the solution-related entities.

Figure 2. An example of an activity protocol excerpt where both the problem and the solution space are populated with design entities (in some state i). Analysis, synthesis, and evaluation of problem- (circles) and solution-related (squares) design entities as transitions between the states of the explored design space (i, i + 1,…). Co-evolution episodes correspond to synthesis moves between the spaces. Legend: PA—problem analysis, PS—problem synthesis, PE—problem evaluation, SA—solution analysis, SS—solution synthesis, SE—solution evaluation.

Figure 3. Triangular proportion visualisation of analysis, synthesis and evaluation transitions using two visualisation variables: distance from the centre of gravity r (maximal distance is R = 1) and angle δ (0–360°).

Figure 4. Triangular proportion visualisation of ASE transitions and the associated proportions of problem-related verbalisations throughout ideation (I1-I2-I3) and concept review (CR1-CR2-CR3) activities for all four teams.

Figure 5. Triangular proportion visualisation of ASE transitions and the associated proportions of problem-related verbalisations sampled from sessions of all four teams.

Figure 6. Boxplots of sequence probabilities based on the previous state transition.

Figure 7. Boxplots of proportions of synthesis of problems (left) and solutions (right) based on the space explored during the previous state transition.

Figure 8. Cumulative number of problem and solution entities generated by all teams throughout the ideation and concept review experimental sessions.

Figure 9. Illustration of co-evolution and refinement in relation to proportions of ASE and rate of new problem and solution entities that design teams generate throughout the conceptual design phase.

Table 1

Mapping of various fine-grain design process notions from design literature onto analysis, synthesis, and evaluation as transitions between the states of the explored design space.

Analysis	Analyse [20], [38], [40], [41], [47], [50], [60], [61]; Simulate [38], [50], [62]; Clarify [41], [63]; Acquire [64]; Calculate [62]; Compare [62]; Correct [63]; Interpret [64]; Understand [14]; Read [65]; Repeat [65]; Request [65]
Synthesis	Synthesise [20], [38], [40], [47]; Generate [8], [14], [41], [50], [60], [61], [62], [66], [67]; Define [28], [42]; Elaborate [60], [65]; Gather [61], [68]; Modify [65], [67]; Add [65]; Create [28]; Compose [50]; Formulate [40]; Model [61]; Patch [62]; Propose [65]; Refine [62]; Redefine [50]; Select [62]
Evaluation	Evaluate [8], [14], [20], [28], [38], [40], [41], [47], [61], [62], [65], [67]; Decide [14], [28], [41], [61]; Select [8], [67]; Accept [41], [62]; Verify [60]; Reject [62]; Suspend [62]; Qualify [65]; Justify [65]

Table 2

The coding scheme used for annotating the verbalisations related to design space exploration, planning, and other issues.

Code	Description	Coders’ Reliability (ĸa)
Problem analysis (PA)	Verbalisations regarding the understanding of problem entities	0.72
Problem synthesis (PS)	Verbalisations resulting in the appearance of new problem entities	0.78
Problem evaluation (PE)	Verbalisations regarding the appropriacy assessment of problem entities	0.78
Solution analysis (SA)	Verbalisations regarding the understanding of solution entities	0.70
Solution synthesis (SS)	Verbalisations resulting in the appearance of new solution entities	0.79
Solution evaluation (SE)	Verbalisations regarding the appropriacy assessment of solution entities	0.78
Process (PROC)	Verbalisations regarding the process of the activity	0.93
Other (O)	Verbalisations that cannot be annotated with the codes defined above	0.95

Table 3

Spearman’s rank-order correlation coefficient test of hypotheses H1a, H1b, and H1c, together with the scatter plots of proportions of ASE and problem/solution-related verbalisations associated with the sampled data points.

Correlation	Scatter Plot
Strong negative correlation between performing analysis and problem-related state transitions:rS (22) = −0.649, p < 0.001	[Image omitted. Please see PDF.]
Strong negative correlation between performing synthesis and solution-related state transitions: rS (22) = −0.749, p < 0.001	[Image omitted. Please see PDF.]
Strong positive correlation between performing evaluation and solution-related state transitions: rS (22) = 0.672, p < 0.001	[Image omitted. Please see PDF.]