A Post-Training Study on the Budgeting Criteria Set and Priority for MALE UAS Design

Abstract

A recent study proposed a systematic “(budgeting) knowledge discovery educational framework” (BKDEF). This BKDEF is focused on guiding staff training courses for enhancing the ability to allocate the “large but limited” budget for single, high-cost product design. However, except for its initial application to support the budget planning for the next generation fighter design, the framework’s effectiveness is still awaiting further scrutiny. This study fills the gap by providing the “second application” of BKDEF, which is to support another similar decision for designing the medium-altitude long-endurance unmanned aerial system (MALE UAS). This paper verified the effectiveness of the framework through an empirical application and obtained the knowledge required to allocate a budget for MALE UAS design following the group-opinion basis. In addition, the original analytical style for the last “decision analysis” phase of BKDEF, which included pure quantitative analytical items in order to understand the similarities and diversities in the individual opinions, was replaced by a comparative study to discover the homogeneity and heterogeneity between the two budgeting decisions in a larger scope. As a consequence, the two criteria sets did not overlap despite both decisions being related to military aircraft design. The absolute weights for the MALE UAS design criteria were more balanced than those for the air-superior fighter design, even if the size of the criteria set was larger. The results pave a way for future studies on how other military aircrafts are designed, as more confidence about the use of a BKDEF can be gained from increasing applications, thus more insightful aerospace knowledge can be exploited in comparisons with these works.

1. Introduction

For organizations that focus on developing new high-tech and high-investment cost products, the budget is usually large. However, the budget is often insufficient to cover all of the desired functions, even if it is quite large relative to the budget allocated for developing other products [1,2]. This is also true when developing high-cost, high-tech products [3,4] such as a medium-altitude long-endurance unmanned aerial system (MALE UAS) [5,6].

To compete for resources, departments in charge of designing different functions of such a product often insist on their personal interests because of departmental selfishness, i.e., they often think that the functionality they develop is the most important one. Thus, disagreements arise when there are such resource competitions. Because of this, the decision makers (DMs) typically have to be responsible for the coordination in order to leverage, utilizes, and allocate the relevant budget resources appropriately.

In order to pursue a “satisfactory” (rather than “the best”) investment, this type of “large but limited” budget should always be allocated prior to the real research and design (R&D) work begins, because the R&D work that follows is often irreversible. This usually involves suitable identification of the product design criteria [7,8]. In addition, based on the belief that a budget should be first allocated to critical product functions of key interests or the core technologies to be developed, these functions, which represent the design criteria of the final product, must be prioritized. Moreover, since the age of scientific decision-making began in the 1970s, it has been recommended that the priority be determined in a scientific manner. As can be imagined for/in the studied decision context, doing so not only mitigates any internal pressures but avoids “wrong decisions” and provides a numerical basis for guiding the allocation and management of the budget.

Fortunately, in management science, the mature development of multi-attribute decision making (MADM) methods may provide not only scientific prioritization over the product design criteria but also a ratio-scale-based assessment about the importance of each criterion. However, in an R&D institution that develops high-tech, high-cost products, the staff or DMs who plan budgets or make eventual decisions for budget allocation usually possess an engineering background and include some with the ambition to devote themselves to R&D and become outstanding engineers throughout their lives. They are not quite familiar with the issues about how to allocate the entire “large but limited” budget resource in terms of the MADM method. Therefore, staff training courses that increase the “decision ability” (i.e., the ability to understand the true preferences of those strategic DMs and to perform the relevant decision analysis work) are required in order to support these budget allocation decisions.

As an overview to these training courses with this special purpose, a recent study proposed a systematic “(budgeting) knowledge discovery educational framework” (BKDEF) [9]. The framework dissected one such training course into four phases designed to help the course participants:

• (1). An initial phase to identify the set of design criteria via a thorough review of both the academic literature and the industrial materials.

• (2). A second phase to construct a “decision hierarchy”, wherein the involved design criteria are organized (mounted) with respect to (w.r.t.) several constructs that are mounted under the decision goal, which is the suitable budget allocation. The Delphi method is suggested for this phase to affirm this decision hierarchy with several rounds of interviews or e-mail inquiries.

• (3). A third phase follows the analytic hierarchy process (AHP) to design expert questionnaires (one for polling the weights of the constructs w.r.t. the decision goal, while others poll the weights of the criteria w.r.t. each construct) according to the decision hierarchy, to investigate the DMs’ opinions by using these questionnaires, and to obtain the pairwise comparison matrices using these matrices. For each DM, a construct weight vector and several criteria weight vectors (which are called the “CWVs” in the study) are obtained.

• (4). The final phase conducts any further decision analysis.

The study that demonstrated the proposed BKDEF [9] was applied to a training course for the staff in an institution making the budget plan for next generation fighting aircraft R&D. This should have made a novel contribution because of the “first application” of BKDEF for both the educational training field and the aerospace industry. However, there are still only a few application studies. Thus, to provide further empirical evidence for BKDEF, this study provided its "second application" to another training course that was held for budget planning prior to the R&D of a MALE unmanned aerial vehicles (UAV) rather than the next generation fighter. This also verified the effectiveness of BKDEF in similar training courses.

Based on this “second application”, knowledge pertaining to the considered criteria in the design of the MALE UAS and the determination of the priority for these criteria was explored during the training. However, the materials used for the last phase of training were altered. In the previous study, methods from the field of data-driven decision-making (DDDM) were used to explore the homogeneity and heterogeneity that exists in the DMs’ group based on the CWVs that were assessed individually for the DMs.

However, this analytical purpose was no longer the purpose of the final (fourth) phase of the course held for MALE UAS R&D budget allocation because the DMs were eager to know the differences between the related constructs and the criteria sets for fighting aircraft design and MALE UAV design, as well as the difference in the priority sequence for the design criteria. Therefore, methods for making relevant comparisons were also taught.

Although studying the progress of the “second application” of BKDEF and comparing the teaching events that occurred between the “first application” and “second application” led to meaningful topics in pedagogy and in-service education training, they were not as interesting to study as the results obtained after the students’ exercise. These results were sufficiently representative for us to explore the practical knowledge about MALE UAS design. MALE UAS has become a popular topic in aerospace due to the important role it continues to play for the military. Furthermore, comparing this set of criteria against the one that was considered during budget planning for fighter design (and the two priorities determined from them) was very meaningful for the aerospace and the air-force industry. In other words, while the importance of “studying the results and comparing them across two studies” is addressed in this paper, due to the limited space, topics about the training course itself are limited because their contributions were relatively minor.

Section 2 reviews the literature about UAS and its design issues and addresses concerns from the R&D institutions. Following this, the set of design criteria for MALE UAS, which was identified through literature study, is presented. Section 3 introduces the systematic training process that was refined based on the original BKDEF in order to evaluate the relevant budgeting decision criteria for MALE UAS R&D. Since the main quantitative method of BKDEF (which is the AHP used in the third phase) is reviewed at the end of the former section, it is omitted from the methodological section. Section 4 reveals the main results obtained after the training, including the decision hierarchy that was confirmed using the Delphi method and the individual set of CWVs that were assessed for each DM using AHP, as well as the “average opinions” aggregated w.r.t. the total budget allocation decision and each construct. Section 5 presents the results from the cross-study comparison and discusses the implications. The results are compared with those used for designing a next generation fighter. This is, in fact, an advanced knowledge discovery work focused on describing the different roles of fighting aircrafts and MALE UAS in an air war. The conclusion of this study and the recommendations for future research are made in Section 6.

2. Materials and Methods: A Literature Study

For the budgeting decision to be analyzed for MALE UAS design and for the comparisons to be made, the relevant literature is reviewed step by step. Section 2.1 gives a review to UAS. Section 2.2 studies the classifications of UASs. The design (functional) factors considered for the R&D of the UAS and the decision hierarchy they formed are presented in 2.3. The main quantitative model used to obtain the results in Section 4, AHP, is reviewed in Section 2.4. For the factors that were considered for the next generation fighting aircraft design and compared in Section 5, the review made by Chi is cited [9].

2.1. The R&D of UAS: An Overall Review

Unmanned aircrafts are formally called unmanned aerial vehicles. The United States (U.S.) Federal Aviation Administration (FAA) clearly defined the term “UAV” as “a device that is used or intended to be used for flight in the air that has no on-board pilot. This includes all classes of airplanes, helicopters, airships, and translational lift aircraft that have no on-board pilot.”

Generally speaking, the types of unmanned UAV include remotely piloted aircraft (RPV), drones, robotic aircraft, and unmanned combat aircraft. With the abovementioned definition, UAVs usually do not carry a human operator and can fly autonomously or are remotely piloted. In addition, UAVs can have diverse capabilities with different designs, and they can be used in military missions and civilian/commercial applications [10]. Since many missions are dull, dirty, or dangerous (3D) for aviators, UAVs are more suitable for accomplishing certain missions. For example, many types of UAV, small and large, have been widely used by government departments or research entities to execute different tasks and research jobs [11].

Moreover, a UAV can be integrated with ground control stations and data links to form a UAS [12]. Therefore, a UAS involves command, control, and communications (C3) systems, and it is necessary for UASs to support people who control the UAVs [5].

Typically, the UAS has the same functional components to manage manned aircrafts but with additional airborne components. Because the UAVs are designed without on-board personnel, the interface between the crew (as a subsystem) and the aircraft control unit is replaced by an electronic intelligence and control subsystem. Other elements—launch, landing, recovery, communication, support, etc.—also have their equivalents in the unmanned system. In addition, when organized as a UAS, “automatic intelligence” should be performed by the UAVs. For example, it should be able to communicate with the control station intelligently to return payload data (such as electro-optical or thermal images, as well as its main status information such as location, airspeed, heading, and height) automatically. Additionally, it might also transmit information about its real conditions, e.g., “housekeeping data” such as the amount of fuel it has and the temperature of its internal components (e.g., engine or electronics) among others [13]. All these require automatic intelligence.

Actually, a UAS can be regarded as a system that includes several subsystems, including those from aircrafts (commonly referred to as UAVs), payloads, control stations (and usually other remote stations), aircraft launch and recovery, support, communication, transmission, etc. [13].

With advanced navigation and communication technologies, the UAS has become a “new capability” that the government (public) and commercial (civil) aviation sectors could utilize [14].

However, understanding the characteristics of different UASs is important for making choices at the beginning of the design process. Compared to a short-range UAS, a long-range and armed UAS is more complex, and the development of it is usually more expensive. Thus, for civilian use, typically a short-range UAS is a more attractive option, while long-haul UAS is primarily for military use because of the complex functions it includes [15]. These differences are summarized in Table 1 [13].

As can be seen from this table, UAS have been widely applied to many domains. Specifically regarding the use of UAS in military operations, a manned system can accomplish many, if not all, of the same goals. However, UAS can reduce the risk in air combats by the ability to support the intelligence of command and control [e.g., the signal intelligence (SIGINT)], target positioning (i.e., precise target designation), and weapon delivery. Additionally, it can improve situational awareness in the air-and-ground combat. As such, it not only mitigates possible risks while managing troops but also facilitates fatal strikes within a very short response time in dangerous areas [12]. This is the reason why UAS is an emerging technology with the potential to be a “revolutionary war weapon”.

The advancement of UAS can be traced back to over a decade ago. Although the U.S. military began studying UAV in 1917 and UAVs were tested during World War I, it did not play a combat role until the Vietnam War. Afterwards, the advanced communication technologies increased the bandwidth of military communications satellites (as well as the technological developments for navigation), which changed “UAVs as a UAS” and made the remote operations of UAS more practical. In addition, the geographical nature that impacted the wars in Iraq and Afghanistan also increased the demand for UAS in order to identify, locate, and attack hidden targets using continuous surveillance with rapid strikes while minimizing collateral damage. In these applications, UAS provided an asymmetric and relatively impeccable technical advantage in these conflicts [12].

In the past two decades, UASs have played key roles for the army in non-military actions, e.g., supporting humanitarian rescue operations in Haiti, or for the purposes of mine detection and chemical/biological/radiological/nuclear (CBRN) reconnaissance. In addition, UASs have made key contributions to the global war on anti-terrorism. On the pro-rata basis, Figure 1 shows how many UASs were equipped against the manned aircrafts for the U.S. military in 2010. Surprisingly, this ratio was 4:6 about 10 years ago (data source: The Military Balance 2010; Weatherington brief).

Schaus and Johnson [16] indicate that the military uses of UAS are increasing in countries around the world. The Stockholm International Peace Research Institute (SIPRI) has identified that at least 68 countries have developed or acquired UASs for military use [15]. Further evidence can be found in the military economy, which is also a system of “supplies and demands”. The countries demanding (importing) UAS technology are shown in Figure 2. In contrast, the countries investing in the R&D of UAS and supplying (exporting) the technologies are shown in Figure 3 [17].

In addition to the “supply chain” of armed UAVs, there are also stats and forecasts about the R&D investments in UAS. Gertler [12] mentioned that the U.S. Department of Defense (DoD) had increased the investment scale for UAS R&D every year. The scale was $284 million in 2000, while it was up to $3.3 billion in 2010. In addition, 2017 market research conducted by Teal Group estimates that the global scale of drone UAV production will increase from $4.2 billion in 2017 to $10.3 billion in 2026, and the spending will increase by another $26 billion over the next decade [18].

The Teal Group’s report also points out that, in terms of global military budgets, the UAS sector is expected to be one of the areas with the most growth in dynamics, and the rapid spending growth in the past decade will continue. In the next decade, it is expected that U.S. procurement will grow moderately, and growth will shift to the international market. The special-purposed unmanned combat aircrafts (UCAVs) are also expected to drive growth in the next decade. These data are shown in Figure 4 [19].

Moreover, according to an analysis by the Association for Unmanned Vehicle Systems International (AUVSI), the U.S. military proposed spending $9.6 billion on UASs in 2019, an increase of 28% compared to 2018. AUVSI also noted that the budget funding for unmanned technologies is comprised of about 1.4% of the total budget of the U.S. DoD in 2019 since each military service—U.S. Navy, U.S. Air Force, and U.S. Army—requested a funding increase for unmanned systems [20].

According to the above study, the R&D of UAS is a prominent subject in the military field, and the investment remains an important issue. These points address the main subject of this study, which is to explore knowledge about UAS R&D in order to guide the budget allocation process.

2.2. UAS: Classification

As discussed in 2.1, UASs have been used in many countries all over the world for both military and civilian domains. Currently, at least 90 countries or non-state groups are known to operate UASs [17]. However, the different types of UASs have different purposes of use. For example, micro and small UASs are usually used in low-altitude and uncontrolled airspace. Often, a light-weighted UAS is less than 150 kg, which is suitable for executing many monitoring tasks for practical industrial applications.

Grimaccia et al. [21] proposed an effective method to classify UASs according to their functional capabilities. However, the U.S. DoD offered another classification, while U.S. National Aeronautics and Space Administration (NASA) provided yet another “classification matrix” [11]. These are shown in Table 2 and Table 3, respectively. In Table 2, these classifications are justified based on weight, flying altitude, mission radius, and endurance [12,22]. In Table 3, the classifications are defined based on weight, airspeed, and type. However, although there are different nomenclatures, the weight can be used as the sole common classification criterion because the other properties of a UAV are usually associated with weight.

Nevertheless, Gupta et al. [5] argued that there is no common classification for a UAS due to the wide variety of capabilities, size, and operating characteristics available. This means the categories can only be listed through an understanding based on the “UAV types” seen to date. The categories, as summarized, are listed in Table 4 along with their associated parameters. These include maximum gross take-off weight (UAV with payload), altitude for normal operation, the radius of the mission, endurance, normal employment, and purposes for use. However, classifying any UAV according to this categorization is also a personal judgment because the “Category” column is the index of the table, while other parameters can be retrieved upon such judgment.

Moreover, as can be summarized from [5,13], these studies also offered another style of categorization, although the categorization can also be done by “text descriptions” instead of the numerical parametric tables mentioned above. These are summarized in Table 5.

Finally, of most importance is that Austin [13] thought that, although all UASs are more complex than a UAV, they could be categorized by their capability or the size of their performance tasks as UAVs that are included within the UAS. He also points out that the categorization rules are subject to change, as technological advances may allow smaller UAVs to be upgraded to play the role of larger UAVs. In other words, the boundaries among the abovementioned classifications or categories may change over time. However, the terms used by Austin in 2011 for classifying the UASs were still effective at the time this paper was written, i.e., from the HALE UAV with a ≥35m wing span down to the NAV with a 40 mm span. The definition of “MALE UAS”, which is relevant to the subject of this study, is also clearly described.

2.3. The Budget Allocation Criteria for the Design of MALE UAS

In the cases where UASs were used in military actions, the MALE UAS outperformed others for the U.S. military, as mentioned in Section 2.1. It succeeded in target indication, military mapping, target surveillance, damage assessment, electronic warfare, battlefield reconnaissance, low-speed/fixed-target attack, communication relay, information attacks, and targeted killings, to name a few examples [23]. The MALE UAS technology is also booming in Europe, where four countries—the United Kingdom (UK), France, Italy, and Germany—are currently operating MALE UASs, while six countries are acquiring unarmed MALE drones, including the Netherlands, Spain, Belgium, Switzerland, Poland, and Greece [24].

According to the literature described in 2.2, a MALE UAS is one of the categories of UASs whose size is much larger than the other categories (except for a HALE UAS). With reference to Table 2, Table 4 and Table 5, NASA Ikhana Predator-B and IAI Heron (Machatz-1) are well-known MALE UASs. The MALE UAS is to be operated above an altitude of 9000 m, thus it requires an advanced aerodynamic design and a control system. It must fly hundreds of kilometers from its ground station for many hours (20–40 h). It should be able to be ignited in a short time and should provide (almost) real-time geo-corrections and multi-spectral images. In addition, in order to carry a variety of sensors, including electro-optic (EO), infrared (IR), and synthetic aperture radar (SAR), it should allow for a high payload [25]. Thus, compared to other UASs with low altitude and/or short endurance abilities, a MALE UAS is more powerful and is mainly used for military purposes, and its design is rather complex. Thus, many countries are now engaged in R&D projects to develop a MALE UAS [24], either to utilize the superior performance of MALE UAS for their own military use or to profit from the military economy (e.g., supplying the demands, although there is an entry barrier because the relevant R&D work is quite complicated and requires an industrial base).

According to the thorough literature review made for the recent world developments of MALE UAS, in both academic and industrial literatures, an “excellent MALE UAS” can be justified in 18 relevant functional aspects, which includes service ceiling, endurance, payload performance, redundant flight control system, avionics system, external payload, system-wide integration, key component design and manufacture, information integration, information transmission, reliability of redundant flight control system, reliability of avionic systems, key component acquisition, DMSMS management, logistic support, system architecture and component expandability, design continuity, and system performance growth. These surely comprise the criteria set critical for designing a new type of MALE UAS, which is also the knowledge explored at the first stage of using the BKDEF.

Factors in this criteria set are further stratified w.r.t. four involved constructs for MALE UAS R&D, i.e., equipment performance, technology capability, logistic support, and system growth, thus a “decision hierarchy” can be organized based on the expert opinions using the Delphi method, as seen in Figure 5. This is also the “evaluation model” for making the budget allocation decision.

The operational definitions of the 18 design factors are shown in Table 6. Subsequent discussions present the foundation of these definitions.

The design of a MALE UAS should consider the factors listed above, but they can be contemplated based on the versatile tasks it has performed in military operations.

Firstly, in aerodynamic R&D, Marqués et al. [26] described that there are several “technical specifications” to increase the performance of a UAV, e.g., service ceiling, endurance, payload, speed, take-off distance, operational radius, and weight, among others. Therefore, “service ceiling” (F1), “endurance” (F2), and “payload” (F3) are the important factors to consider in a UAS design. Next, due to the rapid development of microelectronics and sensors, the trend toward miniaturization of electronic products will enable UASs to have high-performance avionics systems [27]. Tokar [28] also mentioned that the main activity of the U.S. DoD is to develop highly complete, security-critical information system architecture for UASs and fighters. Both studies found that a MALE UAS should be equipped with a fully functional “avionics system” (F5). In addition, Merlin [29] mentioned that the redundant flight control system and triplex avionics in an Ikhana MALE UAS may increase consistency and safety. In a reliability assessment for UAS, Vanek [30] addressed the concept that the MQ-9 Reaper exhibited better reliability than the Predator because of its triple redundant flight control system. Both papers found that the “redundant flight control system” (F4) is a key factor for the design of a MALE UAS. From the above review, which was mainly made for equipment and performance (C1), the “reliability of redundant flight control system” (F11) and “reliability of avionics system” (F12) should also be critical for a MALE UAS.

Francis [31] thought system integration would be a significant challenge for future UAS design. Additionally, Kendoul [32] claimed that system integration is one of the most critical factors for assessing the maturity of UAS technology. Therefore, “system-wide integration” (F7) should also be an addressed factor. Yang [33] mentioned that “design and manufacture of key components” (F8) are quite important for the prototype design of any new product. The budgeting decisions should also address this factor because whether or not the key components can be successfully made usually affects the final design. Therefore, many of the tasks that are to be performed by the MALE UAS, such as surveillance, reconnaissance, and precise target assignment, require sufficient bandwidth to transmit a large amount of information [12]. This requires the immediacy and accuracy of the information transmitted. In addition, the “U.S. Air Force UAS Flight Plan 2009-2047” clearly pointed out that each type of UAS must be equipped with a function module to perform information integration tasks [34]. From this review, it can also be asserted that “information transmission” (F10) and “information integration” (F9) are important “technological capabilities” (C2) in designing a MALE UAS.

The Australia DoD believes that logistics support is one of the main factors that might impact new weapon systems (e.g., UAS) whether or not they can deploy rapidly [35]. In addition, Chiesa et al. [36] emphasized the importance of complete logistics support, as it optimizes the resources and system availability and reduces system operating costs. Since the “U.S. DoD DMSMS (diminishing of manufacturing sources and material shortages) Acquisition Guidelines” emphasized that DMSMS management should be done to avoid any issues in a weapon systems’ continuous maintenance, the impact of a DMSMS should be minimized [37] after the initial key component acquisition. Therefore, for MALE UAS R&D, “key component acquisition” (F13), the logistics support (F15), and the management of DMSMS (F14) are once again the important issues that should be considered. Together with the redundant flight control system (F11) and the reliability of avionic systems (F12) factors, these have formed the factors w.r.t. the “overall logistics support” construct (C3).

As for the last construct, the “system growth” construct (C4), in the study of R&D performance assessment, Samsonowa [38] suggested that good expandability and scalability are two important performance measurement indicators in R&D. Specifically for the design of a UAS, Perhinschi et al. [39] studied flexibility and expandability through modular architecture and standard internal interfaces between components, which allowed for further developments, additions, and modifications in the UAS architecture. These works have shown that “system architecture and component expandability” (F16) is important for the future development of UAS. Egan [40] addressed the role of “design continuity” (F17) in a new solution important for product design, while Han et al. [41] also asserted the opinion that it should be considered for any novel design. Among other studies in system development, Freitas and Wilcke [42] believed that the “system performance growth” (F18) is an important design factor, and Huang et al. [43] mentioned that it is a key factor that should be considered in the R&D of semiconductor technology. As such, for the design and R&D of a MALE UAS, it can be concluded that (F16), (F17), and (F18) are the important factors that one should consider for the “system growth” construct (C4).

2.4. The AHP Method

Following the initial study, which proposed the BKDEF [9], the main quantitative method taught (and used by the trainees) to poll the individual opinions of the DMs was reviewed. AHP is a well-known MADM approach that has been applied for decades [44]. It was proposed by Saaty [45]. The AHP process can be roughly divided into two phases; the first phase determines the weights of the criteria (and also the constructs, when applicable), and the second phase determines any priority over alternatives. However, in BKDEF, only the former phase is sufficient for understanding the DM opinions for allocating a “large but limited” budget.

To evaluate the preferential structure of the entire opinion group, the AHP performed a survey of the CWV-determination process and the associated consistency analysis for each individual DM. AHP assumes that w.r.t. some decision construct or decision goal, the relative importance of the involved criteria can be stated by pair-wise comparison of two criterions at a time during the survey. Thus, the results may form a pair-wise comparison matrix, M_nxn, where n is the number of the criteria, as:

(1)$M=\left[\begin{array}{cccc}{m}_{11}=1& m_{12}& \cdots & m_{1n}\\ m_{21}& m_{22}=1& & \\ ⋮& & \ddots & \\ m_{n1}& & & m_{nn}=1\end{array}\right],\forall i,j,i\ne j,m_{ij}\in \left\{\frac{1}{9},\frac{1}{7},\frac{1}{5},\frac{1}{3},1,3,5,7,9\right\}$

Then, the process that follows is the determination of the CWV (i.e., where the elements are the criteria weights assessed w.r.t. a construct/goal). Based on the data in the above square matrix, M, a column-sums vector of this matrix can be calculated as:

(2)$V={\left[\begin{array}{cccc}{\displaystyle \sum _{i=1}^n{m}_{i1}}& {\displaystyle \sum _{i=1}^n{m}_{i2}}& \cdots & {\displaystyle \sum _{i=1}^n{m}_{in}}\end{array}\right]}_{(1\times n)}$

Dividing each column in M using the vector elements, another square matrix, M’, is obtained as:

(3)$M^{\prime }=\left[\begin{array}{cccc}{m^{\prime }}_{11}=1/{\displaystyle \sum _{i=1}^n{m}_{i1}}& {m^{\prime }}_{12}=m_{12}/{\displaystyle \sum _{i=1}^n{m}_{i2}}& \cdots & {m^{\prime }}_{1n}=m_{1n}/{\displaystyle \sum _{i=1}^n{m}_{in}}\\ {m^{\prime }}_{21}=m_{21}/{\displaystyle \sum _{i=1}^n{m}_{i1}}& {m^{\prime }}_{22}=1/{\displaystyle \sum _{i=1}^n{m}_{i2}}& & \\ ⋮& & \ddots & \\ {m^{\prime }}_{n1}=m_{n1}/{\displaystyle \sum _{i=1}^n{m}_{i1}}& & & {m^{\prime }}_{nn}=m_{nn}/{\displaystyle \sum _{i=1}^n{m}_{in}}\end{array}\right]$

The CWV is thus determined by calculating the row-sums vector of M’, which is:

(4)$CWV={\left[\begin{array}{cccc}{\displaystyle \sum _{j=1}^n{m^{\prime }}_{1j}}& {\displaystyle \sum _{j=2}^n{m^{\prime }}_{2j}}& \cdots & {\displaystyle \sum _{j=1}^n{m^{\prime }}_{nj}}\end{array}\right]}^T$

For the consistency analysis based on consistency ratio (C.R. or CR), which is another well-known method of AHP, please see the relevant literature, e.g., [46]. Because this study used expert-choice as a tool throughout the process of AHP (i.e., survey, CWV-determination, and inconsistency check), the discussions are omitted here.

Repeating the above process, a set of CWVs for the relative weights of the constructs w.r.t. the decision goal and the relative weights of criteria w.r.t. a construct is determined for each DM (i.e., the interviewees). All of these results are numerical and can be averaged and combined. For example, the CWVs that were assessed for the constructs w.r.t. the total decision goal could be geometrically meant and then normalized in order to obtain an overall opinion about the relative priorities over these constructs in the entire decision group. This is exactly the logic needed to assess the group opinions in subsequent sections.

Finally, the use of AHP (and the reason we did not replace the third phase of BKDEF with other methods in this study) is because of the method’s long-lasting popularity. It is still the primary method of MADM in recent studies. For this point, the following articles, which involve a variety of applications in the past three years, are cited [47,48,49,50,51,52]. The effectiveness of AHP in solving practical decision problems encouraged the proposal of hybrid models [53,54,55]. Moreover, for the purpose of more complicated projects, many studies combined AHP with other research methods such as fuzzy set logic/intuitionistic fuzzy sets [56,57,58,59]. Thus, AHP is still a suitable approach for the main quantitative knowledge discovery phase of this study because it has been, and still is, a credible method.

3. Methodology: The Refined BKDEF Education Framework

In this section, the process that helps teach scientific management concepts and the related approaches to polling opinions about the budget use decision of MALE UAS R&D are presented. This process is guided by an education framework that was refined based on the original BKDEF. It also organizes four basic phases. This refined framework is outlined in Figure 6.

As can be seen in Figure 6, in comparison with the original BKDEF, the initial three phases of the refined framework (i.e., (1) identification of the decision criteria set, (2) decision criteria hierarchy construction and confirmation, and (3) investigations for polling the opinions of each DM and the assessment of the group opinions) roughly follow the BKDEF, except the subject of study and training is different. However, the last phase of the training, (4) decision analysis, is totally replaced. In the previous BKDEF study, methods from the field of DDDM were used to explore the homogeneity and heterogeneity that exist in the DMs’ group based on the CWVs that were assessed individually for the DMs. However, this was no longer the analytical purpose set for the training course developed for MALE UAV R&D budget allocation. This was driven by the new requirements raised by the DMs of the institution because they were eager to define the difference between the related constructs and the criteria sets for fighting aircraft design and MALE UAV design, as well as the difference in the priority sequences for the design criteria. As such, the teaching materials of this phase were replaced by the methods for making relevant comparisons and professional justifications and discussions.

In the first phase, the relevant articles in the literature, both academic and industrial, were collected and studied. The topics related to UAV, UAS, MALE UAS, and the relevant issues about the MALE UAS R&D were systematically reviewed. From these works, the set of factors to be considered regarding MALE UAS R&D was filtered, sorted, and determined. Later, this set of criteria was confirmed through expert interviews, including UAV/UAS R&D experts who work at institutes and companies focused on UAV/UAS R&D and manufacturing.

In the second phase, the main aim was to establish a decision hierarchy for MALE UAS R&D. Based upon the set of criteria determined and confirmed, we also consulted several MALE UAS R&D experts in an aerospace institute to understand what constructs should be considered. Classification and decision hierarchy establishment was performed. Based on the constructs shared by experts and the properties of the decision criteria, the constructs were organized w.r.t. the total MALE UAS R&D budgeting decision goals, and the criteria were classified into the constructs. Decision hierarchy confirmation was performed, which formed the desired decision hierarchy, and the experts, in turn, confirmed the form. The design of AHP expert questionnaires was conducted. According to the established decision hierarchy, the AHP questionnaires were designed in order to poll the opinions from the DMs. Note that the staff trainees (at the aerospace institute) followed the qualitative Delphi method in order to confirm the constructs for the decision hierarchy and designed the AHP questionnaires that were to be used in the next phase.

In the third phase, the staff was instructed to have face-to-face interviews with the DMs. After each expert interview, the opinions of the DMs about the constructs and the criteria were collected and numerically assessed according to Section 2.4. The staff trainees were also taught to revisit the DM until positive (consistent) outcomes were observed. Several CWVs were obtained from each DM after the pairwise comparison matrices were verified via an inconsistency analysis. Afterwards, by taking these personal opinions from the aerospace R&D institution, knowledge of the aggregated opinions about how to effectively allocate the “large but limited” budget for MALE UAS R&D was elucidated.

In the final phase, according to the intention as defined by the top management of the R&D institution holding the training course, we had a comparison between the decision criteria for MALE UAS R&D and the decision criteria for the next generation fighter R&D [9]. In this phase, the trainees were taught to use several simple narrative statistics methods, such as average, quantiles, and the partial sum of CWVs, in order to exploit the extensive data and to draw some implications. As a result, the differences between these two R&D projects and between the budget allocation strategies for these two projects were analyzed further. Thus, the reasons for the critical design criteria for MALE UAS R&D and air-superior fighter R&D were defined. Moreover, by using the quantile method in statistics, the criteria were partitioned into several levels where each level had a different significance for the design of MALE UAS. This process provided extensive knowledge to be used for future R&D budget allocations in real practice.

By following the refined BKDEF in Figure 6, from the first phase, the 18 relevant criteria for allocating the MALE UAS R&D budget were determined, and their corresponding operational definitions are summarized in Table 6. In the second phase, the four constructs, the decision hierarchy of these constructs, and their criteria were determined and are shown in Figure 5. Section 4 reveals the main results of this study obtained in the third phase. Section 5 discusses the implications of the final decision analysis phase.

4. Results

Following the decision hierarchy confirmed by the Delphi method in Figure 5 and the operational definitions of the design factors reviewed from the literature and expert opinions in Table 6, a set of pair-wise-comparisons-based questionnaires was designed, and the survey was conducted with several experts in the field of UAS R&D in order to understand critical knowledge about the priority of the involved design factors, their relative weights, and whether or not they were justified numerically using AHP. As discussed, this knowledge provides an important guide to the investments in relevant R&D work, i.e., how to properly allocate a “large but limited” budget for MALE UAS design.

The real survey work was completed between 22 July and 25 August in 2018 with many rounds of face-to-face interviews. During the survey, the investigators utilized laptops with Expert-Choice software installed, thus the results (the opinions of the interviewed experts) could be examined right after the set of questionnaires was completed. This was to facilitate the interview process of the survey so that any inconsistency in the results (e.g., a pairwise matrix was C.R.>0.1 and failed to pass the C.R. validation) could be found immediately and required additional round(s) of an interview could begin at once without travelling again.

4.1. The Experts Sample

Since the destination budgeting decision involves many professional domain know-hows of MALE UAS R&D, the DMs invited to participate in the AHP survey were experts who had engaged in relevant aerospace R&D works (in the institution that holds the training course) for many years, and all of them were members or consultants to the committee that was organized for planning the design project. They held power to make decisions for R&D budget allocation. Some of these DMs were specialists focused on the UAV/UAS body, flying control, and avionics system R&D, while some of them were the managers of the R&D project. The rest were managers in charge of other aviation vehicle projects and/or departments. The different stratifications of the 10 experts who participated in the survey are analyzed in Table 7. Eventually, all of them were shown to be effective through the survey process with the required re-interviews given. Therefore, the overall recovery rate of the survey and the rate of effective questionnaires were both 100%.

As can be seen in Table 7, all interviewed experts were male. The eight interviewees older than 50 had been serving for more than 21 years, while two were younger (31–40) with relatively less working experience (i.e., 11–20 years and 4–10 years). Among them, three were managers and four were advisors, while three were R&D team leaders. According to the academic degrees, four of them had Ph.Ds., and the remaining six had master’s degrees.

4.2. Analysis: Flow

The completed questionnaires successfully collected from the survey, i.e., the source data set, were analyzed using AHP. According to Figure 5, for each interviewed expert, there were five questionnaires collected. One was used for valuing the pairwise comparison matrix for the constructs w.r.t. the total design decision (goal) (i.e., the “MUREM”, which connotes ‘MALE UAS R&D Evaluation Model’ as shown), while the other four valued the matrices for the factors w.r.t. each of the constructs. Based on these valued pair-wise comparison matrices, for each expert, a CWV that carried the relative weights for the four main constructs was assessed, while four CWVs containing the relative weights for the factors under each construct were also assessed. Thus, the priority over the constructs and the priority over the factors under every construct could be obtained. However, the process as stated above only applied to this data set and revealed the opinion of an expert. Based on the concept of group decision-making, the aggregated opinion of this decision group could be obtained by geometrically meaning the CWVs of all experts w.r.t. the same thing (MUREM, C1, C2, C3, C4). Doing so yielded another five CWVs, i.e., the CWV over the four constructs for the design of MALE UAS, the CWV over the six equipment and performance factors, the CWV over the four technological capability factors, the CWV over the five logistics support factors, and the CWV over the three system growth factors. These CWVs justified the group opinions rather than the individual opinions about the design and the budget planning decision. For the mathematical details of the above process, we cite the relevant given formulations included in the BKDEF [9].

The next section focuses on the comparisons made between the group opinions when designing a MALE UAV and those when designing a next generation fighter. The following subsection presents the results obtained from the aggregated basis and omits the individual opinions that were not significant for the decision analysis phase of this study.

4.3. Analysis: Results

The relative weights assessed on the aggregated basis for the four main constructs for MALE UAS design are detailed and compared in Figure 7. For an example of how the individual CWVs for the constructs were combined over the DMs in order to review the group opinion, please see Table A1 in Appendix A in detail.

The preferred priority for the four main constructs was clear from the above figure (i.e., C2 ≻ C3 ≻ C4 ≻ C1). The weights of the first two constructs, technological capability (C2) and logistics support (C3), were 32.9% and 31%, respectively. Their importance reached almost 2/3 of the total importance. In contrast, the importance of the other two constructs, system growth (C4) and equipment performance (C1), was just over 1/3. It is interesting to note that their individual weights were almost on par with a gap of only 0.7% (i.e., 18.4% versus 17.7%).

By interpreting these results, it could be understood that C2 and C3 were both important for MALE UAS design. Addressing the superior technological capability and an overall good logistics support during the design may solidify the development process.

For the C1 construct, which was relatively less important, there were six relevant factors. After the CWV assessment, the relative weights are shown in Figure 8.

As understood from Figure 8, the preferential relationship for the factors was: F3 ≻ F4 ≻ F5 ≻ F2 ≻ F6 ≻ F1. Payload performance (F3) was the most important factor, and that importance was over 1/4. The second factors were in the group that included a redundant flight control system (F4) and avionics system (F5), both of which had weight around 1/5. The factor that was ranked fourth, endurance (F2), still held an importance of 1/7 for equipment performance (C1). The weights for the last two factors, external payload (F6) and service ceiling (F1), were relatively small (i.e., around 1/9 and 1/16, respectively).

For C2, which was the most important construct, there were four factors. After the CWV assessment, the relative weights are shown in Figure 9.

As can be seen in Figure 9, the priority over these factors was: F7 ≻ F8 ≻ F9 ≻ F10. System-wide integration (F7) was not only the most important factor but also the factor that dominated the technological capability (C2) construct (the importance for which was 4/9). The design and manufacturing for the key components (F8) was the following factor with half the importance of 2/9. The third factor was information integration (F9), whose importance was 2/11. The factor ranked last, information transmission (F10), still held an importance of 15.5% for C2. Given the dominance of F7 and the heaviest weight of C2, F7 was perhaps the most important factor of all 16 factors. This is discussed later.

C3, which was also one of the most important constructs (see Figure 7), had five factors. The relative weights are shown in Figure 10.

As can be understood from Figure 10, the priority sequence was: F11 ≻ F12 ≻ F13 ≻ F15 ≻ F14. The reliability of a redundant flight control system (F11) and that of avionic systems (F12), whose importance were both greater than 1/4, were the most important factors for overall logistics support (C3). The acquisition of the key components (F13) was middle in both rank and importance, which was 1/5. The final two factors (F15 and F14, logistic support and DMSMS management) were minor, although their weights were about 15%. As observed, w.r.t. this C3 construct, the heaviest weight assessed for F11 (26.2%) although almost double, did not double the least important factor, F14 (13.9%). However, w.r.t. to the former two constructs, C1 and C2, the gaps between the largest factor weight and the smallest factor weight were salient (26.6% versus 6.8% for C1, which was almost quadruple; 44.3% versus 15.5% for C2, which was almost triple).

C4, a minor construct like C1, had three factors. The relative weights are evaluated in Figure 11.

Figure 11 reveals a priority sequence of the three factors w.r.t. system growth, which was: F16 ≻ F18 ≻ F17. Compared to design continuity (F17), the architecture of the UAS and the expandability of its components (F16) and the expected future system performance growth (F18) were both important relative to F17. For system growth (C4), the importance of F16 was 2/5, while that of F18 was more than 1/3. Together, these two constituted a 3/4 importance. However, as observed for C3, the dominance relationship was not salient in this group of factors. The contradistinction between the most important F16 and the least important F17 was 0.399/0.247, which was even more than that between F11 and F14 (26.2%/13.9%).

4.4. Analysis: An Overall Review

Multiplying the factor weights w.r.t. a construct with the weight of the construct generated the absolute weights for the factors w.r.t. the total design budgeting decision goal (i.e., MUREM). These enabled the ranking process of the involved factors, which is shown in Table 8. For these calculations to obtain the absolute weights aggregated based on the group opinion, the opinion-combining process was used and is detailed in Table A2 in Appendix B.

According to the results in Table 8, overall, system-wide integration (C2-F07) was the most important factor, service ceiling (C1-F01) was the least important one, and (C2-F07)’s importance was more than seven times that of (C1-FO1). Thus, the importance levels of all factors were quite different and spanned a large interval. This implied that the investigation was able to discriminate important factors for MALE UAS design.

The other observation was that these factors could be clustered by reference to their absolute weights. At first, there was a salient gap between the weights assessed for the fourth factor (C3-F13, 0.076) and the fifth factor (C3-F15, 0.062). That is, the top four factors formed the “factor class (I)” that must be considered for MALE UAS design and the budget allocation process. It included C2-F07: System-wide integration, C3-F11: Reliability of redundant flight control system, C3-F12: Reliability of avionic system, and C3-F13: Key component acquisition. It was not surprising that “system-wide integration” (F7) was ranked as the most important factor. As discussed previously, C2 (technological capability) was the most important construct, and F7 dominated other factors of this construct with an importance of 44.3%. However, the other three factors in factor class (I) were factors w.r.t. C3, which was the overall logistics support construct.

Next, according to the same logic, another factor class (II) could be identified, and it included six factors, i.e., C3-F15: Logistic support, C4-F16: UAS system architecture and component expandability, C1-F03: Payload performance, C3-F14: DMSMS management, C2-F08: Key component design and manufacture, and C4-F18: System performance growth. If there was a limited budget and only 10 factors could be addressed during the design process while tasks pertaining to the rest of the factors could be outsourced, the set of the class (I) and class (II) factors (whose absolute weights remained greater than 5% among the total 16 factors), the priority over these factors, and their assessed weights would unquestionably be important guides for the resource input of MALE UAS design. Due to space limitation and their minor importance, the discussions about the eight remaining factors, including the six class (III) factors and the two class (IV) factors, are omitted in this paper.

Another observation could be made based on the above data. In factor class (I), only one factor w.r.t. C2 was included, but there were three factors w.r.t. C3. On one hand, this was due to the surge of the system-wide integration factor, while the other factors in C2 were far from being considered for class (I). On the other hand, this was due to the fact that C3 was also an important construct whose weight (0.310) was not far from C2 (0.329), but the relevant factors (from F11 to F15) were rather evenly weighted, thus the top three factors (F11, F12, and F13) w.r.t. C2 were in factor class (I). When the top 10 factors [i.e., factor classes (I) and (II)] were considered further, there was only one factor w.r.t. C1 (i.e., C1-F03: Payload performance), two factors w.r.t. C2, five factors w.r.t. C3, and also only one factor w.r.t. C4 (i.e., C4-F18: System performance growth). This was important because in 4.3, we observed that the preference relation among the constructs was: C2 ≻ C3 ≻ C4 ≻ C1, which was in fact: (C2~C3)≻(C4~C1). The fact that more factors from C2 and C3 (i.e., 7 = 5 + 2) were filtered in factor classes (I) and (II) while fewer factors (i.e., 2 = 1 + 1) were filtered from C1 and C4 was reflective of the preferential relationship revealed for the constructs. However, as observed in this set of factors, only two factors w.r.t. C2 among four (i.e., F7 and F8) were included, while all five factors w.r.t. C3 were included for similar reasons as the previous discussions, when only class (I) was in the focus.

Finally, by classifying the factors, the weights for the factors in the same class could be aggregated in order to understand the total impact of each class. For factor class (I), the weight aggregated over the four factors included was 0.38. For factor class (II), the aggregated weight over the six factors was 0.34. For factor class (III), it was 0.24, while it was merely 0.04 for class (IV). This means that for the DM, when allocating the budget for designing the MALE UAS, he/she should consider classes (I) and (II) because the factors in these two classes dominated the total importance at 72% for the design. If a larger budget can be allocated, additional factors in class (III) can be considered.

5. Discussion

In addition to the relevant discussions/implications explained in Section 3, further insights can be explored by comparing the results with the previous work that proposed the BKDEF and with the relevant results of the budget allocation issue of the next generation fighting aircraft design. This is exactly the knowledge obtained from the fourth phase of the “second application” of BKDEF.

The design of the next generation fighter has a unique focus, as per the study [9]. The main post-training result, which includes the absolute weights assessed for the factors as required by the DM and the overall rank, is presented in Figure 12.

As shown, the design of the next generation fighter involved 12 design factors. If all of these factors were equally weighted, then every factor would receive a weight of 0.083. However, in the above figure, the first half of these factors already contributed a total importance of 72.4%. In contrast, in Table 8, if the same rule was applied (aggregating the first nine factors among 18), the total importance of them would only be 66.6%. In other words, the importance of the “first half” far exceeded the average by 22.4% when allocating the budget for designing the fighter, but this was only 16.6% when allocating for designing the MALE UAS. How does this happen if the two studied subjects are also “aircrafts”?

Before further implications can be drawn, data from Table 8, the absolute weights, and the priority for the factors of the MALE UAS design problem are collectively depicted in Figure 13 for a visual comparison.

As is generally understood, the purpose of most fighters is a matter of “life and death”. In the age of machine-gun-based air combatting (i.e., dog fights), the maneuverability and control of a fighter, fast speed, and a high operational altitude were all critical. The advent of air missiles changed the method of air fighting. The ability to perform beyond-visual-range attacks became the first factor, and the functionality of the radar system was another key factor. This also drove the debut of hidden fighters. Later, with increased precision in cruise missiles, the war parties tended to target a military airport directly. Therefore, the ability to perform vertical and/or short landing maneuvers and/or take-offs were addressed for fighting aircraft design. These innovations evolved naturally with the historical technological competitions for weapons, implying that there have been “paradigm shifts” in the fighting aircraft design subject to different historical backgrounds, which is still true today.

However, there is one thing that should remain unchanged over time—in any era, the more powerful the designed technologies become, the higher the chance will be for a fighter to defeat their enemy. However, for the different eras that DMs are facing, there are different “powerful technologies” that must be addressed in the design. Therefore, in order to allocate a budget for next generation fighter design, these “powerful technologies” must be specified (although they were specified indirectly through the “mind mining” process with the DMs using AHP). This was the reason for the great imbalance in the factor weights observed in Figure 12, within which the earlier but mature and fundamental technologies were ranked in the “bottom half”.

In contrast, in this study, the subject was totally different. This study was about the design for MALE UAS despite the fact that a similar analysis was performed according to the BKDEF, the subject of which was also the air vehicle.

Because the main purpose of MALE UAS is not battling but investigation and reconnaissance, the functions that should be considered for its design are more like those for a cargo aircraft or a civil airplane, which address the stability or steadiness during the task. Thus, when the total number of 18 factors are considered comprehensively, there are nine factors (half of the factors) whose assessed weights are between the [0.044, 0.066] interval, which is the ±20% range around the mean. This is reflected in Figure 13, where the factor weights are more balanced, even if there are more factors to be considered in the decision context.

To briefly summarize, the results were drastically different because the studied subjects, i.e., the design budgeting for MALE UAS and the design budgeting for next generation fighters, are drastically different. Their purposes are fundamentally quite different. This was also true when both factor sets (i.e., the set of design criteria) were examined, where it was found that they did not intersect with each other. In other words, there was no commensurable factor present in both sets. However, due to the large difference between a MALE UAS and an air-superior fighter, as discussed above, this was likely normal.

Finally, some issues that were analyzed during the comparison were presented. These served as supplementary materials for aerospace. In earlier years, the reconnaissance aircrafts were re-equipped based on civil airplanes. Due to their low speed, many of them were shot down during their missions. Thus, fighters were later used for reconnaissance. Since the cold war, the design of the super-high-altitude reconnaissance aircrafts emphasized their ability in stealth against radar. For example, a Thunderbird has never been shot down during a mission. This has become the emphasis when designing the fifth generation fighters, which is also reflected in Figure 12.

With exception to the fighters, there are many military aircrafts that serve battling purposes, e.g., ground attackers, bombers, reconnaissance crafts, anti-submarine aircrafts, early warning aircrafts, aerial refueling crafts, and attack helicopters. If the studied subject were to be further focused on these aircrafts before another R&D project was launched (thus another strategy for budget allocation would be made), the BKDEF could be applied once again. However, unlike this study, wherein there was no overlap between the factors for MALE UAS design and those for air-superior fighter design, some intersections between the design factors (criteria sets) would be expected.

Furthermore, after World War II, the cost for producing aircrafts increased tremendously. Thus, maintaining several production lines for many single-purposed aircrafts is burdensome. Therefore, designing multi-purpose aircrafts has become a technological trend. This is also true for the military aircrafts, e.g., the F-16 Falcon and the F-18 Hornet fighters may also perform ground attacks and bombing missions. Therefore, in accordance with this trend, the future design of military air vehicles should increase the number of design factors, and an overlap between the factor sets should be expected.

6. Conclusions

The interesting implications discussed in the former section are mainly for comparing the post-training results between two studies, which include supplemental knowledge from the aerospace domain. The style of such a comparison work also enriches the BKDEF educational framework because it provides an alternative way to perform the decision analysis for the final phase of BKDEF instead of making a purely quantitative analysis using the methods in DDDM. In the previous work [9], the main aim of decision analysis was to explore the homogeneity and heterogeneity in the polled opinions that might exist in the decision group. However, in this work, the main purpose of this phase was to conduct a comparative study in order to discover and compare the large difference between the criteria set used for the design (and budgeting) of MALE UAS and that needed for designing an air-superior fighter. In other words, this study proposed another novel method of knowledge discovery for the decision analysis phase of BKDEF. However, apart from this, other possible points may also highlight the possible contributions of this study.

This paper revealed the “second application” of BKDEF, i.e., the BKDEF was applied for the second time in the training course for an in-service for budgeting staff. As discussed, although the application process was not specifically addressed in this paper, it underlined the results obtained after each phase of the process. As these positive results were obtained from the post-training exercises, the effectiveness of the BKDEF was confirmed. This is important for verifying BKDEF because, to the authors’ knowledge, BKDEF, which focuses on the budget allocation for single high-cost product design, had never been applied after it was proposed (with its “first application”) until the application in this study. In other words, this study not only provided another empirical application of BKDEF but also verified this novel educational framework. This is extremely meaningful because it strengthens the practical applicability of a framework that had only one initial application to date.

The main results of the study, including the set of design criteria for MALE UAS design, the priority of these criteria, and their relative importance, may indicate a complete set of knowledge for decision-making. The set of criteria (Table 6) is sound because each included a criterion (factor) identified in terms of a rigorous literature study. The relative importance of a criterion w.r.t. the upper construct was assessed numerically in the CWV, which was aggregated over the entire decision group (Figure 8, Figure 9, Figure 10 and Figure 11). In addition, the relative importance of the construct was also assessed numerically w.r.t. the total design decision for MALE UAS in another CWV (Figure 7). These not only allowed the assessment of the absolute weight of each design factor (Table 8) but also enabled prioritization over all the considered design factors (Figure 13). Since the above process was performed with a group-opinion basis (10 experts), the information derived from these empirical assessments provides practical decision knowledge for making a suitable budget allocation plan for the MALE UAS design project.

Although one possible direction of research includes further applications of BKDEF and extended comparative studies for other military aircrafts, this was omitted here because it was discussed in Section 5. Since the effectiveness of BKDEF was verified twice, it or a similar research method (like the one slightly modified from BKDEF by this study) would be fine to use in other topics, even if only a special audience might be interested in the obtained results.

Author Contributions

Conceptualization, L.-P.C. and J.-H.H.; Methodology, C.-H.F. and Z.-Y.Z.; Software, Z.-Y.Z.; Validation, L.-P.C.; Investigation, L.-P.C. and C.-H.F.; Resources, L.-P.C., C.-H.F. and Z.-Y.Z.; Writing—original draft preparation, C.-H.F. and J.-P.C.; Writing—review and editing, Z.-Y.Z.

Funding

This research was funded by Ministry of Science and Technology (Taiwan), 107-2410-H-992-046 & 106-2410-H-038-001 and the APC was funded by 107-2410-H-992-046.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table A1

Aggregating the criteria weight vectors (CWV) for the constructs w.r.t. MUREM (the total decision goal of MALE UAS R&D Evaluation Model): the combining details.

	DM1	DM2	DM3	DM4	DM5	DM6	DM7	DM8	DM9	DM10	Combined
C1	0.495	0.093	0.129	0.073	0.075	0.058	0.118	0.092	0.193	0.636	0.177
C2	0.117	0.132	0.610	0.423	0.514	0.268	0.555	0.118	0.143	0.233	0.329
C3	0.332	0.217	0.061	0.352	0.185	0.598	0.287	0.531	0.452	0.049	0.310
C4	0.067	0.558	0.201	0.153	0.226	0.075	0.040	0.259	0.212	0.082	0.184

Appendix B

Table A2

Aggregating the opinions for the absolute weights of all factors: the combining details.

	DM1	DM2	DM3	DM4	DM5	DM6	DM7	DM8	DM9	DM10	Combined
F1	0.029	0.008	0.003	0.001	0.002	0.005	0.006	0.007	0.046	0.023	0.068
F2	0.066	0.006	0.005	0.004	0.023	0.004	0.007	0.015	0.080	0.279	0.147
F3	0.128	0.008	0.058	0.023	0.024	0.005	0.033	0.029	0.021	0.160	0.266
F4	0.170	0.036	0.010	0.016	0.010	0.025	0.032	0.018	0.009	0.031	0.212
F5	0.082	0.028	0.018	0.012	0.013	0.017	0.032	0.018	0.006	0.057	0.198
F6	0.021	0.007	0.034	0.017	0.003	0.002	0.008	0.006	0.030	0.086	0.109
F7	0.039	0.033	0.328	0.191	0.249	0.171	0.130	0.034	0.048	0.133	0.443
F8	0.028	0.013	0.030	0.110	0.170	0.060	0.252	0.046	0.010	0.062	0.220
F9	0.028	0.040	0.089	0.050	0.073	0.027	0.086	0.026	0.043	0.014	0.182
F10	0.022	0.047	0.163	0.071	0.022	0.010	0.086	0.011	0.043	0.025	0.155
F11	0.183	0.019	0.004	0.061	0.119	0.079	0.055	0.061	0.113	0.024	0.262
F12	0.057	0.023	0.033	0.037	0.035	0.039	0.117	0.190	0.073	0.010	0.244
F13	0.042	0.023	0.015	0.040	0.014	0.308	0.057	0.099	0.045	0.006	0.196
F14	0.023	0.076	0.002	0.083	0.011	0.022	0.037	0.061	0.159	0.006	0.139
F15	0.017	0.076	0.007	0.131	0.006	0.151	0.021	0.121	0.061	0.003	0.159
F16	0.022	0.186	0.052	0.083	0.035	0.055	0.008	0.052	0.096	0.052	0.399
F17	0.022	0.186	0.128	0.025	0.056	0.015	0.008	0.052	0.015	0.009	0.247
F18	0.022	0.186	0.021	0.045	0.134	0.005	0.024	0.155	0.101	0.021	0.354

Figures and Tables

Figure 1. Quantities of UASs versus manned aircrafts in the U.S. military in 2010.

Figure 2. Countries acquiring the armed UASs.

Figure 3. Countries supplying the armed UASs.

Figure 4. The world military UAS budget forecast (FY: fiscal year).

Figure 5. The decision hierarchy for medium-altitude long-endurance unmanned aerial system (MALE UAS) design.

Figure 6. The refined (budgeting) knowledge discovery educational framework (BKDEF) framework.

Figure 7. The relative importance and priority of the four main constructs.

Figure 8. The relative importance and priority of the factors with respect to (w.r.t.) equipment performance (C1).

Figure 9. The relative importance and priority of the factors w.r.t. technological capability (C2).

Figure 10. The relative importance and priority of the factors w.r.t. overall logistics support (C3).

Figure 11. The relative importance and priority of the factors w.r.t. system growth (C4).

Figure 12. The factors for next generation fighter design: an overview.

Figure 13. The factors for MALE UAS design: an overview.

Table 1

The uses of unmanned aerial system (UAS) technology in the different civilian and military domains.

For Civilian Use
Aerial photography	Film, video, and stills
Agriculture	Crop monitoring and spraying, herd monitoring, and cattle driving
Coastguard	Search and rescue, coastline and sea-lane monitoring
Conservation	Pollution and land monitoring
Customs and excise	Surveillance for illegal imports
Electricity companies	Powerline inspection
Fire services and forestry	Fire detection and incident control
Fisheries	Fisheries protection
Gas/oil supply companies	Land survey and pipeline security
Information services	News information and pictures, featured pictures, e.g., wildlife
Lifeboat institutions	Incident investigation, guidance, and control
Local authorities	Survey and disaster control
Meteorological services	Sampling and analysis of atmosphere for forecasting
Traffic agencies	Monitoring and control of road traffic
Oil companies	Pipeline security
Ordnance survey	Aerial photography for mapping
Police authorities	Search for missing persons, security, and incident surveillance
Rivers authorities	Water course and level monitoring, flood and pollution control
Survey organizations	Geographical, geological, and archaeological survey
Water boards	Reservoir and pipeline monitoring
For Military Use
Navy	Shadowing enemy fleets, decoying missiles via the emission of artificial signatures, electronic intelligence, relaying radio signals, protection of ports from offshore attack, placement and monitoring of sonar buoys, and possibly other forms of anti-submarine warfare
Army	Reconnaissance, surveillance of enemy activity, monitoring of nuclear, biological or chemical (NBC) contamination, electronic intelligence, target designation and monitoring, location and destruction of land mines
Air Force	Long-range high-altitude surveillance, radar system jamming and destruction, electronic intelligence, airfield base security, airfield damage assessment, elimination of unexploded bombs

Table 2

U.S. Department of Defense (DoD) classification for UAVs.

Weight (kg)	Normal Operating Altitude (ft.)	Mission Radius (km)	Endurance (hrs)	Representative UAV
<2	<400	5	<1	Black Widow, Raven
2~25	<3000	25	2–8	Aerosonde, Scan Eagle, Puma
25~150	<5000	50	4–12	Manta B
150~600	<10,000	200–500	8–14	SIERRA, Viking 400, Tiger Shark
>600	<18,000	1000	>20	Ikhana (Predator B)
>600	>18,000	5000	>24	Global Hawk

Table 3

NASA’s UAS classification matrix.

Category	I	II	III
Weight	≤55 lb (25 KG)	55–330 lb (25–150 KG)	>330 lb (150 KG)
Airspeed (kt)	≤70	≤200	>200
Type	Model or sUAS	sUAS	UAS

Table 4

Categorizing the UASs based on the “unmanned aerial vehicles (UAV) type”.

Category	Weight	Normal Operating Altitude	Radius of Mission	Endurance	Normal Employment	Typical Uses (*)
Mirco	<2 KG	Up to 200 ft	5KM	Few hours	Tactical platoon (single operator)	R, I, S
Mini	2~20 KG	Up to 3000 ft	25KM	Up to 2 days	Tactical subunit (manual launch)	S, DG
Small	20~150 KG	Up to 5000 ft	50KM	Up to 2 days	Tactical unit (employs launch system)	S, DG
Tactical	150~600 KG	Up to 10,000 ft	200KM	Up to 2 days	Tactical formation	S, DG
MALE	>600 KG	Up to 45,000 ft	Unlimited	Days/weeks	Operational/Theater	S, CT
HALE	>600KG	Up to 65000ft	Unlimited	Days/weeks	Strategy/National	S, DG, SR
Strike/Combat	>600KG	Up to 65000ft	Unlimited	Days/weeks	Strategy/National	S, DG, SR

* Table legends: R: reconnaissance, I: inspection, S: surveillance, DG: data gathering, CT: cargo transportation, SR: signal relay.

Table 5

Categorizing the UASs based on text-based feature descriptions.

Category	Feature Description
NAV	Nano Air Vehicles: It is recommended that these be used for radar obfuscation, or, if the camera, propulsion, and control subsystem can be made small enough, for ultra short range monitoring.
MAV	Micro UAVs: The MAV was originally defined as a drone with a wingspan of no more than 150 mm. A MAV is mainly used for operations in urban environments, especially in buildings. It is necessary to fly slowly, preferably to stop and sit on the wall or on a pillar. MAVs are usually expected to be manually launched, thus the winged versions have very low wing loads, which make them very susceptible to atmospheric turbulence. This type of problem can exist with all types of MAVs.
MUAV	Mini UAVs: Refers to UAVs below a certain mass (not yet defined), possibly less than 20 kg but not as small as a MAV, capable of a manual launch and operating up to approximately 30 km. They are used by mobile battle groups and are also used for various civilian purposes.
Close-range UAV	Close-range UAVs are used by mobile forces and in other military/naive operations and for a variety of civilian purposes. They typically operate up to a range of approximately 100 km and are capable of performing a variety of tasks such as reconnaissance, target designation, NBC (nuclear, biological and chemical) monitoring, airport security, ship-to-shore surveillance, power line inspection, crop spraying, traffic monitoring, etc.
TUAV	Medium-range/Tactical UAV: Its operating range is between 100 and 300 km. Compared to HALE and MALE, these UAVs sizes are smaller and their control system is simpler, mostly operated by the Army and Navy.
MALE UAV	Medium-altitude long-endurance UAVs: Their flight altitude is between 5000–15000 m and the endurance is 24 h. Their functions are similar to the HALE system, but they usually operate in a shorter range but still exceed 500 km. They require being operated at a fixed base.
HALE UAV	High-altitude long-endurance UAVs: The flight altitude is over 15,000 m and their endurance is more than 24 h. They conduct extreme remote (cross-global) reconnaissance and surveillance, and arming a HALE is a future trend. They are usually operated by the Air Force from a fixed base.

Table 6

Operational definitions of the decision factors for MALE UAS design budgeting.

Construct	Evaluation Factors	Operational Definition
(C1)Equipment & Performance	(F1)Service ceiling	Service ceiling is the maximum usable altitude of an aircraft when it climbs with its engine output balancing with gravity force and ultimately cannot reach any higher altitude.
	(F2)Endurance	Endurance is the maximum length of time from the moment the aircraft first taxies out with full tanks to the end of the flight.
	(F3)Payload performance	Aircraft can internally or externally carry a variety of instruments to meet mission requirements. The payload performance refers to the performance of these instruments, for example, the maximum precision of optical distance sensors and the ultimate sensitivity of the electronic reconnaissance system.
	(F4)Redundant flight control system	Redundant flight control system represents the conjugation of multiple control systems. The conjunctions aim to rule out the possibility of operation failure caused by a single control system.
	(F5)Avionics system	Avionics system generally refers to a combination of multiple advanced technologies including management and illustration of communication and navigation. Terminologically, avionics is a compound word formed from aviation and electronics.
	(F6)External payload	External payload indicates the unmanned aircraft’s expanded capability, which covers the number of stations and their loading capacity designed under both flaps and the longitudinal axis.
(C2)Technological Capability	(F7) System-wide integration	System-wide integration refers to the consolidation of the interface between various systems, such as flight control system, avionics, engine controls, and flap operation systems, to coordinate of all the signals and ultimately guarantee the aircraft’s overall function and mission performance.
	(F8)Key component design and manufacturing	Key component design and manufacture refer to the process of design and production of a certain component that not only has the characteristics of high value and low substitutability but also attributes of great influence among all systems.
	(F9)Information integration	Information integration is a simultaneous incorporation of data collection and analysis. Specifically, information is gathered from various detection systems on UAV, transmitted to the ground, and put through the human-machine interface or the big-data analysis to reach a further result. http://www.lokisys.com/2015/01/integration-vs-interface/
	(F10)Information transmission	Information transmission is a sequence of approach to transfer the detected information from the aircraft to the ground then transmit across the interface of segments through routing and digital convergence procedures.
(C3)Overall Logistics Support	(F11)Reliability of redundant flight control system	The redundant flight control system conjunctions aim to rule out the possibility of operation failure caused by a single control system and ultimately promote the reliability of the whole control system.
	(F12)Reliability of avionic systems	Avionics is a specialized system that combines management and illustration of multiple advanced technologies, including communication and navigation. It also embodies the junction of aviation and electronics. In support missions, the reliability refers to the overall performance and accuracy to hit the target.
	(F13)Key component acquisition	Key component acquisition is obtaining the process of a certain component that not only has the characteristics of high value and low substitutability but also greatly contributes to systems performance.
	(F14)DMSMS management	DMSMS management should prevent key components at the service of certain major systems from being out of stock for any reason.
	(F15)Logistic support	Logistic support is an integrated process to manage all kinds of resources, such as human resources, materiel resources, and financial resources, to strategically optimize inventory management and acquisition.
(C4)System Growth	(F16)System architecture and component expandability	The capability is to develop novel functions or to boost module efficiency of UAS by modifying its system architecture and component expandability.
	(F17)Design continuity	Design continuity generally refers to the component compatibility of the succeeding system with prior development. The continuity could reduce the unfamiliarity in operation processes.
	(F18)System performance growth	System performance growth refers to the adaptation capability contributed by rectifying or refining the current system to enhance its performance.

* DMSMS: Diminishing of manufacturing sources and material shortages.

Table 7

The different stratifications of the interviewed experts.

Stratification	Attribute	#Experts	Percentage
Gender	Male	10	100%
	Female	0	0%
Degree	Ph.D.	4	40%
	Master	6	60%
Occupancy	Manager	3	30%
	Advisor	4	40%
	R&D Leader	3	30%
In-service	4–10 years	1	10%
	11–20 years	1	10%
	>21 years	8	80%

Table 8

The absolute weights of the factors and overall rank.

Construct–Factor	Absolute Weight	Rank	Factor Class	Consistency
C2-F07: System-wide integration	0.108	1	(I)	Inconsistency= 0.01 with 0 missing judgments.
C3-F11: Reliability of redundant flight control system	0.101	2	(I)
C3-F12: Reliability of avionic system	0.094	3	(I)
C3-F13: Key component acquisition	0.076	4	(I)
C3-F15: Logistic support	0.062	5	(II)
C4-F16: UAS system architecture and component expandability	0.060	6	(II)
C1-F03: Payload performance	0.058	7	(II)
C3-F14: DMSMS management	0.054	8	(II)
C2-F08: Key component design and manufacture	0.053	9	(II)
C4-F18: System performance growth	0.053	10	(II)
C1-F04: Redundant flight control system	0.046	11	(III)
C2-F09: Information integration	0.044	12	(III)
C1-F05: Avionic system	0.043	13	(III)
C2-F10: Information transmission	0.038	14	(III)
C4-F17: Design continuity	0.037	15	(III)
C1-F02: Endurance	0.032	16	(III)
C2-F06: External payload	0.024	17	(IV)
C1-F01: Service ceiling	0.015	18	(IV)