A Hybrid Model for Evaluating the Outcomes of Student Pilots within the Didactic System of Aviation Education

Abstract

The agenda of the international community is focused on the global problem of aviation in the lack of pilots while maintaining the quality of their professional education. The trend is to explore the potential of students and help increase resistance to obstacles on the way to results, but especially on the way to the creation of the required competencies, knowledge, and skills. The main goal of this paper is to develop a hybrid expert model evaluating the results and risks of student (pilot) outcomes to improve the quality of individual results in the didactic system of aviation education (study results from theoretical subjects, evaluation from training on flight simulators, and from practical flight training) within one information platform. The model is based on the modern theory of intellectual knowledge analysis, fuzzy set theory, and a systems approach. Information from the repository of 696 individual results of undergraduate students (pilots) from the total number of 2682 undergraduate students of all specializations in aviation education in the period 2005–2022 was used in the creation of the model. The results were tested on the examples of five undergraduate students (pilots) and demonstrated the applicability of the expert methodology for evaluating the quality and risks of individual results in aviation education for individual study counseling within one information platform.

1. Introduction

Human resource management, education, and consulting are seen as important activities in favor of achieving business plans in the commercial sphere, or in the case of a university, it is the quality of individual study results in which we identify various problems of application practice. In connection with this, the current issue is the quality of individual student’s results during higher education, which are influenced by internal and external factors in the educational process, in the context of the 4th goal of the Quality of Education within the 17 goals of sustainable development defined by the United Nations [1]. The specific target group needed for the aviation sector is a group of students (pilots) in training.

After the historic losses incurred by the airline industry because of the pandemic and the related travel restrictions, the International Air Transport Association (IATA) expects the industry to return to profitability in 2023 [2]. For example, travel outside Europe (74% in 2019) is still weaker than intra-European travel (85% in 2019) [3]. For air transport, the IATA expects that the labor and skill shortage constraints observed in 2022 will steadily dissipate over time. There is the possibility that such shortages may be more enduring if hiring does not keep up with the needs dictated by the pace of the industry’s demand for recovery [4]. Much attention is therefore devoted to aviation training and capacity building. For example, the International Civil Aviation Organization Council presented at the 41st ICAO General Assembly a status report on the implementation of the ICAO Civil Aviation Training activities and capacity-building strategies in aviation to support member states [5]. The agenda of the international community is focused on the global problem of aviation. Global forecasts predict a significant labor shortage in the aviation industry in the coming years. Multiple outlook reports indicate that, over the next 20 years, there is a need for over 500,000 new pilots, 622,000 maintenance technicians, and 858,000 cabin crew to meet growing worldwide demand [6]. The ICAO founded the Next Generation of Aviation Professionals (NGAP) initiative with the aim of attracting and educating qualified junior experts in the field of aviation, for example, also within the framework of the ALICANTO program—The International Association of Aviation and Aerospace Education [7]. Members of the ALICANTO association or the Partnership of a European Group of Aeronautics and Space Universities (PEGASUS) are looking for solutions and partial contributions to improve the quality of the aviation personnel education system [8]. The presented paper has the stated ambition.

The research topic is related to the individual results of students (pilots) and their integrated, electronic expert evaluation within the didactic system of aviation education.

The research problem is an information model using the fuzzy logic of students’ (pilots) performance evaluation based on data from the databases of the didactic system of aviation education (study results from theoretical subjects, evaluation from training on flight simulators, and from practical flight training), for the integrated assessment of performance and risks of education results, due to timely response, goal-oriented assistance and an increase in the quality of individual study counseling for the students (pilots) within the framework of aviation education. The integrated expert assessment of the students’ (pilots) performance is carried out by a team of actors (group of experts) described in Section 2.1. Formal Formulation of the Problem. On this basis, the paper proposes Hypotheses 1 and 2.

Hypothesis 1 (H1):

For the continuous and comprehensive final professional assessment of student performance in aviation education, it is important to have integrated data on the results of education, rather than working only with separate data repositories that are not digitally compatible.

Hypothesis 2 (H2):

If the criteria for assessing the riskiness of aviation education results are applied within one information platform, we can provide a warning for reaction, help, and individual advice to the student pilot.

The main goal of this paper is to develop a hybrid expert model evaluating the results and risks of student (pilot) outcomes to improve the quality of individual results in the didactic system of aviation education (study results from theoretical subjects, evaluation from training on flight simulators, and from practical flight training) within one information platform.

The model can determine the assessment of the quality of the student’s individual results in aviation education, which considers the levels of theoretical training, the formation of the student’s competence, the risks of study results, flight simulator training, and practical flight training, as well as the observed behavioral competence during the student’s practical flight training.

Since the model uses data on students’ results on the one hand and combines the experience and knowledge of flight training instructors (experts and teachers) on the other hand, the model is called a hybrid model. In addition to final grades, the complex model also provides intermediate results, so it can be used during the entire period of the training of students, and if the student does not have an acceptable level of mastery of knowledge and the acquisition of relevant competencies, it recommends mandatory remedial training. As a result, the level of the overall assessment of the quality of individual results of a student is obtained, which is a comprehensive indicator and increases the degree of validity of making further management decisions regarding the possibility of improving the quality of the pilot’s education and reducing the risks to education results.

The use of the key results is limited to universities cooperating with organizations approved for flight training (ATO) in the framework of training future pilots or aviation schools that are interested in a unified information platform for evaluating results, early identification of risks, and providing individual advice to the student to ensure the highest quality in aviation education. Each university has an academic information system for registering the results of completed courses and earned credits. Not every university has an academic information system that registers the results of flight training, if it is provided by an external organization, for the evaluation of students’ knowledge, skills, and competencies, including the identification of risks in the education of future pilots for a quick response and individual counseling to the student.

Standard methods of evaluating student results are generally focused on the evaluation of subjects, expressed numerically based on verification, in the evaluation “from A to FX” (from 100% to 50%) and in the registration of credits for completed subjects. The student’s results are registered in the academic information system. These information systems act based on ex-post evaluations, they are not proactive to help the student analyze the results and find solutions, they only indicate the numerical and credit output of their education. They do not inform the student whether it is “a lot, a little, undesirable...” for their future competence to perform the profession of a pilot. The presented proposal enables an immediate reaction and individual study counseling in the event of undesirable results (undesirable expert evaluation, identified risk of an undesirable result or evaluation expressed as “D, 61–70%”). Based on the expert agreement of the study advisor, the head of the department, the teacher of the theoretical subject, or the flight instructor, the student is provided with solution advice in favor of a better educational outcome. In case of disagreement between the experts, the majority decides on the solution. The mentioned fact is related to the implementation of the proposal in practice. Current academic information systems do not have such a proactive function.

Since the input data for the pilot student learning outcomes assessment model is obtained from subjects (teachers and instructors), it is therefore subjective and unclear. In addition, teachers or instructors evaluate students within their competencies and the psychophysiological characteristics of the individual. In this case, the learning outcomes are unclear. When developing intellectual systems, knowledge about the studied subject area is not complete and reliable. In this regard, the use of precise methods and statistical approaches based on the weighted sum does not allow taking into account the verbal inaccuracy and subjectivity of expert information, which in turn imposes restrictions on the quality display of knowledge for decision-making. To evaluate the training results, on the one hand, data on the training results of student pilots are used, and on the other hand, the experience and knowledge of flight training instructors, expressed in a qualitative form, are combined. This proves the fact that it is appropriate to use the theory of fuzzy sets, intellectual analysis of knowledge, and membership functions to display knowledge in the researched problem. It is well known that this theory allows you to reveal the uncertainties of the input data and effectively increase the accuracy of modeling.

Overview of Domestic and Foreign Research Studies

The work of not only theoretical training lecturers, flight instructors, and examiners but also scientific and pedagogical teams in the field of aviation education is based on international standards and recommended practice [9], the commitment of all ICAO state parties to ensure the highest possible degree of uniformity in regulations, standards, procedures, and organization in relation to aircraft, personnel, air travel, and ancillary services in all matters in which uniformity will facilitate and improve air navigation [10]. Aviation education enables the sharing of experiences found in the work on a methodology for implementing the competency-based training and assessment (CBTA) framework in aviation manpower planning and identification of the pilot performance gaps and the role of quality training [11]. Groppe and Brock researched the question of the cross-cultural interactions in the cockpit that represent a potential source of acculturative stress [12]. Cherngab et al. focused their attention on the relationship between civil pilots’ resilience, psychological well-being, and work performance [13]. Safety is the primary concern of the civil aviation industry. The study of Kılıç et al. aimed to explore the importance of learning from failures and improving the future performance of students by teaching this notion during an undergraduate course in the curriculum of a pilot training program [14]. The findings of the work of the Demeroitu et al. are interesting in the performance of simulator training [15]. Drone pilots form a separate target group within aviation education [16]. Novak et al. notes that in connection with the growing need to improve the quality and efficiency of the education process, it is necessary to address international cooperation in the education of experts in air transport [17]. Škvarekova et al. investigate the measurement of the pilot’s workload in the framework of improving safety and quality in civil aviation [18]. Kandera et al. are focused on the assessment of the mental health of pilots [19]. The paper of Zgodavova et al. highlights the importance of the human factor and its impact on the safety of flight operations [20]. The work of Korniienko et al. shares experience from the research of air rescue services around the world and the specific work of rescue helicopter pilots [21]. Tirpáková et al. investigated the impact of the COVID-19 pandemic on aviation, which also affected the didactics of aviation education [22]. Socha et al. presented research on the possibilities of developing pilot training as well as data-driven training [23]. In a paper by the researcher Weber, various models were proposed to evaluate the performance of airline pilots [24]. We also find inspiration in the contribution of Wen-Chin et al. whose purpose was to evaluate the relationship between the pilot’s mental workload and operational performance [25].

This paper is organized as follows: the formal statement of the problem and a hybrid complex model in terms of its components: a fuzzy model for evaluating the results of theoretical training of students; expert models for evaluating the results of training on flight simulators and practical flight training of students; an expert model for assessing the competence of practical flight training of students by means of observed behavior; a model of aggregation of raw data are described in Section 2. In Section 3, the research results are verified and tested on real data, and an example of evaluation is given. In Section 4, a discussion of the research results and an analysis of the advantages and disadvantages are introduced. The conclusion is drawn in Section 5. Ideas for improvements and future work are identified, namely the development of an innovative information technology module for the didactic system of aviation education and its support in the form of software.

2. Materials and Methods

2.1. Formal Formulation of the Problem

To improve the quality of individual results in the didactic system of aviation education, a formal formulation of a complex hybrid model for evaluating the results of student preparation is given.

First, the subjects of management are introduced, since the researched task belongs to the field of expert evaluation: experts are persons who analyze and evaluate the educational achievements of students to obtain an overall assessment of the competence of a graduate of aviation education; decision makers (DM) are management entities that make further decisions about the level of aviation education of students, which may be acceptable or include additional, special, or mandatory remedial training; and a system analyst is a person who configures the entire process of evaluating a hybrid model or an innovative module of a didactic system of aviation education.

Let us have a set of students (pilots) $P=\left(p_1;p_2;\dots ;p_n\right)$ to evaluate their success in aviation education. The hybrid complex model for evaluating the results of training of students for the purpose of timely response and targeted assistance to improve the quality of individual training and counseling of the student within the framework of aviation education is presented as a set of the following models: $M_{TS}$—a fuzzy model for evaluating the results of theoretical training of students; $M_{FS}$—expert model for evaluating the results of student training on flight simulators; $M_{PFT}$—expert model for evaluating the results of practical flight training of students; $M_{OB}$ is an expert model for evaluating the competence of practical flight training of students by means of observed behavior; and $M_{IA}$ is a model for aggregating raw data for deriving a general assessment of the quality of individual training of a student within the framework of aviation education. The mathematical model for evaluating the learning outcomes of students is proposed in the form of the following operator:

(1)$M\left(M_{TS},M_{FS}, M_{PFT},M_{OB},M_{IA}\right)\to f$

As a result, we obtain an output grade f, which is a general assessment of the quality of individual training of a student within the framework of aviation education, which takes into account the levels of theoretical training, different degrees of influence on the formation of the competence of a student, the risks of the results of training students on the flight simulator, the results training of practical flight training, the observed behavior of competence of practical flight training. The output assessment$f=\left(y; CA\right)$consists of $y$—a quantitative overall assessment of the competence of a graduate of aviation education and $CA$—a qualitative level of student training. Analysis of these indicators during the entire period of the training of a student makes it possible to reduce risks and increase the safety of the training results.

The hybrid model for evaluating the learning outcomes of students will be illustrated in the form of a structural diagram, Figure 1.

M_TS—a fuzzy model for evaluating the results of theoretical training of students; M_FS—expert model for evaluating the results of student training on flight simulators; and M_PFT—expert model for evaluating the results of practical flight training of students. M_OB is an expert model for evaluating the competence of practical flight training of students by means of observed behavior. M_IA is a model for aggregating raw data for deriving a general assessment of the quality of individual training of a student within the framework of aviation education. PSA—exercises for training piloting a single-engine aircraft; DUAL—flights with an instructor; and SOLO—separate flights without an instructor.

Figure 1 reflects the structural scheme of evaluating the results of training students to improve the quality of individual results in the didactic system of aviation education. The assessment is based on the results of 3-year theoretical training (bachelor’s degree), flight simulator training, practical flight training, and the results of acquired competencies in practical flight training. After that, the initial data are aggregated to derive a general assessment of the effectiveness of the quality of individual training of a student within the framework of aviation education. Based on the results, either the acceptance of the level of effectiveness of individual training of the student is determined, or it is recommended to undergo corrective training of the corresponding educational component. These aspects improve and increase the level of management of the educational process in the pilot training system, which entails a reduction in the risks of training results.

2.2. A Hybrid Model for Evaluating the Learning Outcomes of Students (Pilots)

The stages of designing a hybrid model for evaluating the learning outcomes of students are presented in terms of the given fuzzy and expert mathematical models to obtain an output assessment $f$$M_{TS}$—a fuzzy model for evaluating the results of theoretical training of students

For this model, a set of evaluation criteria is offered, representing the disciplines of the theoretical cycle. Theoretical training disciplines are selected according to the accredited curriculum of a higher educational institution or the curriculum of an organization approved for flight training by a national aviation authority, a so-called approved training organization (ATO). Our study uses the disciplines of aviation education in the “PILOT” study program of the Technical University of Košice (TUKE), Slovakia. The information criteria $K_{TS}=\left(K_{11},K_{12},\dots ,K_{21},\dots ,K_{3m}\right)$are entered, which are divided into three groups (years of study at the bachelor′s level) $C=\left(C_1,C_2,C_3\right)$.

According to each criterion, which is a theoretical discipline, after mastering it, the student receives a corresponding point percentage assessment and a linguistic expert assessment of the learning results $T=\left(\mathrm{A},\mathrm{B},\mathrm{C},\mathrm{D},\mathrm{E},\mathrm{FX}\right)$. For example, the results achieved by a student while studying a subject are evaluated according to six classification levels:

• “A”—excellent if 91–100%;

• “B”—very good if 81–90%;

• “C”—good if 71–80%;

• “D”—acceptable if 61–70%;

• “E”—sufficient if 51–60%;

• “FX”—not sufficiently if 0–50%.

A student completes a subject and receives credit if their results have been graded from “A” to “E”.

An expert on the didactic system of aviation education receives for some students (pilots) P a corresponding point percentage assessment for the mastered subjects of theoretical training. Below are a set of evaluation criteria for the theoretical disciplines of aviation education at the bachelor’s level in the “PILOT” study program.

$C_1$—The first year of study:

$K_{11}$—English 2;

$K_{12}$—Physics 2;

$K_{13}$—Aviation communication;

$K_{14}$—Aviation legislation;

$K_{15}$—Aviation meteorology 1;

$K_{16}$—Aviation meteorology 2;

$K_{17}$—Air navigation 1;

$K_{18}$—Aviation rules 1;

$K_{19}$—Mathematics 2;

$K_{110}$—Physical Education 1;

$K_{111}$—Physical Education 2;

$K_{112}$—Fundamentals of computer science;

$K_{113}$—Basics of flight 1.

$C_2$—The second year of study:

$K_{21}$—Economics;

$K_{22}$—Airports and transport infrastructure;

$K_{23}$—Organization of air traffic 1;

$K_{24}$—Search and rescue service;

$K_{25}$—Devices and systems 1;

$K_{26}$—Avionics 1;

$K_{27}$—Aircraft construction;

$K_{28}$—Aeronautics 2;

$K_{29}$—Air traffi—c organization 2;

$K_{210}$—Semester project 1;

$K_{211}$—Aviation engines.

$C_3$—The third year of study:

$K_{31}$—Flight planning and monitoring;

$K_{32}$—Operational procedures in commercial air transport;

$K_{33}$—Avionics 2;

$K_{34}$—Flight technical characteristics;

$K_{35}$—Fundamentals of flight 2;

$K_{36}$—Colloquial exam;

$K_{37}$—Air transport process;

$K_{38}$—Defense of the final thesis;

$K_{39}$—Final thesis;

$K_{310}$—Weight and balance;

$K_{311}$—Human capabilities and limitations.

A set of percentage points for mastered subjects of theoretical training is obtained, which is denoted by $O_{TS}=\left(O_{i11},O_{i12},\dots ,O_{i21},\dots ,O_{i311}\right),i=\overline{1,n}$ respectively for a set of pilot students $P=\left(p_1;p_2;\dots ;p_n\right)$according to evaluation criteria $K_{TS}=\left(K_{11},K_{12},\dots ,K_{21},\dots ,K_{311}\right)$. In general, the set of criteria is open and represents the official curriculum, and their number does not affect the calculation of the complex hybrid model.

A fuzzy model for evaluating the results of theoretical training of students is offered in the form of a step-by-step algorithm.

First step. Introduction of the “Importance of Discipline” for the level of pilot competence

To determine the level of competence for each cycle of theoretical training, the theory of fuzzy sets and fuzzy logic procedures are used, since each assessment on a 100-point scale, in a certain range, is obtained from subjects (teachers and trainers) and has a fuzzy, fuzzy character. In addition, if you analyze the results of the assessment of individual subjects, then you can see that for some teachers, grades from a scale of 80–100 prevail for students of different levels of training and others from 60–90, etc. Of course, all subject’s accredited training programs are important, but each of them has a certain degree of influence on the formation of each student pilot competency. The objective reality is that assessment results depend on many factors. Of course, it is impossible to conduct research on the teaching of the same subject by different teachers to the same students. However, it is known from the theory of expert evaluation that different experts give their conclusions within the limits of their competencies and the psychophysiological characteristics of the individual. Therefore, to obtain a real level of quality training of students, the following approach is proposed.

The point “Importance of Discipline” $T=\left\{t_i,i=1,2,\dots ,m\right\}$ is considered, which represents the evaluation of all the evaluation criteria of the theoretical disciplines of aviation education that could satisfy the DM. “Importance of Discipline” is characterized by the fact that each of the theoretical subjects has different degrees of influence on the formation of the competence of a pilot student.

The set of “Importance of Discipline” $T=\left\{t_i,i=1,2,\dots ,m\right\}$ will be chosen by the expert independently, analyzing each theoretical discipline and the teacher who practices it, while choosing the optimal value. For example, some teachers have a maximum score of 85% in the subject. Conversely, in some subjects, the minimum grade is 75%. It cannot reflect the objective reality of the quality of the graduate’s knowledge.

Second step. Calculation of estimates of the proximity of the student’s learning results to the “Importance of Discipline”

The approach to building the membership function is described as follows. A set of values is determined, which are relative estimates of the proximity of the elements of point percentage estimates for the mastered subjects of theoretical training $O_{TS}=\left(O_{i11},O_{i12},\dots ,O_{i21},\dots ,O_{im}\right)$ to the corresponding element of “Importance of Discipline”:

(2)$\mu \left(O_{ij}\right)=1-\frac{\left|t_j-O_{ij}\right|}{\max \left\{t_j-\mathrm{nim}\left(K_j\right);\max \left(K_j\right)-t_j\right\}},i=\overline{1,n}, j=\overline{1,m}.$

where $\mathrm{nim}\left(K_j\right)$ is the lowest grades (not necessarily the minimum) received by students in the corresponding subject $K_j\mathrm{and}$ $\max \left(K_j\right)$ is the highest marks (not necessarily the maximum) in the corresponding subject $K_j$.

The determined matrix $\mu =\mu \left(O_{ij}\right)$ characterizes by columns the relative evaluations of the proximity of the training results of the student (pilot) $P_i$ to the “Importance of Discipline” T for each specific subject and removes the issue of different evaluation scales. As a result, $\mu \left(O_{ij}\right)\in$[0; 1].

Third step. Fuzzification of input hybrid data

The term set of linguistic expert evaluations of learning outcomes $T=\left(\mathrm{A},\mathrm{B},\mathrm{C},\mathrm{D},\mathrm{E},\mathrm{FX}\right)$ is determined on a percentage scale according to the following content, the larger the value, the higher the level of the criterion. According to the above, the following division of intervals is proposed: $FX$—[0; 50], $E$—[51; 60], $D—$[61; 70], $C—$[71; 80], $B—$[81; 90], and $A—$[91; 100].

The dependence of linguistic expert evaluations of the students’ training results on the criterion of evaluating theoretical subjects and quantitative evaluation of the “Importance of Discipline” will be carried out with the help of intellectual analysis of knowledge and functions of belonging. From a formal point of view, there is an uncertainty of the “high level” type, which is described by the membership function “value x greater”. It is natural to express the known data from the quadratic S-spline of the membership function:

(3)${\epsilon }_{ij}=\frac{1}{100}·\{\begin{array}{cc}\sqrt{\frac{\mu \left(O_{ij}\right)}{2}}\left(b-a\right)+a,& 0\le \mu \left(O_{ij}\right)\le 0.5;\\ b-\sqrt{\frac{1-\mu \left(O_{ij}\right)}{2}}\left(b-a\right),& 0.5<\mu \left(O_{ij}\right)\le 1.\end{array}$

Here, $a;b$ are the values of the ends of the intervals depending on the linguistic variable T. The larger the value of ${\epsilon }_{ij}\in \left[0;1\right]$, the higher the level of the criterion $K_j$. In addition, the value ${\epsilon }_{ij}$ represents the disclosure of the uncertainty of the educational achievement of the student (pilot) $P_i$ in the relevant subject $K_j$, $i=\overline{1,n}, j=\overline{1,m}$.

Thus, there was a transition from linguistic evaluations and evaluations of the closeness of the learning outcomes of the student (pilot) $P_i$ to the “Importance of Discipline” and one normalized evaluation.

Fourth step. Considering the importance of theoretical disciplines in the acquisition of relevant aviation education competencies

Let DM have its considerations regarding the importance of the coefficients for each discipline $\left\{w_1,w_2,\dots ,w_m\right\}$, from the interval [1;10]. If DM does not need this, then the criteria are considered equally important. For data comparison, normalized weighting factors are determined:

(4)$\overline{w_j}=\frac{w_j}{{\sum }_{j=1}^m{w}_j}, j=\overline{1,m}.$

where the condition is met $\sum _{j=1}^{\mathrm{m}}\overline{w_j}=1$.

It is noted that the components of the vector $\left\{w_1,w_2,\dots ,w_m\right\}$ can be selected in other ways, depending on the specific situation.

Fifth step. Defuzzification of the data

An aggregate risk assessment of aviation education results is calculated. For this, it is proposed to use convolutions, depending on the considerations of the DM regarding the risk of aviation education results:

(5)${\delta }_{1i}=\frac{1}{{\sum }_{j=1}^m\frac{\overline{w_j}}{{\epsilon }_{ij}}};$

(6)${\delta }_{2i}={\displaystyle \prod }_{j=1}^m{\left({\epsilon }_{ij}\right)}^{\overline{w_j}};$

(7)${\delta }_{3i}={\displaystyle \sum }_{j=1}^m\overline{w_j}·{\epsilon }_{ij};$

(8)${\delta }_{4i}=\sqrt{{\displaystyle \sum }_{j=1}^m\overline{w_j}·{({\epsilon }_{ij})}^2},i=\overline{1,n}.$

where ${\delta }_1$—pessimistic considerations regarding the risk of aviation education results; ${\delta }_2$—careful considerations regarding the risk of aviation education results; ${\delta }_3$—average considerations regarding the risk of aviation education results; and ${\delta }_4$—optimistic considerations regarding the risk of aviation education results.

Note that steps 4–5 are given for the calculation of the entire cycle of theoretical training. In the case of calculating intermediate values, for example in the 1st or 2nd year of study, the index j runs through the value of the number of disciplines in the corresponding year of study.

To derive the linguistic level $R$ of risks within the framework of the theoretical mastery of subjects, the following linguistic conclusions are offered: $r_1$= “insignificant risk of aviation education results”; $r_2$= “low risk of aviation education results”; $r_3$= “average risk of aviation education results”; $r_4$= “high risk of aviation education results”; and $r_5$= “critical risk of aviation education results”.

As a result of the verification of the fuzzy model for evaluating the results of the theoretical training of students on real data (the didactic system of aviation education), the levels for comparing the $\delta$ ratings with the linguistic $R=\left\{r_1,r_2,r_3,r_4,r_5\right\}$ were established as follows: $\delta \in$ [0.5025; 0.6]—$r_5$; $\delta \in$ (0.6; 0.7]—$r_4$; $\delta \in$ (0.7; 0.8]—$r_3$; $\delta \in$ (0.8; 0.9]—$r_2$; and $\delta \in$ (0.9; 1]—$r_1$.

$M_{FS}$—Expert model for evaluating the results of student pilot training on flight simulators

For this model, the evaluation criterion $K_{FS}$—“Training on the simulator 3” is proposed. Without reducing the generality, this discipline is studied at the university TUKE (Slovakia) during the training of students in the specialty “PILOT” study program”, with one credit for pilot students of the 3^rd year of study, in the winter semester. Similar to the theoretical training, the results achieved by the student during the study of the subject “Training on the simulator 3” are evaluated according to six classification levels $T=\left(\mathrm{A},\mathrm{B},\mathrm{C},\mathrm{D},\mathrm{E},\mathrm{FX}\right)$ is determined on a percentage scale: $FX$—[0; 50], $E$—[51; 60], $D—$[61; 70], $C—$[71; 80], $B—$[81; 90], and $A—$[91; 100]. We denote the obtained estimate by $O_{iFS}$, respectively for a student, pilot $p_i, i=\overline{1,n}.$ In addition, let the training instructor make their judgment about the risks to the student pilot training results in the flight simulator. For such a conclusion, we introduce the linguistic variable $FS=\left\{f{s}_1;f{s}_2;\dots ;f{s}_5\right\}$, where: $f{s}_1$—insignificant risk of training results on the flight simulator; $f{s}_2$—low risk of training results on the flight simulator; $f{s}_3$—average risk of training results on the flight simulator; $f{s}_4$—high risk of training results on the flight simulator; $f{s}_5$—critical risk of training results on the flight simulator.

First, let us complete fuzzification of the results of learning the percentage scale. For this purpose, it is proposed to use intellectual analysis of knowledge with the help of membership functions. For example, it is natural to use a quadratic S-spline:

(9)$\mu \left(O_{iFS}\right)=\{\begin{array}{cc}0,& O_{iFS}\le 51;\\ 2{\left(\frac{O_{iFS}-51}{44}\right)}^2,& 51<O_{iFS}\le 73;\\ 1-2{\left(\frac{95-O_{iFS}}{44}\right)}^2,& 73<O_{iFS}<95,\\ 1,& O_{iFS}\ge 95.\end{array},i=\overline{1,n}.$

Thus, we will obtain the normalized output estimates $\mu \left(O_{iFS}\right)$ from the interval [0; 1] for $n$ student pilots.

Next, the normalized baseline score and the training instructor’s reasoning are aggregated using the following membership function:

(10)${\theta }_i=\theta \left(\mu \left(O_{iFS}\right)\right)=\{\begin{array}{cc}0,& \mu \left(O_{iFS}\right)<0;\\ {\left(\mu \left(O_{iFS}\right)\right)}^k,& 0\le \mu \left(O_{iFS}\right)<1;\\ 1,& \mu \left(O_{iFS}\right)\ge 1.\end{array}i=\overline{1,n}.$

where k is the risk threshold of training results on the simulator, the value of which varies depending on the expert opinion of the $FS$. This threshold can be obtained by training on real result data. For example, let us experimentally set: k $=\frac{2}{9}$ when we have expert opinion $f{s}_1$; k $=1$ when we have expert opinion $f{s}_2$; k$=\frac{4}{9}$—expert opinion $f{s}_3$; k$=\frac{5}{9}$—expert opinion $f{s}_4$; k$=\frac{11}{5}$—expert opinion $f{s}_5$.

Thus, aggregated normalized estimates ${\theta }_i, i=\overline{1,n}$ from the interval [0; 1], regarding the evaluation of student pilot training results on flight simulators.

The presented research is universal and is not limited only to studies in higher educational institutions. In this regard, if, when evaluating the training results of students, the training instructor does not have the opportunity to express their considerations regarding the risks of the training results, then the calculation according to Formula (10) is skipped, and the value is taken for further calculations $\mu \left(O_{iFS}\right)$.

$M_{PFT}$—Expert model for evaluating the results of practical flight training of students

Evaluation of the results of practical flight training of students is carried out by flight training instructors. Thus, to obtain a “private pilot license” (PPL) (min. 45 flight hours), you need to acquire skills in two stages: $PF{T}_1$—aircraft piloting technique (to learn to fly an aircraft, these are exercises PSA/01 to PSA/17; PSA–piloting a single-engine aircraft); $PF{T}_2$—aircraft navigation control (navigational flights on selected routes, these are exercises PSA/18 to PSA/24).

Thus, for the expert model of evaluating the results of practical flight training of students, we will have two groups of criteria: $PF{T}_1=\left\{{\mathrm{PSA}}_1; {\mathrm{PSA}}_2;\dots ;{\mathrm{PSA}}_{17}\right\}$ and $PF{T}_2=\left\{{\mathrm{PSA}}_{18}; {\mathrm{PSA}}_{19};\dots ;{\mathrm{PSA}}_{24}\right\}$.

Both stages of the training are performed as flights with an instructor (DUAL) or as separate flights without an instructor (SOLO).

The evaluation of the performed exercises can be: “completed” or “not completed”. Another rating scale used in the “PILOT” study program of TUKE University (Slovakia) uses five levels of flight training evaluation: “1” excellent (in academic plan A 91–100%); “2” very good (B 81–90%); “3” good (C 71–80%); “4” acceptable (D 61–70%); “5” not sufficiently (FX).

Based on these marks for all of the exercises, the student pilot’s overall flight training score is determined, which is reported by the approved training organization (ATO) to the National Aviation Authority for the purpose of obtaining a pilot license.

Formally, the expert model for evaluating the results of practical flight training of students will be presented as follows.

Let us denote the grades received by the student (pilot) $p_i, i=\overline{1,n}$on PSA exercises as follows:

• Flights with the instructor $O_{iPF{T}_1}\left(dual\right)=\left\{O{D}_{iPSA1};O{D}_{iPSA2};\dots ;O{D}_{iPSA17}\right\}$, $O_{iPF{T}_2}\left(dual\right)=\left\{O{D}_{iPSA18};O{D}_{iPSA19};\dots ;O{D}_{iPSA24}\right\}$;

• Individual flights without an instructor $O_{iPF{T}_1}\left(solo\right)=\left\{O{S}_{iPSA1};O{S}_{iPSA2};\dots ;O{S}_{iPSA17}\right\}$, $O_{iPF{T}_2}\left(solo\right)=\left\{O{S}_{iPSA18};O{S}_{iPSA19};\dots ;O{S}_{iPSA24}\right\}$.

where all the assessments are $\left(O{D}_{iPSA1},\dots ,O{D}_{iPSA24},O{S}_{iPSA1},\dots ,O{S}_{iPSA24}\right)\in \left\{1;2;3;4;5\right\}$.

In the first stage, we will calculate the average values for the exercises within the selected sets:

(11)$\{\begin{array}{l}O{A}_{iPF{T}_1}\left(dual\right)=\frac{1}{17}(O{D}_{iPSA1}+O{D}_{iPSA2}+\dots +O{D}_{iPSA17})\\ O{A}_{iPF{T}_2}\left(dual\right)=\frac{1}{6}(O{D}_{iPSA18}+O{D}_{iPSA19}+\dots +O{D}_{iPSA24})\\ O{A}_{iPF{T}_1}\left(solo\right)=\frac{1}{17}(O{S}_{iPSA1}+O{S}_{iPSA2}+\dots +O{S}_{iPSA17})\\ O{A}_{iPF{T}_2}\left(solo\right)=\frac{1}{6}(O{S}_{iPSA18}+O{S}_{iPSA19}+\dots +O{S}_{iPSA24})\end{array}i=\overline{1,n}.$

In the second stage, the overall score in the recommendations of the ATO is calculated:

(12)$O{A}_{iPFT}=\frac{1}{4} \left(\begin{array}{c}O{A}_{iPF{T}_1}\left(dual\right)+O{A}_{iPF{T}_2}\left(dual\right)+\\ +O{A}_{iPF{T}_1}\left(solo\right)+O{A}_{iPF{T}_2}\left(solo\right)\end{array}\right),i=\overline{1,n}.$

In the final stage, to compare the data, it is proposed to model the uncertainty with a membership function using a quadratic Z-spline:

(13)${\omega }_i=\{\begin{array}{cc}1,& O{A}_{iPFT}\le 1;\\ 1-\frac{2}{9}{\left(O{A}_{iPFT}-1\right)}^2,& 1<O{A}_{iPFT}\le 2.5;\\ \frac{2}{9}{\left(4-O{A}_{iPFT}\right)}^2,& 2.5<O{A}_{iPFT}\le 4,\\ 0,& O{A}_{iPFT}>4.\end{array},i=\overline{1,n}.$

The resulting aggregated normalized assessment of the results of practical flight training of students has the following meaning: when the value of the assessment approaches 1, then the student has acquired the best skills in the stages of aircraft piloting technique and aircraft navigation control.

Thus, at the output of the expert model for evaluating the results of practical flight training, we have normalized and compared ratings ${\omega }_i$by pilots $p_i, i=\overline{1,n}$.

$M_{OB}$—Expert model for evaluating the competence of practical flight training of students by means of observed behavior

A set of criteria for evaluating the competence of practical flight training of students by means of observed behavior is proposed, which is divided into nine groups $G=\left(G_1,G_2,\dots ,G_9\right)$. The assessment criteria in each group G are presented in the form of a question to describe the competence. Indicators for “observed behavior” are used from the officially published document Competency Assessment and Evaluation for Pilots, Instructors and Evaluators/Guidance material published by the International Air Transport Association (IATA) [26].

The document is based on an idea: an adapted competency model, which is a group of competencies with their associated description and performance criteria adapted from an ICAO (International Civil Aviation Organization) competency framework that the ATO approved training organization/AOC (air operator certificate/air operator certificate holder (operator) uses to develop competency-based training and assessment for pilots and instructor–evaluators.

Some fragments of indicators for evaluating the competence of practical flight training of students by means of observed behavior are given. All other indicators are given in [1].

Group $G_1$—application of knowledge demonstrating knowledge and understanding of relevant information, operating instructions, aircraft systems, and environment. This group consists of seven criteria $O{B}_{11}-O{B}_{17}$. For example, $O{B}_{15}$—a student knows where to get the necessary information.

Group $G_2$—application of procedures and compliance with rules, which is determined following official operating instructions and relevant regulations. This group also consists of seven criteria $O{B}_{21}-O{B}_{27}$. For example, $O{B}_{22}$—the student applies appropriate operational instructions, procedures, and methods promptly.

Group $G_3$—communication. Communicates using appropriate means in the work environment, both in staff and non-staff situations. This group consists of ten criteria $O{B}_{31}-O{B}_{310}$. For example, $O{B}_{32}$—student appropriately chooses what, when, how and with whom to communicate.

Group $G_4$—aircraft flight path control and automation. Controls the flight path using automation. This group consists of six criteria $O{B}_{41}-O{B}_{46}$. For example, $O{B}_{43}$—student safely controls the flight path to achieve optimal performance.

Group $G_5$—control of the flight path of the aircraft with manual control. This group consists of seven criteria $O{B}_{51}-O{B}_{57}$. For example, $O{B}_{55}$—student maintains the planned flight path during manual flight while managing other tasks and distractions.

Group $G_6$—leadership and teamwork. Influences others to achieve a common goal and collaborates to accomplish team goals. This group consists of eleven criteria $O{B}_{61}-O{B}_{611}$. For example, $O{B}_{65}$—student pilot gives and receives constructive feedback.

Group $G_7$—problem-solving and decision-making. This group consists of nine criteria $O{B}_{71}-O{B}_{79}$. For example, $O{B}_{76}$—the student pilot uses appropriate and timely decision-making techniques.

Group $G_8$—perception, awareness, and management of information to predict its impact on work. This group consists of seven criteria $O{B}_{81}-O{B}_{87}$. For example, $O{B}_{84}$—the student checks the accuracy of the information for errors.

Group $G_9$—maintaining an available workload by prioritizing and distributing tasks using appropriate resources. This group consists of nine criteria $O{B}_{91}-O{B}_{99}$. For example, $O{B}_{82}$—student effectively plans, prioritizes, and schedules appropriate tasks.

To present an expert model for assessing the competence of practical flight training of students through observed behavior, we will present the following approach. The idea is that in some cases the results of the assessment of competencies and management of threats and errors may not be relevant to the assessment of competence to the learning objectives of the session. In this case, the flight instructor must evaluate the associated “observable behavior” of each competency with the following values, while determining: $q_i—$the number of “observable behaviors” demonstrated by the corresponding student pilot $p_i$ when they were required; ${\phi }_i—$is the frequency of “observed behavior” demonstrated by the student (pilot) $p_i$, when they were required.

The following linguistic variables are proposed for flight training instructor assessment of the quantity ($q_i$) and frequency (${\phi }_i$) “observed behavior” [1]: $L_q=${few, hardly any; some; many; most; all, almost all} and $L_{\phi }=${rarely; occasionally; regularly; very often; always, almost always}.

Next, it is necessary to associate the results of the evaluation of the linguistic variables $L_q$ and $L_{\phi }$ with a certain scale. For this, the following characteristic functions are considered, respectively.

(14)$g\left(L_{q_i}\right)=\{\begin{array}{c}\begin{array}{cc}0.2& if “few, hardly any”;\\ 0.4& if “some”;\\ 0.6& if “many”;\\ 0.8& if “most”;\\ 1& if “all, almost all”.\end{array} \end{array}$

(15)$g\left(L_{{\phi }_i}\right)=\{\begin{array}{c}\begin{array}{c}\begin{array}{cc}0.2& if “rarely”;\\ 0.4& if “occasionally”;\\ 0.6& if “regularly”;\\ 0.8& if “very often”;\\ 1& if “always, almost always”.\end{array} \end{array} \end{array}$

The purpose of this defined normalized numerical scale is to enable further comparison and calculation.

Furthermore, to aggregate the values $g\left(L_{q_i}\right)$, $g\left(L_{{\phi }_i}\right)$ within the criterion $O{B}_h,$(h–number of criteria), similarly, intellectual analysis of knowledge is used by modeling the uncertainty of the “average value” type on based on multidimensional membership functions. For example, such modeling is based on a cone-shaped membership function, and the value of the center of the base of the cone is a unit vector, and the scaling is based on the coordinates of the vector $\left(g\left(L_{q_{hi}}\right); g\left(L_{{\phi }_{hi}}\right)\right), i=\overline{1,n}$; $h=\overline{1,73}$ is equal to $\left(2;2\right)$:

(16)$H_{hi}=\{\begin{array}{c}1-g_{hi}, if g_{hi}<1,\\ 0, else.\end{array}h=\overline{1,73}.$

where $g_{hi}=\frac{1}{2}·\sqrt{{\left(g\left(L_{q_i}\right)-1\right)}^2+{\left(g\left(L_{{\phi }_i}\right)-1\right)}^2}$.

In this way, an aggregated value was obtained for each criterion. Next, we will use the weighted average amount to obtain one rating for students:

(17)${\Delta }_i=\frac{1}{73}{\displaystyle \sum }_{h=1}^{73}{H}_{hi}. i=\overline{1,n}.$

From a mathematical point of view, the obtained initial estimates will be from the interval ${\Delta }_i\in$[0.434; 1], this explains the setting of the base of the cone and its scaling.

To comply with the relevant standards, the obtained value is compared with the following linguistic assessment of $HW$ competence with the following linguistic conclusions: $h{w}_1$= “exemplary manner”; $h{w}_2$= “effectively”; $h{w}_3$= “adequately”; $h{w}_4$ = “minimal acceptable”; $h{w}_5$ = “ineffectively”.

According to industry best practice, the ATO policy should be as follows [1]: the prescribed standard is $h{w}_3$ for each pilot and the minimum acceptable standard is—$h{w}_4$. As a result of the verification of the expert model for assessing the competence of practical flight training of students using observed behavior on real data (the didactic system of aviation education) and the above industry practices, the levels for comparing the $\Delta$ scores with the linguistic $HW=\left\{h{w}_1,h{w}_2,h{w}_3,h{w}_4,h{w}_5\right\}$ are as follows: $\Delta \in$ [0.434; 0.58]—$h{w}_5$; $\Delta \in$ (0.58; 0.64]—$h{w}_4$; $\Delta \in$ (0.64; 0.78]—$h{w}_3$; $\Delta \in$ (0.78; 0.86]—$h{w}_2$; $\Delta \in$ (0.86; 1]—$h{w}_1$.

At the same time, if the student receives: $h{w}_5$ then remedial training is required; $h{w}_4$then you need to pay attention and recommend remedial training; and $h{w}_1-h{w}_3$ then corrective training is not required.

$M_{IA}$—Model for aggregating raw data for deriving a general assessment of the quality of individual training of a student (pilot) within the framework of aviation education

At the input of the model for aggregating output data to derive a general assessment of the quality of individual training of a student (pilot) within the framework of aviation education, we have normalized and compared values obtained for students (pilots) $p_i, i=\overline{1,n}$ based on the above models, namely: ${\delta }_i$—aggregate risk assessment of the results of aviation education within the framework of theoretical mastery of subjects; ${\theta }_i$—aggregated normalized score for evaluating the results of student pilot training on flight simulators; ${\omega }_i$—aggregated normalized evaluation of the results of practical flight training of students; ${\Delta }_i$—output assessments of the competence of practical flight training of students using observed behavior. All input data $\left\{{\delta }_i;{\theta }_i;{\omega }_i;{\Delta }_i\right\}$ are normalized and compared.

The following approach is proposed to obtain the output estimate of $f$.

In the first stage, let DM need to set the weighting coefficients $\left\{w_1,w_2,w_3,w_4\right\}$ for each model of evaluating the learning outcomes of pilot students $M_{TS},M_{FS}, M_{PFT},M_{OB}$ from the interval [1; 10]. Normalized weighting factors are determined for data comparison:

(18)$\overline{w_{\propto }}=\frac{w_{\propto }}{{\sum }_{\propto =1}^4{w}_{\propto }},\propto =\left\{1,2,3,4\right\},\overline{w_{\propto }}\in \left[0;1\right].$

After that, one quantitative overall assessment of the quality of individual training of a student (pilot) in the framework of aviation education is calculated, separately for each student pilot $P=\left(p_1;p_2;\dots ;p_n\right)$, using a weighted average convolution:

(19)$y\left(p_i\right)=\overline{w_1}\cdot {\delta }_i+\overline{w_2}\cdot {\theta }_i+\overline{w_3}\cdot {\omega }_i+\overline{w_4}\cdot {\Delta }_i.t=\overline{1,n}.$

We note that if the DM does not need to distinguish the importance of assessment models, then the weighting factors are balanced, and Formula (19) will express the arithmetic mean value.

The following term-set of linguistic variables is proposed to derive the qualitative level of training of the pilot $CA=\left\{c{a}_1; c{a}_2;\dots ;c{a}_5\right\}$: $c{a}_1$ = “high level of individual training of a pilot”; $c{a}_2$= “the level of individual training of a pilot is above average”; $c{a}_3$ = “average level of individual training of a pilot”; $c{a}_4$ = “low level of individual training of the pilot”; $c{a}_5$ = “unacceptable level of individual training of a pilot”. As a result of verification on real data, using the didactic system of aviation education, the levels for comparing the y score with the linguistic $CA$ were established as follows: $y\in$ [0.5; 0.6]—$c{a}_5$; $y\in$ (0.6; 0.7]—$c{a}_4$; $y\in$ (0.7; 0.8]—$c{a}_3$; $y\in$ (0.8; 0.9]—$c{a}_2$; $y\in$ (0.9; 1]—$c{a}_1$.

DM decision levels can always be changed without violating the minimum requirements of the approved training standards under the supervision of the national aviation authority.

Thus, the vagueness of input expert evaluations is revealed, thereby improving the effectiveness of the model, which can derive a quantitative overall assessment of the quality of individual training of a student. All this makes it possible to increase the degree of validity of making further management decisions regarding the possibility of improving the quality of pilot training, and individual study counseling. In addition, the level of management of the educational process in the pilot training system increases, which entails a reduction in the risks of training results.

Another important aspect is that the initial grades are stored in the database in the didactic system of aviation education. When obtaining a sufficient number of them, it is possible to improve the settings of the model parameters by applying the methods of neuro-fuzzy networks and their training. The hybrid model has a modular principle, and its components can be replaced by other models or not all involved in the evaluation process. Therefore, the presented hybrid model for evaluating learning outcomes can be easily developed and adapted for other students. For example, doctors, military personnel, ship captains, and others for whom the acquisition of practical skills is an important component of training.

3. Results

The results of the research were tested and verified on the real data of students (pilots) of the Technical University of Košice (Slovakia) in the “PILOT” study program at the Faculty of Aeronautics. Information from the repository of 696 individual results of undergraduate students (pilots) in aviation education in the period 2009–2023 was used in the creation of the model.

For visual interpretation and giving an example of evaluating the learning results of students (pilots) using the developed hybrid model, five students (male and female) $P=\left(p_1;p_2;\dots ;p_5\right)$ were selected. Real data on the success of theoretical and flight training were taken from the didactic system of aviation education. The results of point percentage and linguistic evaluations for the mastered subjects of theoretical training and on the flight simulator of the five selected students, for individual years of the 3-year bachelor’s study in the “PILOT” study program, are presented in Table 1.

The calculation is carried out based on the developed hybrid model for evaluating the learning outcomes of students. For this purpose, evaluation is carried out separately for fuzzy and expert models $M_{TS},M_{FS}, M_{PFT},M_{OB}$. Deriving a general assessment of the quality of individual training of a student within the framework of aviation education is carried out using $M_{IA}$—a model of aggregation of initial data. Assessments of learning outcomes were obtained both from the didactic system of aviation education and from flight training instructors and experts with more than 10 years of experience in aviation education to express their opinion on some criterion.

3.1. Evaluation with the Fuzzy Model for Evaluating the Results of Theoretical Training of Students (Pilots)—$M_{TS}$

It is proposed to consider the fuzzy model of evaluating the results of theoretical training of pilot students in the form of a step-by-step algorithm.

First step. Introduction of the “Importance of Discipline” for the level of pilot competence

Let the expert set the “Importance of Discipline” $T$ for the pilot’s competence level, as well as the lowest ($\mathrm{nim}\left(K_j\right)$) and highest scores ($\max \left(K_j\right)$), Table 2.

Second step. Calculation of estimates of the proximity of the student’s learning results to the “Importance of Discipline”

The relative estimates of the proximity $\mu \left(O_{ij}\right)$of the training results of the student $p_i$ to the “Importance of Discipline” T for each specific subject are calculated according to Formula (2), Table 2. Moreover, we will take the credited score as 91%.

Third step. Fuzzification of input hybrid data

For fuzzification of input hybrid data, interval division is used: $FX$—[0; 50], $E$—[51; 60], $D—$[61; 70], $C—$[71; 80], $B—$[81; 90], and $A—$[91; 100]. The dependence of the linguistic expert evaluations of the student’s training results on the criterion for evaluating theoretical subjects and the quantitative evaluation of the “importance of the discipline” will be carried out according to Formula (3).

Fourth step. Considering the importance of theoretical disciplines in the acquisition of relevant aviation education competencies

Let DM can specify the importance of the coefficients for each discipline $\left\{w_1,w_2,\dots ,w_m\right\}$, from the interval [1;10]. Normalized weighting factors are determined according to Formula (4).

Fuzzification of input hybrid data, coefficient weights, and normalized weight coefficients are presented in Table 3.

Fifth step. Defuzzification of the data

An aggregate risk assessment of the results of aviation education is calculated, for example, according to Formula (7)–average considerations regarding the risk of the results of aviation education: ${\delta }_{31}=$ 0.9587; ${\delta }_{32}=$ 0.8647; ${\delta }_{33}=$ 0.9572; ${\delta }_{34}=$ 0.8471; ${\delta }_{35}=$ 0.8109.

To derive the linguistic level $R$ of risks within the framework of the theoretical mastery of subjects, comparing the initial grades, we obtain that student (pilots) $p_2, p_4, p_5$have “a low risk of aviation education results”, and $p_1, p_3$—“insignificant risk of aviation education results”.

3.2. Evaluation with the Expert Model for Evaluating the Results of Student Training on Flight Simulators—$M_{FS}$

According to the evaluation criterion $K_{FS}$—“Training on the simulator 3”, we will receive the following assessments: $O_{1FS}=$ 95; $O_{2FS}=$ 93; $O_{3FS}=$ 85; $O_{4FS}=$ 92; $O_{5FS}=$ 83. In addition, the training instructor expresses his considerations regarding the risks of student training results on the flight simulator: $\{p_1,p_4\}\in f{s}_1$—insignificant risk of training results on the flight simulator; $\{p_2, p_3,p_5\}\in f{s}_2$—low risk of training results on the flight simulator.

Fuzzification of the data of the results of the percentage scale is carried out according to Formula (9): $\mu \left(O_{1FS}\right)=$ 1; $\mu \left(O_{2FS}\right)=$ 0.996; $\mu \left(O_{3FS}\right)=$ 0.897; $\mu \left(O_{4FS}\right)=$ 0.991; $\mu \left(O_{5FS}\right)=$ 0.851.

Next, the normalized initial assessment and the reasoning of the training instructor are aggregated using the membership function according to Formula (10): ${\theta }_1=$ 1; ${\theta }_2=$ 0.996;${\theta }_3=$ 0.897; ${\theta }_4=$0.998; ${\theta }_5=$ 0.851.

3.3. Evaluation with the Expert Model for Evaluating the Results of Practical Flight Training of Students (Pilots)—$M_{PFT}$

Input data for the expert model are obtained from flight training instructors. Students for the PPL license acquire skills in two stages: $PF{T}_1$—aircraft piloting technique (exercises PSA/01 to PSA/17) and $PF{T}_2$—aircraft navigation control (exercises PSA/18 to PSA/24). The average values for the exercises are presented in Table 4.

According to the model, the total score is calculated according to Formula (12): $O{A}_{1PFT}=$ 1; $O{A}_{2PFT}=$ 2; $O{A}_{3PFT}=$ 3; $O{A}_{4PFT}=$ 3.25; $O{A}_{5PFT}=$ 2.

Furthermore, for data comparison, values are calculated using the quadratic Z-spline according to Formula (13): ${\omega }_1=$ 1; ${\omega }_2=$ 0.78; ${\omega }_3=$ 0.22; ${\omega }_4=$ 0.13; ${\omega }_5=$ 0.78.

3.4. Evaluation with the Expert Model for Evaluating the Competence of Practical Flight Training of Students (Pilots) by Means of Observed Behavior—$M_{OB}$

Similarly, the input data for the expert model were obtained from flight training instructors for each student. The “observed behavior” scores were defined as the$q_i$number and ${\phi }_i$ frequency of the “observed behavior” exhibited by student pilot $p_i$, when requiring $i=\overline{1,5}$. According to the following linguistic variables $L_q=${few, hardly any; some; many; most; all, almost all} and $L_{\phi }=${rarely; occasionally; regularly; very often; always, almost always}. The input data of the results of the evaluation of the linguistic variables $L_q$ and $L_{\phi }$ relate to the quantitative evaluations using the characteristic functions (14) and (15). Next, the values $g\left(L_{q_i}\right)$ and $g\left(L_{{\phi }_i}\right)$ are aggregated within the criteria $O{B}_h,$according to Formula (16). Input data and aggregation results are presented in Table 5.

Furthermore, to obtain one grade for students, the weighted average amount is used according to Formula (17): ${\Delta }_1=$ 0.92; ${\Delta }_2=$ 0.881; ${\Delta }_3=$ 0.942; ${\Delta }_4=$ 0.847; ${\Delta }_5=$ 0.825. The obtained values are compared with the linguistic competence assessment of $HW$; it is obtained that the students $\{p_1;p_2;p_3\}\in h{w}_1$ = “exemplary manner” and $\{p_4;p_5\}\in h{w}_2$ = “effectively”.

3.5. Evaluation with the Model for Aggregating Raw Data for Deriving A General Assessment of the Quality of Individual Training of A Student within the Framework of Aviation Education—$M_{IA}$

To derive a general assessment of the quality of individual training of a student within the framework of aviation education, we normalized and compared the data, Table 6. In addition, in Table 5, the weighting coefficients w, which sets the DM for each model of evaluating the results of training of students, are given.

First, for data comparison, normalized weighting factors are determined, according to Formula (18): $\overline{w_1}$= 0.27; $\overline{w_2}$= 0.24; $\overline{w_3}$= 0.27; $\overline{w_4}$= 0.22.

After that, one quantitative overall assessment of the quality of individual training of a student in the framework of aviation education is calculated, separately for each student using a weighted average convolution (19): $y\left(p_1\right)$=0.972; $y\left(p_2\right)$= 0.877; $y\left(p_3\right)$= 0.74; $y\left(p_4\right)$= 0.69; $y\left(p_5\right)$= 0.815.

To determine the qualitative assessment of the training level of students, it is obtained that: $p_1\in c{a}_1$= “high level of individual training of a student”; $\{p_2;p_5\}\in c{a}_2$= “the level of individual training of a student is above average”; $p_3\in$ $c{a}_3$= “average level of individual training of a student”; and $p_4\in c{a}_4$= “low level of individual training of the student”.

4. Discussion

The results of the paper support the competency-based training and assessment (CBTA) framework and the importance of quality training for pilots’ outcomes, as also presented by Ziakkas et al. [11]. The paper presents an innovative tool to strengthen the resilience and support of the student (pilot) in aviation education, which was considered by Cherngab et al. [13] as an important element in connection with the psychological well-being and work performance of the pilot. The presented results of the paper provide a comprehensive information platform for the integrated assessment of students’ outcomes in key components of the aviation education of future pilots including risk assessment and counseling, in contrast to other studies that examine separate aspects, such as pedagogical approaches to future pilot failures [14], burnout among pilots and related factors with performance during simulator training [15], or pilot load measurement, etc. [18].

In the paper, a hybrid model for evaluating the results of training of students is built to improve the quality of individual results in the didactic system of aviation education. For this purpose, the following was developed: a vague model for evaluating the results of theoretical training of students; an expert model for evaluating the results of student training on flight simulators; an expert model for evaluating the results of practical flight training of students; an expert model for assessing the competence of practical flight training of students using observed behavior; and a model for aggregating initial data to derive a general assessment of the quality of individual results of a student (pilot ) within the framework of aviation education.

The hybrid complex model can adequately evaluate the quality of individual results of a student (pilot) within the framework of aviation education, which consists of a quantitative overall assessment of the competence of a graduate of aviation education and the qualitative level of the training of a student. The basis of the research, to process expert information and fuzzy input data, is the apparatus of fuzzy sets, the methods of system analysis, observed behavior, and intellectual analysis of knowledge, using the membership functions of one and many variables, while quantitative and qualitative input data are considered. All this in a complex allows revealing the ambiguities of incoming expert opinions to increase the degree of validity of decisions regarding the quality of individual results of a student. The value of the model is that it allows for obtaining a comprehensive quantitative assessment, based on which the acceptance of the level of individual training of the student is determined, or it is recommended to undergo corrective training of the corresponding educational component. All this improves the quality and increases the level of management of the educational process in the aviation education system, which entails a reduction in the risks of results.

The evaluation procedure remains classic and well known, with students after successfully mastering educational disciplines receiving appropriate points, and flight training instructors also present both points and their expert conclusions. After that, the data were processed by appropriate fuzzy and expert models, where the parameter setting was performed on real data. The developed approaches make it impossible to have a subjective influence on the evaluation process and the overall result.

The advantages of the hybrid model for evaluating the learning outcomes of students derive from the well-founded advantages of the developed models. The hybrid model is based on various fuzzy and expert models that allow a comprehensive assessment of pilot training results. The model does not depend on the number of groups of criteria and the criteria themselves, which makes it possible to apply it to various higher educational institutions or educational programs of organizations approved for flight training. The model considers the “Importance of Discipline” as a degree of influence on the formation of the competence of a student, considers the instructors regarding the risk of training results on the flight simulator, reveals the ambiguity in the results of practical flight training, corresponds to the industry practices of the ATO policy, recommends corrective training, and derives the level of quality of individual student training. Models reveal the ambiguities of input data, increase the degree of validity of decisions regarding corrective training recommendations, and focus on an unbiased assessment of students, which in the complex increases the controllability of the educational process in the pilot training system and reduces the risks to the results.

A limitation of our study was the use of different types of membership functions and data fuzzification approaches, as well as convolution to obtain aggregated estimates. Another limitation is related to the study and the fitting of the model to the TUKE student (pilot) data. As mentioned in Section 1, each university has an established academic information system to record the results of completed courses and earned credits. Not every university has an academic information system that records the results of flight training, if it is provided by an external organization, to assess the knowledge, skills, and competences of students, including the identification of risks in the education of future pilots for quick response and individual counseling to the student. It is important to reproduce this study on students (pilots) of other educational institutions and the territories of other countries to be able to compare the initial data. Such limitations may lead to minor ambiguities in the results. Instead, the adequacy of the developed hybrid model and the applied mathematical apparatus were proven and confirmed.

The rationality of the obtained overall assessment of the quality of individual training of a student (pilot) within the framework of aviation education proves the advantages of the developed model. The reliability of the obtained results is ensured by the justified use of the apparatus of fuzzy sets, system analysis, and intellectual analysis of knowledge, which is also confirmed by the results of the research.

5. Conclusions

Research has been carried out on the actual task of developing a complex hybrid model for evaluating the results of pilots to improve the quality of individual results in the didactic system of aviation education. At the same time, the following results were obtained:

• A fuzzy model for evaluating the results of theoretical training of students was developed. A set of criteria for theoretical disciplines of aviation education is proposed in the “PILOT” study program, TUKE. The model uses an adequate apparatus of fuzzy sets and allows obtaining a quantitative risk assessment of the results of aviation education, using the concept of “discipline importance” to consider different degrees of influence on the formation of student (pilot) competence. The risk levels of aviation education results are established;

• An expert model for evaluating the results of student training on flight simulators was developed. The peculiarity of this model is that, in addition to the quantitative assessment of the training results, the instructor expresses their considerations regarding the risks of the training results of student pilots on the flight simulator. Based on the intellectual analysis of knowledge and belonging functions, aggregated normalized evaluations of the results of student training on flight simulators are obtained;

• An expert model for the evaluation of the results of the practical flight training of students was developed using the example of obtaining a PPL license. The model is based on the evaluated results of practical flight training of pupils according to the rules PSA/01–PSA/24. The model reveals the vagueness of the input assessments and can derive a summary standardized assessment of the results of the practical flight training of student pilots, which increases the degree of appropriateness when making subsequent management decisions. The model is open to assessments of different types of pilot training, licenses, and qualifications;

• An expert model was developed for assessing the competence of practical flight training of students (pilots) using observed behavior. Indicators for “observed behavior” are used from official IATA documents. Based on intellectual analysis of knowledge, through uncertainty modeling, aggregation of the number and frequency of “observed behavior” demonstrated by the corresponding student was carried out. At the output of the model, we have an aggregated normalized assessment, based on which recommendations for remedial training are made, taking into account the levels of training and standards of practice;

• An input data aggregation model was proposed to derive a general quality assessment of individual training of a student within the framework of aviation education;

• The results of the study were tested and verified on real data of undergraduate students in the “PILOT” study program. At the same time, the adequacy of the complex hybrid model developed in the study, based on fuzzy and expert models, was experimentally confirmed. The results demonstrate the applied value of the assessment.

Further research on the issue can be seen in the subsequent development of an innovative information technology module and software support of the aviation didactic system. The innovative module, hybrid model, and software will serve as a means of supporting decision-making, improving the quality of the educational process, and individual study counseling using risk assessment of results in aviation education.

Praxeological solutions for the integration of the proposed fuzzy model of the evaluation of the results of pilot students within the didactic system of aviation education into educational practice are possible on two levels: as a web analytical tool based on the designed algorithm and as a mobile application that can be used by all internal stakeholders (university teachers and students) and external stakeholders (aeronautical training providers and ATOs) within aviation education.

Author Contributions

Conceptualization, M.K. and V.P.; methodology, M.K. and V.P.; software, V.P.; validation, M.K., V.P. and M.K.J.; formal analysis, M.K. and V.P.; investigation, M.K., V.P. and M.K.J.; resources, M.K.; data curation, V.P., M.K.J. and J.S.; writing—original draft preparation, M.K., V.P. and M.K.J.; writing—review and editing, M.K., V.P., M.K.J. and J.S.; visualization, V.P.; supervision, M.K.; project administration, V.P.; funding acquisition, M.K. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

The data presented in this study are available from the corresponding author upon request. The repository of 696 individual results of undergraduate students (pilots), from the total number of 2682 undergraduate students of the Faculty of Aeronautics of the Technical University of Košice (Slovakia) in the period 2005–2022, was used in the creation of the model.

Acknowledgments

The authors would like to thank the Faculty of Aeronautics of the Technical University of Košice for the anonymous data of selected undergraduate students (pilots) for the verification of the hybrid model algorithm. The authors of the paper thank the Office of the Government of the Slovak Republic for a grant to support researchers of Ukraine, project code: 09I03-03-V01-00059.

Conflicts of Interest

The authors declare no conflict of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figure and Tables

Figure 1. Structural diagram for evaluating the learning outcomes of students (pilots).

Table 1

Input data on the learning outcomes of students (pilots).

Year of Study	Criteria	$p_1$	$p_2$	$p_3$	$p_4$	$p_5$
$C_1$ 2019/2020	$K_{11}$	89/B *	81/B	88/B	96/A *	81/B
	$K_{12}$	88/B	66/D *	100/A	69/D	74/C *
	$K_{13}$	93/A	86/B	90/B	86/B	89/B
	$K_{14}$	96/A	81/B	96/A	85/B	72/C
	$K_{15}$	93/A	77/C	98/A	89/B	68/D
	$K_{16}$	99/A	98/A	99/A	95/A	97/A
	$K_{17}$	94/A	97/A	100/A	83/B	92/A
	$K_{18}$	91/A	88/B	94/A	94/A	86/B
	$K_{19}$	98/A	78/C	100/A	91/A	93/A
	$K_{110}$	credited	credited	credited	credited	credited
	$K_{111}$	credited	credited	credited	credited	credited
	$K_{112}$	91/A	84/B	92/A	61/D	53/E
	$K_{113}$	95/A	86/B	98/A	88/B	67/D
$C_2$ 2020/2021	$K_{21}$	81/B	98/A	100/A	98/A	84/B
	$K_{22}$	100/A	83/B	98/A	99/A	98/A
	$K_{23}$	85/B	79/C	97/A	88/B	76/C
	$K_{24}$	100/A	96/A	100/A	100/A	96/A
	$K_{25}$	91/A	81/B	95/A	85/B	89/B
	$K_{26}$	84/B	85/B	90/B	81/B	72/C
	$K_{27}$	95/A	81/B	98/A	85/B	76/C
	$K_{28}$	92/A	88/B	97/A	87/B	76/C
	$K_{29}$	99/A	98/A	98/A	83/B	78/C
	$K_{210}$	92/A	81/B	91/A	91/A	91/A
	$K_{211}$	99/A	91/A	99/A	91/A	57/E *
$C_3$ 2021/2022	$K_{31}$	95/A	90/B	87/B	64/D	87/B
	$K_{32}$	87/B	76/C	81/B	59/E	54/E
	$K_{33}$	91/A	84/B	86/B	87/B	57/E
	$K_{34}$	89/B	69/D	96/A	79/C	76/C
	$K_{35}$	91/A	93/A	93/A	81/B	93/A
	$K_{36}$	93/A	63/D	95/A	61/D	84/B
	$K_{37}$	91/A	91/A	94/A	75/C	70/D
	$K_{38}$	95/A	51/E	93/A	62/D	83/B
	$K_{39}$	credited	credited	credited	credited	credited
	$K_{310}$	94/A	91/A	100/A	71/C	79/C
	$K_{311}$	100/A	98/A	100/A	90/B	96/A
	$K_{FS}$	95/A	93/A	85/B	92/A	83/B

* “A”—excellent if 91–100%; “B”—very good if 81–90%; “C”—good if 71–80%; “D”—acceptable if 61–70%; “E”—sufficient if 51–60%.

Table 2

Data on the closeness of the study results of the student to the “Importance of Discipline”.

Year of Study	Criteria	$T$	$nim\left(K_j\right)$	$max\left(K_j\right)$	$p_1$	$p_2$	$p_3$	$p_4$	$p_5$
$C_1$ 2019/2020	$K_{11}$	93	75	96	0.78	0.33	0.72	0.83	0.33
	$K_{12}$	85	65	100	0.85	0.05	0.25	0.2	0.45
	$K_{13}$	90	75	95	0.8	0.73	1	0.73	0.93
	$K_{14}$	91	70	96	0.76	0.52	0.76	0.71	0.1
	$K_{15}$	93	61	99	1	0.5	0.84	0.88	0.22
	$K_{16}$	98	85	100	0.92	1	0.92	0.77	0.92
	$K_{17}$	95	80	100	0.93	0.87	0.67	0.2	0.8
	$K_{18}$	91	80	96	1	0.73	0.73	0.73	0.55
	$K_{19}$	95	75	100	0.85	0.15	0.75	0.8	0.9
	$K_{110}$	91	91	100	1	1	1	1	1
	$K_{111}$	91	91	100	1	1	1	1	1
	$K_{112}$	85	51	95	0.82	0.97	0.79	0.29	0.06
	$K_{113}$	91	61	100	0.87	0.83	0.77	0.9	0.2
$C_2$ 2020/2021	$K_{21}$	91	75	100	0.38	0.56	0.44	0.56	0.56
	$K_{22}$	95	80	100	0.67	0.2	0.8	0.73	0.8
	$K_{23}$	95	75	100	0.5	0.2	0.9	0.65	0.05
	$K_{24}$	93	85	100	0.13	0.63	0.13	0.13	0.63
	$K_{25}$	91	75	98	1	0.38	0.75	0.63	0.88
	$K_{26}$	95	65	98	0.63	0.67	0.83	0.53	0.23
	$K_{27}$	93	70	98	0.91	0.48	0.78	0.65	0.26
	$K_{28}$	95	70	98	0.88	0.72	0.92	0.68	0.24
	$K_{29}$	95	70	100	0.84	0.88	0.88	0.52	0.32
	$K_{210}$	91	75	95	0.94	0.38	1	1	1
	$K_{211}$	95	51	100	0.91	0.91	0.91	0.91	0.14
$C_3$ 2021/2022	$K_{31}$	95	61	95	1	0.85	0.76	0.09	0.76
	$K_{32}$	93	51	95	0.86	0.6	0.71	0.19	0.07
	$K_{33}$	95	51	95	0.91	0.75	0.8	0.82	0.14
	$K_{34}$	95	61	98	0.82	0.24	0.97	0.53	0.44
	$K_{35}$	95	80	95	0.73	0.87	0.87	0.07	0.87
	$K_{36}$	93	61	95	1	0.06	0.94	0	0.72
	$K_{37}$	93	61	95	0.94	0.94	0.97	0.44	0.28
	$K_{38}$	93	51	98	0.95	0	1	0.26	0.76
	$K_{39}$	91	91	100	1	1	1	1	1
	$K_{310}$	91	61	100	0.9	1	0.7	0.33	0.6
	$K_{311}$	95	81	100	0.64	0.79	0.64	0.64	0.93

Table 3

Fuzzification of input hybrid data.

Year of Study	Criteria	$w$	$\overline{w}$	$p_1$	$p_2$	$p_3$	$p_4$	$p_5$
$C_1$ 2019/2020	$K_{11}$	8	0.0261	0.87	0.85	0.87	0.97	0.85
	$K_{12}$	8	0.0261	0.88	0.62	0.94	0.64	0.75
	$K_{13}$	10	0.0327	0.97	0.87	0.9	0.87	0.88
	$K_{14}$	9	0.0294	0.97	0.86	0.97	0.87	0.73
	$K_{15}$	9	0.0294	1	0.76	0.97	0.88	0.64
	$K_{16}$	9	0.0294	0.98	1	0.98	0.97	0.98
	$K_{17}$	9	0.0294	0.98	0.98	0.96	0.84	0.97
	$K_{18}$	9	0.0294	1	0.87	0.97	0.97	0.86
	$K_{19}$	8	0.0261	0.98	0.73	0.97	0.97	0.98
	$K_{110}$	7	0.0229	1	1	1	1	1
	$K_{111}$	7	0.0229	1	1	1	1	1
	$K_{112}$	7	0.0229	0.97	0.89	0.97	0.64	0.53
	$K_{113}$	9	0.0294	0.98	0.87	0.97	0.88	0.64
$C_2$ 2020/2021	$K_{21}$	7	0.0229	0.85	0.96	0.95	0.96	0.86
	$K_{22}$	9	0.0294	0.96	0.84	0.97	0.97	0.97
	$K_{23}$	9	0.0294	0.86	0.74	0.98	0.86	0.72
	$K_{24}$	9	0.0294	0.93	0.96	0.93	0.93	0.96
	$K_{25}$	9	0.0294	1	0.85	0.97	0.86	0.88
	$K_{26}$	9	0.0294	0.86	0.86	0.87	0.86	0.74
	$K_{27}$	8	0.0261	0.98	0.85	0.97	0.86	0.74
	$K_{28}$	10	0.0327	0.98	0.87	0.98	0.86	0.74
	$K_{29}$	10	0.0327	0.97	0.98	0.98	0.86	0.75
	$K_{210}$	8	0.0261	0.98	0.85	1	1	1.00
	$K_{211}$	9	0.0294	0.98	0.98	0.98	0.98	0.53
$C_3$ 2021/2022	$K_{31}$	10	0.0327	1	0.88	0.87	0.63	0.87
	$K_{32}$	9	0.0294	0.88	0.76	0.87	0.54	0.53
	$K_{33}$	9	0.0294	0.98	0.87	0.87	0.87	0.53
	$K_{34}$	10	0.0327	0.87	0.64	0.99	0.76	0.75
	$K_{35}$	10	0.0327	0.97	0.98	0.98	0.83	0.98
	$K_{36}$	8	0.0261	1	0.63	0.98	0.61	0.87
	$K_{37}$	10	0.0327	0.98	0.98	0.99	0.75	0.64
	$K_{38}$	8	0.0261	0.99	0.51	1	0.64	0.87
	$K_{39}$	8	0.0261	1	1	1	1	1
	$K_{310}$	9	0.0294	0.98	1	0.97	0.75	0.76
	$K_{311}$	9	0.0294	0.96	0.97	0.96	0.86	0.98

Table 4

Input data from flight training instructors.

Exercises	$p_1$		$p_2$		$p_3$		$p_4$		$p_5$
	DUAL	SOLO	DUAL	SOLO	DUAL	SOLO	DUAL	SOLO	DUAL	SOLO
PSA/01 to PSA/17	2	1	3	2	3	2	3	3	3	2
PSA/18 to PSA/24	1	1	1	2	3	4	3	4	1	2

Table 5

Aggregation inputs and results for “observed behavior”.

Group of Criteria	Criteria	$p_1$		H1	$p_2$		H2	$p_3$		$H_3$	$p_4$		H4	$p_5$		H5
		$L_q$	$L_{\phi }$		$L_q$	$L_{\phi }$		$L_q$	$L_{\phi }$		$L_q$	$L_{\phi }$		$L_q$	$L_{\phi }$
$G_1$	$O{B}_{11}$	1	1	1	0.8	0.8	0.859	1	1	1	0.8	0.6	0.776	0.8	0.6	0.776
	$O{B}_{12}$	0.8	0.8	0.859	0.8	0.6	0.776	0.8	1	0.9	0.8	0.6	0.776	0.8	0.6	0.776
	$O{B}_{13}$	0.6	0.8	0.776	0.6	0.6	0.717	1	0.8	0.9	0.6	0.6	0.717	0.8	0.6	0.776
	$O{B}_{14}$	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.8	0.6	0.776
	$O{B}_{15}$	1	1	1	1	0.8	0.9	1	1	1	1	0.6	0.8	0.8	0.6	0.776
	$O{B}_{16}$	1	1	1	1	0.8	0.9	1	1	1	1	0.8	0.9	0.8	0.6	0.776
	$O{B}_{17}$	1	1	1	1	0.8	0.9	1	1	1	1	0.8	0.9	0.8	0.6	0.776
$G_2$	$O{B}_{21}$	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.8	0.8	0.859
	$O{B}_{22}$	0.8	0.8	0.859	0.8	0.8	0.859	1	0.8	0.9	0.6	0.8	0.776	0.8	0.8	0.859
	$O{B}_{23}$	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.8	0.8	0.859
	$O{B}_{24}$	0.6	1	0.8	0.6	0.8	0.776	0.8	1	0.9	0.6	0.8	0.776	0.8	0.8	0.859
	$O{B}_{25}$	1	0.8	0.9	1	0.8	0.9	1	0.8	0.9	1	0.8	0.9	0.8	0.8	0.859
	$O{B}_{26}$	1	1	1	0.8	1	0.9	1	1	1	0.6	0.8	0.776	0.6	0.8	0.776
	$O{B}_{27}$	0.8	0.8	0.859	0.8	0.8	0.859	1	0.8	0.9	0.8	0.8	0.859	0.8	0.8	0.859
$G_3$	$O{B}_{31}$	0.8	1	0.9	0.6	0.8	0.776	0.8	1	0.9	0.6	0.8	0.776	0.6	0.8	0.776
	$O{B}_{32}$	0.8	0.8	0.859	0.8	0.8	0.859	1	1	1	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{33}$	1	1	1	1	0.6	0.8	1	1	1	0.6	0.6	0.717	0.6	0.6	0.717
	$O{B}_{34}$	1	1	1	0.8	0.8	0.859	1	1	1	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{35}$	1	1	1	1	1	1	1	1	1	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{36}$	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{37}$	1	1	1	1	0.8	0.9	1	1	1	1	0.8	0.9	0.8	0.8	0.859
	$O{B}_{38}$	1	1	1	0.8	1	0.9	1	1	1	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{39}$	1	1	1	1	1	1	1	1	1	1	0.8	0.9	0.8	0.8	0.859
	$O{B}_{310}$	0.8	1	0.9	0.6	0.8	0.776	0.8	1	0.9	0.6	0.6	0.717	0.6	0.6	0.717
$G_4$	$O{B}_{41}$	1	1	1	1	1	1	1	1	1	0.6	0.8	0.776	0.6	0.8	0.776
	$O{B}_{42}$	1	1	1	1	1	1	1	1	1	0.6	0.8	0.776	0.6	0.8	0.776
	$O{B}_{43}$	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859	0.6	0.8	0.776	0.6	0.8	0.776
	$O{B}_{44}$	0.8	1	0.9	0.8	0.8	0.859	1	1	1	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{45}$	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{46}$	0.8	1	0.9	0.8	1	0.9	1	1	1	0.8	1	0.9	0.8	0.8	0.859
$G_5$	$O{B}_{51}$	0.6	0.8	0.776	0.6	0.8	0.776	0.8	0.8	0.859	0.6	0.8	0.776	0.6	0.8	0.776
	$O{B}_{52}$	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859	0.6	0.8	0.776	0.6	0.8	0.776
	$O{B}_{53}$	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.6	0.8	0.776	0.6	0.8	0.776
	$O{B}_{54}$	0.8	1	0.9	0.6	0.8	0.776	1	1	1	0.6	0.8	0.776	0.6	0.8	0.776
	$O{B}_{55}$	0.8	1	0.9	0.8	1	0.9	1	1	1	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{56}$	0.8	1	0.9	0.8	0.8	0.859	1	1	1	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{57}$	1	1	1	1	1	1	1	1	1	1	1	1	0.8	0.8	0.859
$G_6$	$O{B}_{61}$	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{62}$	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{63}$	1	1	1	1	1	1	1	1	1	1	1	1	0.8	0.8	0.859
	$O{B}_{64}$	1	1	1	0.8	0.8	0.859	1	1	1	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{65}$	1	0.8	0.9	0.8	0.8	0.859	1	0.8	0.9	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{66}$	1	1	1	0.8	1	0.9	1	1	1	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{67}$	1	1	1	1	1	1	1	1	1	1	0.8	0.9	0.8	0.8	0.859
	$O{B}_{68}$	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.8	0.8	0.859
	$O{B}_{69}$	0.8	0.8	0.859	0.8	0.8	0.859	1	0.8	0.9	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{610}$	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.8	0.8	0.859
	$O{B}_{611}$	0.8	0.8	0.859	0.8	0.8	0.859	1	0.8	0.9	0.8	0.8	0.859	0.8	0.8	0.859
$G_7$	$O{B}_{71}$	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{72}$	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.8	0.8	0.859
	$O{B}_{73}$	1	1	1	1	0.8	0.9	1	1	1	1	0.8	0.9	0.8	0.8	0.859
	$O{B}_{74}$	1	1	1	0.8	0.8	0.859	1	1	1	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{75}$	1	1	1	0.8	1	0.9	1	1	1	0.8	1	0.9	0.8	0.8	0.859
	$O{B}_{76}$	1	0.8	0.9	1	0.8	0.9	1	0.8	0.9	1	0.8	0.9	1	0.8	0.9
	$O{B}_{77}$	1	1	1	0.8	1	0.9	1	1	1	0.8	1	0.9	0.8	0.8	0.859
	$O{B}_{78}$	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{79}$	0.8	1	0.9	0.8	1	0.9	1	1	1	0.8	1	0.9	0.8	0.8	0.859
$G_8$	$O{B}_{81}$	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.8	0.8	0.859
	$O{B}_{82}$	0.8	1	0.9	0.8	0.8	0.859	0.8	1	0.9	0.6	0.8	0.776	0.6	0.8	0.776
	$O{B}_{83}$	0.8	1	0.9	0.8	1	0.9	1	1	1	0.6	0.8	0.776	0.6	0.8	0.776
	$O{B}_{84}$	0.8	0.8	0.859	0.8	0.8	0.859	1	0.8	0.9	0.6	0.8	0.776	0.6	0.8	0.776
	$O{B}_{85}$	1	1	1	1	1	1	1	1	1	1	1	1	0.6	0.8	0.776
	$O{B}_{86}$	1	1	1	0.8	0.8	0.859	1	1	1	0.8	0.8	0.859	0.6	0.8	0.776
	$O{B}_{87}$	1	1	1	1	1	1	1	1	1	1	1	1	0.6	0.8	0.776
$G_9$	$O{B}_{91}$	1	1	1	0.8	0.8	0.859	1	1	1	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{92}$	1	0.8	0.9	1	0.8	0.9	1	0.8	0.9	0.6	0.8	0.776	0.6	0.8	0.776
	$O{B}_{93}$	1	1	1	1	1	1	1	1	1	0.6	0.8	0.776	0.6	0.8	0.776
	$O{B}_{94}$	0.6	0.8	0.776	0.6	0.6	0.717	0.8	0.8	0.859	0.6	0.8	0.776	0.6	0.8	0.776
	$O{B}_{95}$	0.8	1	0.9	0.8	1	0.9	0.8	1	0.9	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{96}$	0.8	0.8	0.859	0.8	0.8	0.859	1	0.8	0.9	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{97}$	0.8	0.8	0.859	0.8	0.8	0.859	1	0.8	0.9	0.8	0.8	0.859	0.8	0.8	0.859
	$O{B}_{98}$	0.8	1	0.9	0.8	0.8	0.859	0.8	1	0.9	0.8	0.6	0.776	0.8	0.6	0.776
	$O{B}_{99}$	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.8	0.859	0.8	0.6	0.776	0.8	0.6	0.776

Table 6

Data of output estimates obtained based on models $M_{TS},M_{FS}, M_{PFT},\mathrm{and} M_{OB}$.

Assessment Model	w	$p_1$	$p_2$	$p_3$	$p_4$	$p_5$
$M_{TS}$	10	0.9587	0.8647	0.9572	0.8471	0.8109
$M_{FS}$	9	1	0.996	0.897	0.998	0.851
$M_{PFT}$	10	1	0.78	0.22	0.13	0.78
$M_{OB}$	8	0.92	0.881	0.942	0.847	0.825

References

1. Sustainable Development Goals. United Nations. Department of Economic and Social Affairs. Available online: https://sdgs.un.org/goals (accessed on 29 January 2023).

2. A Return to Industry Profitability in 2023. IATA Economics’ Chart of the Week. Available online: https://www.iata.org/en/iata-repository/publications/economic-reports/a-return-to-industry-profitability-in-2023/ (accessed on 29 January 2023).

3. Aviation Intelligence Unit Performance 2022—Outlook 2023. EUROCONTROL. Available online: https://www.eurocontrol.int/sites/default/files/2022-12/eurocontrol-analysis-paper-2022-review-2023-outlook.pdf (accessed on 29 January 2023).

4. Global Outlook for Air Transport: Sustained Recovery Amidst Strong Headwinds. IATA. Available online: https://www.iata.org/en/iata-repository/publications/economic-reports/global-outlook-for-air-transport---december-2022/ (accessed on 29 January 2023).

5. Provisional agenda for the 41st Sesion of the ICAO Assembly: Item 26: ICAO Civil Aviation Training and Capacity Building. 2022; Available online: https://www.icao.int/Meetings/a41/Documents/WP/wp_001_en.pdf (accessed on 29 January 2023).

6. Addressing the Shortage of Pilots and Aviation Professionals—A Global Approach to a Global Poblem. Agenda Item 25: ICAO Civil Aviation Training and Capacity Building. 2021; Available online: https://www.icao.int/Meetings/a40/Documents/WP/wp_239_en.pdf (accessed on 29 January 2023).

7. The International Association of Aviation and Aerospace Education. 2018; Available online: https://alicanto.org/ (accessed on 29 January 2023).

8. Partnership of a European Group of Aeronautics and Space Universities (PEGASUS). Available online: https://www.pegasus-europe.org/ (accessed on 29 January 2023).

9. ICAO Annex 1 to the Convention on International Civil Aviation—Personnel Licensing. Available online: https://store.icao.int/en/annex-1-personnel-licensing (accessed on 29 January 2023).

10. Convention on International Civil Aviation ICAO—Doc 7300/9. Available online: https://www.icao.int/publications/Documents/7300_9ed.pdf (accessed on 29 January 2023).

11. Ziakkas, D.; Suckowb, M.W.; Pechlivanisc, K. The Implementation of Competency-Based Training and Assessment (CBTA) Framework in Aviation Manpower Planning. Transp. Res. Procedia; 2022; 66, pp. 226-239. [DOI: https://dx.doi.org/10.1016/j.trpro.2022.12.023]

12. Groppe, M.; Brock, I. Cross-cultural Interactions on the Flight Deck: Applying a Socio-Ecological Model to Acculturative Stress. Transp. Res. Procedia; 2022; 66, pp. 240-252. [DOI: https://dx.doi.org/10.1016/j.trpro.2022.12.024]

13. Cherngab, C.-F.G.; Sherc, J.-S.; Chuc, H.; Yu, L. The Relationship Between Civil Pilots’ Resilience, Psychological Well-being, and Work Performance. Transp. Res. Procedia; 2022; 66, pp. 16-25. [DOI: https://dx.doi.org/10.1016/j.trpro.2022.12.003]

14. Kılıç, B.; Soran, S. How Can an Ab-Initio Pilot Avert a Future Disaster: A Pedagogical Approach to Reduce The Likelihood of Future Failure. J. Aviat.; 2019; 3, pp. 1-14. [DOI: https://dx.doi.org/10.30518/jav.508336]

15. Demerouti, E.; Veldhuis, W.; Coombes, C.; Hunter, R. Burnout among pilots: Psychosocial factors related to happiness and performance at simulator training. Ergonomics; 2019; 62, pp. 233-245. [DOI: https://dx.doi.org/10.1080/00140139.2018.1464667] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29648499]

16. Doroftei, D.; De Cubber, G.; De Smet, H. Reducing drone incidents by incorporating human factors in the drone and drone pilot accreditation process. AHFE 2020. Advances in Intelligent Systems and Computing; Zallio, M. Springer: Cham, Switzerland, 2021; Volume 1210, pp. 71-77. [DOI: https://dx.doi.org/10.1007/978-3-030-51758-8_10]

17. Novák, A.; Novák Sedláčková, A.; Kazda, A. Implementácia projektu KAAT—Znalostnej aliancie v leteckej deprave—Príbeh úspešného projektu (Implementation of the KAAT project—Knowledge alliance in air transport—The story of a successful project. Perner’s Contacts; 2020; 15, pp. 1-7. [DOI: https://dx.doi.org/10.46585/pc.2020.1.1553]

18. Škvareková, I.; Ažaltovič, V.; Pecho, P.; Kandera, B. The measurement of pilot workload. Proceedings of the Zvyšovanie Bezpečnosti a Kvality v Civilnom Letectve, Medzinárodná Vedecká Konferencia Organizovaná v Rámci Riešenia Projektu VEGA 1/0624/18—Modely Podnikania Regionálnych Letísk v Kontexte Dopravnej Politiky Štátu a Európskej Únie; Žilina, Slovakia, 22–24 January 2020; [DOI: https://dx.doi.org/10.26552/zbk.Z.2020.1.9]

19. Kandera, B.; Chodelka, F.; Škvareková, I. Assessment of menthal health of pilots. Proceedings of the Zvyšovanie Bezpečnosti a Kvality v Civilnom Letectve, Medzinárodná Vedecká Konferencia Organizovaná v Rámci Riešenia Projektu VEGA 1/0624/18—Modely Podnikania Regionálnych Letísk v Kontexte Dopravnej Politiky Štátu a Európskej Únie; Žilina, Slovakia, 22–24 January 2020; [DOI: https://dx.doi.org/10.26552/zbk.Z.2020.1.3]

20. Zgodavova, Z.; Rozenberg, R.; Szabo, S.; Sabo, J. The impact of changes of psychophysiological factors on the flight crew performance. Int. Rev. Aerosp. Eng.; 2019; 12, pp. 150-158. [DOI: https://dx.doi.org/10.15866/irease.v12i3.16480]

21. Korniienko, A.; Daňko, P. Preliminary research of air rescue services in the world and improvement for the Slovak republic. Acta Avion. J. Sci.; 2020; 22, pp. 6-13. [DOI: https://dx.doi.org/10.35116/aa.2020.0010]

22. Tirpáková, M.; Šváb, P.; Korba, P.; Hovanec, M. The impact of the covid-19 outbreak on aviation. Sci. Pap. Sil. Univ. Technol. Organ. Manag. Ser.; 2020; 148, pp. 253-262. [DOI: https://dx.doi.org/10.29119/1641-3466.2020.148.19]

23. Socha, V.; Socha, L.; Matyáš, R.; Hanáková, L.; Van den Bergh, S.; Šerlová, M.; Lališ, A. Pilots’ Performance Assessment: Evidence Based Approach. Proceedings of the 2019 International Conference on Military Technologies (ICMT); Brno, Czech Republic, 30–31 May 2019; [DOI: https://dx.doi.org/10.1109/MILTECHS.2019.8870130]

24. Weber, D. Judging Airline Pilots’ Performance with and without an Assessment Model: A Comparison Study of the Scoring of Raters from Two Different Airlines. J. Aviat. /Aerosp. Educ. Res.; 2016; 25, pp. 39-70. [DOI: https://dx.doi.org/10.15394/jaaer.2016.1645]

25. Li, W.-C.; Chiu, F.-C.; Wu, K.-J. The evaluation of pilot’s performance and mental workload by eye movement. In Proceeding of the 30th European Association for Aviation Psychology Conference; Sardinia, Italy, 24–28 September 2012; Available online: http://dspace.lib.cranfield.ac.uk/handle/1826/8338 (accessed on 29 January 2023).

26. Competency Assessment and Evaluation for Pilots, Instructors and Evaluators. Guidance Material. 2021; Available online: https://www.iata.org/contentassets/c0f61fc821dc4f62bb6441d7abedb076/competency-assessment-and-evaluation-for-pilots-instructors-and-evaluators-gm.pdf (accessed on 29 January 2023).