An Intelligent Chinese Driver Road Performance Assessment Model (RPAM) for Future Licensing Examinations

Abstract

As the demand for private vehicles rises, there has been a gradual increase in the number of motor vehicles on the roads, leading to a growing concern about addressing traffic safety. Currently, China’s approach to assessing driver capabilities remains rooted in traditional, non-intelligent, and standardized evaluation methods based on examination subjects. The traditional model often falls short in providing constructive feedback on a driver’s real-world vehicle handling abilities, as many of the examination subjects can be practiced in advance to achieve a mere passing result, which, undoubtedly, increases the likelihood of underqualified drivers on the road. To address the issues of the current examination-oriented driver evaluation system in China, we propose a road performance assessment model (RPAM) that assesses drivers comprehensively by evaluating their road environment perception and vehicle operation abilities based on an in-vehicle and out-vehicle perception system. The model leverages patterns of the driver’s head posture, along with real-time information on the vehicle’s behavior and the road conditions, to quantify various performance metrics related to reasonable operation processes. These metrics are then integrated to generate a holistic assessment of the driving capabilities. This paper ultimately conducted tests of the RPAM on one actual examination route in Beijing. Two drivers were randomly selected for the examination. The model successfully computed the overall ability scores for each driver, validating the effectiveness.

1. Introduction

With the rapid progress of electrification and automation in the automotive industry, vehicles have experienced significant advancements in their automation levels and ease of operation [1]. This has made their operation and control more accessible for human drivers. However, the number of traffic accidents caused by driver violations or improper vehicle handling on the roads has not shown a significant decline [2]. While advanced driver assistance systems (ADASs) and other autonomous technologies can mitigate the risk of collisions to some extent, they cannot entirely compensate for the deficiencies in environmental perception and behavioral execution exhibited by some drivers [3,4].

To enhance overall road safety in urban environments, it is essential to address the issue of driver capabilities [5]. This involves improving the standards for driver competency testing and capability assessment, selecting individuals who are genuinely prepared for safe road operation, and eliminating those who pass tests primarily due to exam-oriented preparation.

China, as a nation with a substantial number of driver examinations each year, currently relies on a standardized and formulaic evaluation system to expedite driver competency testing. The existing Subject 3 examination employs a machine that uses a simple base station GPS for vehicle positioning, roadside RSUs (roadside units) for real-time transmission of traffic signals and road signs status, and pre-established points on a fixed course to evaluate candidates. While this system can identify some driver errors resulting from improper operations, such as driving over lane boundaries or running red lights, it still requires the presence of a safety judge in the front passenger seat of the vehicle to assess potential conflicts with the environment. The safety judge checks adherence to qualified driver norms and driver distractions. Sometimes, their evaluation is subjective and may be subject to external influences, like personal interactions and dealings, rendering the Subject 3 examination less intelligent and standardized.

Furthermore, the current Subject 3 examination consists of individual sub-tests with specific objectives, and candidates often perform predefined actions on familiar road segments. Some relatively straightforward skills, such as high-speed vehicle control, are assessed in overtaking exercises, while adjustments to vehicle angles during movement are evaluated in turning exercises. This compartmentalized approach not only oversimplifies the comprehensive evaluation of a driver’s abilities but also lacks rationality and transparency.

In response to the existing exam-oriented and non-theoretically validated Subject 3 examination, this paper proposes a de-subproject, scenario-based, and de-processed driver road performance assessment approach with an in-vehicle and external vehicle integrated perception system. We conducted an examination of road testing methods and characteristics in other international regions. The prevailing driver competency assessment systems [6,7] predominantly employ statistical approaches for testing, posing significant challenges in practical exam application. This limitation renders them ineffective in the context of the extensive scale of driving examination demands, such as those encountered in China. Our approach is characterized by its intelligent assessment ability and considers a driver’s environmental perception and behavioral capabilities in both routine and high-risk emergency scenarios, aided by high-precision maps and information from inside vehicles. Its aim is to replace the current China Subject 3 evaluation process to enhance the assessment of drivers’ real capabilities for road operation.

This article is organized as follows: We comprehensively outline the derivation process and application methods of the proposed intelligent driver RPAM in the second section. In the third section, we detail the hardware requirements and accompanying software solutions necessary for deploying the RPAM in real-world vehicle setups. In Section 4, we present the vehicles utilized in our testing experiments and highlight the performance of the assessment model. Finally, in the last section, we summarize the achievements of this study and provide a glimpse into the future prospects of intelligent driver assessment models.

2. Driver Road Performance Assessment Model

The road performance assessment model incorporates a holistic evaluation of multiple scenarios along with real-time confidence value estimations. The overall framework is illustrated in Figure 1.

The RPAM conducts separate assessments of the driver’s perception and control capacities for each scenario. It computes the overall performance score by tallying the total number of scenarios successfully navigated in each testing round. Concurrently, it utilizes a confidence value model to dynamically gauge the driver’s operational status. In instances where the confidence value falls below a predefined threshold, the test is immediately deemed unsuccessful. The subsequent sections in this chapter will expound in detail upon the evaluation methodology employed by the RPAM.

2.1. Driver Perception Capability Assessment

The first stage of driver assessment quantitatively assesses the driver’s environmental perception capability. This assessment is based on the area covered by the driver’s field of view, which includes both the directly scanned area and the indirectly reflected area in the rear-view mirrors. The evaluation of a driver’s environmental perception capability involves calculating the area within the driver’s line of sight where their attention is focused.

In accordance with pertinent research, we defined specific parameters for this assessment. We assumed that the driver’s field of vision encompasses an angle of 60 degrees, with their attention span extending over a distance of 50 m (through consultations with internal experts from the Research Institute for Road Safety of MPS China). To streamline the computation of the field of vision in our practical vehicle application, we simplified the observed circular arc by dividing it into three triangular regions, as illustrated in Figure 2. This simplification was employed to reduce computational complexity, especially at the boundaries of the arc in radians.

Figure 2a represents the driver’s observation area, which takes the form of a circular arc. Figure 2b shows a simplified representation of the observation area. The gap between the first point and second point is 20 degrees, and between the central and second points it is 10 degrees. When a vehicle traverses an intersection, the driver’s head undergoes rotational movements due to the act of observation, causing changes in the position and orientation of the simplified observation area.

To calculate the area observed by the driver when entering the intersection scene, encompassing the stages prior to entering and being within the intersection, we devised a method to compute the intersection and union of polygons. Given that the aforementioned polygons are non-concave, we applied the Sutherland–Hodgman clipping algorithm to compute the intersection [8]. The union of the polygons was then determined by merging the results obtained from two iterations of the Sutherland–Hodgman algorithm, where the source and clipping polygons are exchanged [9].

Additionally, we determined the environmental perception capability score based on the observed area in relation to the total scene area. The area of an observed and overall polygon were then calculated using Equation (1), where (x, y) are the coordinates of the points on the outline of the region.

(1)$Siz{e}_{polygon}=\frac{1}{2}|\sum _{i=1}^{i=n}(x_i{y}_{i+1}-x_{i+1}{y}_i)|$

The driver score for the ability to observe can be calculated using Algorithm 1. n in the equation represents the number n vertex of the polygon.

Algorithm 1 Calculation of the observed assessment score

if vehicle enters ’test start mark’ then      while not vehicle enters ’intersection mark’ do          $HeadOrientation\leftarrow (HeadRotationAngle,VehicleHeading)$          $Polygo{n}_{P_1}\leftarrow HeadOrientation$          $Polygo{n}_{view}\leftarrow intersect(Polygo{n}_{P_1},Polygo{n}_{road})$          if first frame then               continue          else               $Polygo{n}_{view}\leftarrow union(Polygo{n}_{view},Polygo{n}_{last})$          end if          $Polygo{n}_{view}=Polygo{n}_{last}$       end while       $Siz{e}_{view}\leftarrow Polygo{n}_{view}$       $Siz{e}_{road}\leftarrow Polygo{n}_{road}$       $ratio=(Siz{e}_{view}/Siz{e}_{road})$       if $ratio<=0.5$ then          $Score=0$      else          $Score=100\ast (ratio-0.5)/0.5$      end if end if

We posit that drivers should complete their environmental observation before their vehicle formally enters an intersection; hence, the assessment of environmental perception capabilities will commence at the moment of entry into the testing scenario and conclude upon entry into the intersection. In the case of lane-changing scenarios, an observation is considered to be completed when the driver’s field of vision scans the area where the rear-view mirror is located. A left lane change involves scanning the left rear-view mirror area, while a right lane change pertains to scanning the right rear-view mirror area. Within lane-changing scenarios, a score of 100 is assigned if the driver observes the rear-view mirror area, and a score of 0 is given if no observation is recorded.

The system will maintain a record of the entire observation area polygon acquired during the observation process. Simultaneously, it will record the traffic participants observed by the driver during this process, encompassing pedestrians and vehicles, based on the outcomes of laser radar target identification. The observation analysis capability score, denoted as $Scor{e}_2$, can be determined based on the number of observed and unobserved traffic participants according to Equation (2).

(2)$Scor{e}_2=100\times \frac{{\left(nu{m}_{spotted}\right)}^{0.5}}{{(nu{m}_{spotted}+nu{m}_{unspotted})}^{0.5}}$

The inclusion of $Scor{e}_2$ is motivated by the fact that when assessing the overall size of the area observed by a driver, consideration is limited to the scanning area over a particular timeframe without taking into account the presence of newly introduced traffic participants. This oversight can lead to situations in which drivers have scanned the entire area, yet have failed to notice new developments within areas they had previously observed. The purpose of $Scor{e}_2$ is to assess a driver’s dynamic environmental perception capabilities. The relationship between $Score$, $Scor{e}_2$, and the driver’s ability is described in Figure 3.

Finally, the overall score for a driver’s environmental perception capability can be expressed as $EnvS$ in Equation (3).

(3)$EnvS=k\ast Score+(1-k)\ast Scor{e}_2$

In recognition of the influence of the number of traffic participants on a driver’s willingness to observe, k is assigned a value lower than 0.5. In our experimental setup, k was set to $0.4$ and $1-k$ was $0.6$.

2.2. Driver Control Capability Assessment

We quantify a driver’s ability to control a vehicle into two distinct components: (1) execution capability and (2) execution efficiency. Execution capability assesses whether a driver possesses the aptitude to sensibly control the vehicle within a given scenario. It primarily focuses on the precision of trajectory control, ensuring the optimal alignment of the vehicle’s path with the most favorable trajectory. Execution efficiency, on the other hand, evaluates a driver’s capacity to effectively maneuver the vehicle through scenarios, determining whether the driver expends excessive, unnecessary time on straightforward operations. It primarily concentrates on the efficiency of vehicle control, ensuring the swift passage of the vehicle through a scenario without disrupting other traffic participants.

We denote the execution capability score as $Es$, which is established by creating atomic behavior and behavior outcome variables and designing a library of standard execution procedures and execution outcomes. Specifically, for the process of driving a vehicle from point A to point B, this paper derives a Bezier curve as the standard path [10,11]. This was accomplished by calculating the trajectory curve using a triple Bezier method based on the vehicle’s initial heading angle at point A and the expected heading angle at target point B. The $Es$ is obtained through the calculation of the difference in distance between the actions taken by the driver and the standard actions. In essence, it quantifies a driver’s actual trajectory from point A to point B in comparison to the Bezier curve, providing an assessment of their control capability.

(4)$D_H(A,B)=max({sup}_{a\in A}{inf}_{b\in B}d(a,b),{sup}_{b\in B}{inf}_{a\in A}d(b,a))$

We employ two different trajectory assessment algorithms for a comprehensive evaluation, namely, computing the Hausdorff distance and calculating the root mean square error (RMSE) between the driver’s route and standard trajectory. Specifically, the Hausdorff distance is computed as the maximum of the minimum distances between each point in one trajectory to every point in the other trajectory [12], as detailed in Equation (4). A and B here represent each set of points in different trajectories, $sup$ signifies the operation of selecting the maximum value from distances between all points, $inf$ represents the operation of calculating distances from all possible points, and $d(a,b)$ denotes the distance metric from point “a” to point “b”, which can be the Euclidean distance. A larger Hausdorff distance implies a greater deviation area between the two trajectories, indicating a poorer fit between them.

(5)$RMSE(A,B)=\sqrt{\frac{1}{n}\sum ({(x{1}_i-x{2}_i)}^2+{(y{1}_i-y{2}_i)}^2)}$

The RMSE, on the other hand, assesses the average difference in distance between each point in one trajectory and the nearest point in the other, as computed using Equation (5). (x1, y1) belong to trajectory one and (x2, y2) are trajectory two. A larger RMSE signifies a greater average deviation between the two curves, suggesting a weaker fit and worse driving behavior [13,14].

In addressing the issue of a driver’s efficiency in executing vehicle maneuvers within a given scenario, we aim to ensure that the total time taken by the driver, from entering a scenario until completely exiting it, closely aligns with the time standards provided by a reference times library. The standard time library is derived through an analysis of historical road traffic flow. It involves collecting and averaging the time taken for all vehicles to execute right, straight, or left turns through a fixed intersection during predefined time intervals. This library is periodically updated to ensure its accuracy.

The driver’s execution efficiency is evaluated by referring to this library for the specific scenario and time frame. The closer the driver’s performance time is to the standard value, the higher their efficiency rating (while a time below the standard value indicates sufficient proficiency). The specific evaluation method is presented in Equation (6).

(6)$Ef=\left\{\begin{array}{cc}100,\hfill & \mathrm{if}t<S_t\left(scenario\right)\hfill \\ 100e^{-\frac{(t-S_t\left(scenario\right))}{1.2\ast S_t\left(scenario\right)}},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$

To comprehensively assess a driver’s vehicle control capabilities by quantifying both execution capability and execution efficiency, we further calculate a composite score based on the obtained trajectory deviation metrics and efficiency scores according to Equation (7).

(7)$OpeS=\frac{Ef}{e^{\frac{D_H(A,B)+RMSE(A,B)}{2\alpha }}}$

In the equation, the value of $\alpha$ is determined by different scenarios. In scenarios where the driver needs to travel longer distances, the $\alpha$ value should be larger. This is used to set the standard for trajectory deviation.

2.3. Driver Overall Performance Assessment

In RPAM, the maximum scores for a driver’s assessment of environmental perception and vehicle operational abilities within each scenario are both set as 100. During a single round of tests, a driver may navigate through multiple scenarios, and RPAM will evaluate the performance in each scenario. Once the tests are completed, the model assesses the driver’s overall cumulative abilities based on the total number of scenarios that the driver has traversed. With an increase in the number of test scenarios, the evaluation of a driver’s abilities approaches their true level of competence. The score for a test round of a driver’s capabilities can be calculated using Equation (8).

(8)$TotalS=\sum \frac{EnvS+OpeS}{2n}$

n in the equation represents the number of scenarios that the test driver passed within this round of tests.

The evaluation of a driver’s driving capabilities within the test scenarios is primarily focused on the driver’s observation and the reasonableness of the vehicle’s passage through the scenarios. It does not encompass judgments regarding the safety of the external environment. During these assessments, the test subjects may engage in interactions with other vehicles that pose significant risks, or they may perform hazardous maneuvers in complex environments. Additionally, they might violate traffic regulations. Some of these risky actions may take place outside the test scenarios. Evaluating these risky behaviors and actions is crucial for determining whether a driver can independently operate a vehicle on the road.

Therefore, RPAM also incorporates the ability to assess driving risk. To achieve this, we introduce the real-time confidence value proposed by our previous work [15], which allows for real-time assessment of the vehicle’s operating state. The final equation for this aspect is presented as Equation (9).

(9)${Conf}_{t+1}=\left\{\begin{array}{c}{Conf}_t-{Of}_t-{Sf}_t+{Re}_t\hfill \\ Mf,\mathrm{if}{Conf}_{t+1}>Mf\hfill \\ 0,\mathrm{if}{Conf}_{t+1}<0\hfill \end{array}\right.$

The confidence value model is a comprehensive assessment model that provides real-time numerical values from 0 to 100, indicating the stability and safety status of a vehicle. It derives the real-time safety rating for the test vehicle based on factors such as the vehicle’s interactions with surrounding traffic participants, the rationality of the vehicle’s own behaviors, the influence of other vehicles on the test vehicle, and whether the test vehicle exhibits any degree of rule violations or irrational operations. When this value remains at a low numerical level for a certain period, it suggests that the driver may be operating the vehicle in a less cautious manner, indicating a higher level of risk.

We set the threshold for the confidence value system at 60. When the real-time confidence value output by the test subject falls below 60 and remains in that state for 10 s, RPAM directly assigns a total score of 0 to the driver’s on-road testing capabilities. Therefore, the overall calculation framework for RPAM is as presented in the following Equation (10):

(10)$Ability=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}(Conf<60)\&(acu>10)\hfill \\ TotalS,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$

$acu$ in the equation represents the time real-time confident score, which is below 60. It is reset to 0 once the confident score is higher than the threshold.

3. System Hardware and Software Structure

The road performance assessment model integrates perception technology to conduct assessments and evaluations of the driver’s environmental perception capabilities and operational skills. The model actively employs the visual sensor system installed within the examination vehicle to collect information about the driver’s posture, particularly the turning angles of the head. Additionally, external sensors mounted on the vehicle are used to dynamically monitor the environmental conditions, the status of surrounding traffic participants, and the ego vehicle’s operational status. Finally, by synthesizing real-time data on the driver’s actions, the vehicle’s status, dynamic traffic participant states, and information about the static environment, the model comprehensively evaluates the driver’s ability to perceive and respond to conventional and hazardous scenarios over the entire driving examination, resulting in a quantified score.

There are two primary systems to make RPAM work: (1) the in-vehicle driver behavior recognition system and (2) the vehicle perception system.

3.1. Hardware and Software for In-Vehicle Driver Behavior Recognition

In order to implement an evaluation methodology based on the driver’s perceptual capabilities and a comprehensive assessment method based on confidence levels, we devised a driver behavior recognition system focused on the driver’s head posture and driving operations within the vehicle.

The in-vehicle driver behavior recognition consists of two main components: (1) the driver’s head posture recognition and (2) the vehicle operation recognition. The driver’s head posture recognition extracts information about the driver’s head position and orientation by deploying a camera at specific angles between the main driver and co-driver seats for image capture and analysis. The vehicle operation recognition primarily collects data from the vehicle’s internal communication network. It performs tasks such as recognizing the position of the throttle and brake pedals, determining the angle of the steering wheel, and identifying controls for the vehicle’s lighting system. The subsequent sections will elaborate on the primary functions and implementation processes of each system.

3.1.1. Driver’s Head Posture Recognition

By deploying a camera situated between the primary and co-driver seats, specific angles of the driver’s head turn are captured and analyzed. The implementation process unfolds as follows: Initially, in the context of the interior images captured by the camera, the YOLOv5 object detection algorithm is employed to perform facial recognition [16]. At this stage, the detected faces may encompass both the driver and potentially other unrelated passengers in the rear view of the driver. To ensure that the detected face only corresponds to the driver, a maximum face filtering method and a minimum face size threshold are incorporated.

In order to enhance the real-time capability of the entire system, our model simplifies the analytical approach to the driver’s head observation angles. Specific attention to the precise gaze direction of the driver’s eyes is omitted and, instead, we assume that the extension line through the centers of the driver’s eyes serves as the central axis for the driver’s head observation angles. Facial feature points are extracted from the identified regions of interest associated with the driver’s face. The Dlib method is utilized for this purpose, extracting 68 feature point locations of the eyes, mouth, nose, and contours of the face [17]. Key points such as the eye, nose, and mouth position are selected. Then, the Perspective-n-Point (PnP) algorithm is applied to compute the pose transformation from the face coordinate to the world coordinate [18]. Hence, we can calculate the orientation angles of the driver’s head, including yaw, pitch, and roll.

As depicted in Figure 4, this process enables a detailed assessment of the driver’s real-time head movements and posture, facilitating a comprehensive understanding of their attentiveness and focus during vehicle operation.

3.1.2. Driver’s Action Recognition

Due to the constrained space and numerous obstructions within the vehicle cabin, the vision-based recognition of driver operations is considerably intricate; therefore, the system analyzes the vehicle’s internal network signals to indirectly denote the actions of drivers and find out the status of the ego.

The specific approach involves developing a chassis signal parsing program based on the provided on-board diagnostics (OBD) signal matrix and encoding methods from the vehicle’s chassis controller area network (CAN) bus. This program is designed to extract information such as throttle position, steering wheel angle, brake pedal status, handbrake status, and lighting status, as well as the status of various internal vehicle switches. Based on the binary encoding of the original CAN signals, we identify corresponding signal names and states. The encoding is then transformed into boolean status signals and floating-point numerical signals, making it more amenable for subsequent tasks. To ensure signal stability across multiple consecutive frames, a data redundancy approach is applied for signal filtering. Ultimately, the ROS (Robot Operating System) topic subscription and publication mechanism is utilized within the ROS domain of the computing unit for broadcasting. These parsed signals serve as reference values for the driver’s vehicle operations, aiding in the recognition of the driver’s in-vehicle control actions such as accelerating or braking, and providing the information for the real-time confidence calculation.

3.2. Hardware and Software for Vehicle Perception System

To comprehensively evaluate the driver’s perceptual and control capabilities, it is essential to conduct a thorough and precise observation and understanding of the external environment. RPAM utilizes a vehicle perception system based on external sensors and GPS positioning to recognize and cognitively comprehend the surroundings.

The vehicle perception system consists of two main components: (1) a high-precision map system and (2) a comprehensive environmental perception system. The high-precision map provides static road construction information for the RPAM, and the comprehensive environmental perception system is specialized in analyzing extravehicular point cloud data generated by an LiDAR system to identify dynamic information related to traffic participants. The detailed implementations of each unit are described below.

3.2.1. High-Precision Map Creation

The high-precision map is capable of storing and retrieving comprehensive data pertaining to the overall road structure, road regulations, static signage, and real-time traffic signal information for the designated examination route. The specific implementation process is outlined as follows:

We employ a combination of LiDAR and GNSS (Global Navigation Satellite System)/INS (Inertial Navigation System) as data sources to collect point cloud information for the testing area. Subsequently, utilizing the SC-LeGO-LOAM [19], we construct a point cloud map, which serves as the foundational representation of the road environment. Then, the point cloud map is imported into a Vector Map Builder, where it is further refined through the addition of vectorized elements. This vector map encapsulates intrinsic road features, such as lanes, parking lines, traffic lights, and intersections. This vector map is fine-tuned through on-site calibration and testing to enhance its accuracy and fidelity. Finally, we integrate real-time traffic signal information into the map through roadside RSUs components, ensuring that the map reflects the real-time state of traffic lights accurately. All the road environmental data and detection results mentioned above are utilized within the RPAM model to ascertain whether the vehicle’s behavior conforms to road traffic regulations. This process, in conjunction with the traffic responsibility determination by traffic management authorities, ensures traffic safety and stability during the driving examination process.

3.2.2. Comprehensive Environmental Perception

The comprehensive environmental perception system is designed to leverage the capabilities of LiDAR for the analysis of collected point clouds. This facilitates the identification of dynamic information pertaining to external traffic participants. The LiDAR device is conventionally positioned on the top of the testing vehicle. After conducting scans, point cloud data are processed using a pretrained model based on the PointPillars algorithm, enabling object recognition from a BEV perspective, as shown in Figure 5. This procedure yields detection boxes and provides crucial information including target center coordinates, dimensions of the target bounding box, and target motion characteristics based on the identified categories.

4. Real-Word Examination and Evaluation

4.1. Evaluation Platform

To validate the feasibility and effectiveness of the RPAM system, we utilized a Geely Geometry E model [20], which is an automatic transmission vehicle, for testing. The Geely Geometry E is a compact A-class car, which is frequently employed in China’s driver’s license examinations due to its small and lightweight nature, implying relatively lower driving skill requirements. Subsequently, we conducted testing in real-world vehicle examination segments within the Beijing Changping District Northern Driver Examination Center.

We made hardware modifications to the Geely Geometry E vehicle as the base model lacks an in-cabin driver monitoring camera and external LiDAR perception equipment. To address this, we custom-fitted a Robosense 80-line LiDAR unit on top of the vehicle via a specialized mounting bracket. Simultaneously, we installed a driver monitoring camera on the interior control side. In order to obtain real-time vehicle positioning for high-precision map matching, we incorporated two GPS signal receivers and an inertial navigation system. The post-modification appearance and interior of the vehicle are depicted in Figure 6.

For the computational platform of the RPAM system used in this test, we opted for a compact domain controller based on the NVIDIA Orin chipset. The domain controller was strategically positioned at the front of the vehicle to facilitate harness management and program debugging, taking into account cost-effectiveness and spatial constraints. To facilitate testing and debugging, a portable screen with a microphone speaker was placed behind the front passenger seat within the vehicle. This setup not only provided convenience for program debuggers but also alerted the driver to the testing program’s operational status through voice announcements.

In order to monitor the real-time location and performance status of the test driver within the scenarios, including the effectiveness of their observation actions, we developed a front-end user interface for the program. The interface is divided into sections for map display, examination system status indications, and driver information, as illustrated in the following Figure 7. The blue vehicle model on the map represents the position of the test vehicle, with the model displaying both the location and vehicle orientation.

4.2. Test on Driver’s Perception Ability

The RPAM system begins recording the driver’s head rotation angle as their observation within the vehicle when they execute scenarios such as crossing intersections, changing lanes, or overtaking. In order to demonstrate its effect, this paper selected a four-lane intersection as the test scenario. With the point cloud data and high-precision map available, we calculated the road area observed by the driver in the moments leading up to traversing the intersection, using the method mentioned earlier, and we also simplified the information of the actual road by smoothing the intersection and setting sharp angles at the corners, disregarding curved transition areas.

We present the observation results of two different drivers passing through the same four-lane intersection, as shown in Figure 8. In the figure, the blue rectangle represents the driver’s vehicle, the green area denotes the intersection of the driver’s current frame of observation with the road structure, the red area indicates all the areas observed by the driver up to the current frame, and the white area represents the total area of the road scene that needs to be observed.

It can be observed that Test Driver 1 performed comprehensive observations covering the entire road condition before entering the intersection, perceiving the overall road status. In contrast, Test Driver 2 made only a slight head rotation for observation, resulting in a lower coverage of the road scene and poor observation habits during the intersection-crossing scenario.

Specifically, Test Driver 1 observed a total area of 1780.02 square meters before entering the intersection, while Test Driver 2 observed a total area of 1405.01 square meters. Test Driver 1 had an observation area ratio of 0.95906 at this intersection, resulting in an evaluation score of 91.81. In comparison, Test Driver 2 had an observation area ratio of 0.75701 at the same intersection, with an evaluation score of 51.40.

Simultaneously, during the process of passing through the intersection, Test Driver 1 encountered three traffic participants and observed two of them, while Test Driver 2 encountered only one traffic participant, which he successfully observed. According to RPAM, the final perceptual ability scores were calculated for both test drivers. Test Driver 1 scored 85.71 and Test Driver 2 scored 80.56. In this single scenario assessment, both drivers passed in terms of perceptual ability.

4.3. Test on Driver’s Control Ability

In a left turn intersection scenario, we conducted an assessment test of the vehicle control abilities among two test drivers using RPAM. Both drivers passed through the intersection between 3:00 and 4:00 p.m., and the traffic signal was green when they entered the intersection. Through three days of manual calibration, we created a simple standard schedule. By consulting the table, we obtained a standard green light passing time of 9 s for the test period at the intersection. During the verification process, Test Driver 1 took 11 s to pass through the intersection, resulting in a $Ef$ score of 83.09, while Test Driver 2, who completed it faster with 8 seconds, scored 100.

Simultaneously, we collected vehicle trajectories of the two test drivers passing through the intersection scene using GPS data, recorded at a frequency of 20 Hz. Since it was the same intersection, there was only one standard trajectory. Figure 9 shows a comparative illustration of the standard trajectory with the trajectories of the two drivers along with the actual satellite map of the scenario.

The red line stands for the standard route, the blue is for Tester 1, and the green is for Tester 2. It can be seen that Tester 1’s trajectory has a better fit. For the RMSE score of the two testers, Tester 1 achieved 0.25787, and Tester 2 achieved 0.42855. Hausdorff’s score of Tester 1 was 0.42396 and Tester 2 was 1.18460. Both the RMSE and Haus value of Tester 1 were better than Tester 2.

Combining the $Ef$, RMSE, and Hausdorff value using Equation (7), the final control ability $Opes$ for Tester 1 was 74.16 and for Tester 2 was 76.43. Tester 2 obtained a better overall ability of performing a left turn under this left-turn scenario.

4.4. Reliability Test

The vehicles equipped with the RPAM system underwent extended and repeated real-world testing under the supervision of safety officers dispatched by Public Road Security. An auxiliary brake was installed on the passenger side of the vehicle for added safety. The test involved four male drivers of varying ages, all of whom possessed valid driver’s licenses; however, their driving experience differed, with driving periods ranging from 3 months to 3 years.

Each driver took turns operating the vehicle along the same route, with each driving session lasting for a total of 2 h. From the perspective of the working duration, the test results revealed the overall stability of the system. The domain controller’s computing capacity was able to effectively handle continuous operations, encompassing tasks such as positioning, laser radar detection, and the concurrent functioning of the examination system’s software and hardware. This setup ensured stable performance while maintaining sufficient charge in the vehicle’s battery.

5. Conclusions

The paper introduced an intelligent driver capability assessment model, RPAM, designed to provide a more detailed and intelligent evaluation of whether individuals undergoing driver training possess the actual ability to drive independently on the road. This model categorizes a driver’s vehicle operation ability into two components, environmental perception ability and behavioral operational capability, while evaluating their control of the vehicle through various scenarios. Additionally, it utilizes the confidence value model to supervise the driver’s overall driving safety and stability during the entire testing process, offering a comprehensive assessment of whether the driver possesses the capacity to drive a vehicle safely, reasonably, and efficiently on the road.

The paper involved the modification of a Geely Geometry E vehicle and the installation of the necessary hardware and accompanying software for RPAM. Extensive testing was carried out on real roads in China to assess the model’s validity and durability over an extended period. The test results verified the RPAM system’s competence in evaluating the reasonableness of driver observations during operation, the efficiency of navigating through various scenarios, and the adequacy of vehicle operation skills. Furthermore, it confirmed the system’s sustained stability and functionality over an extended period.

We believe that, after further testing and optimization, RPAM has the potential to enhance the quality and safety of driver examinations in China. We aim to progressively conduct field application tests of the RPAM system in more actual examination sites. This involves collecting extensive data feedback from test drivers to further optimize specific parameters within each equation. Simultaneously, based on the pass rates of test drivers within the current examination system, we intend to guide further algorithmic optimizations of the RPAM model through data-driven feature mining. In future optimization, our focus can be placed on a more detailed perception of the driver’s state, such as confirming the objects of their gaze, and refining the observation process to catch any missed crucial traffic participants, thereby more logically quantifying the driver’s real-time perceptual ability.

Author Contributions

Conceptualization, J.G., B.Z., J.L. and Y.C.; methodology, J.G., B.Z. and J.L.; software, B.Z. and J.L.; validation, J.G., Y.L., Y.M. and Y.C.; formal analysis, B.Z. and Y.C.; investigation, J.G.; resources, J.G. and J.L.; data curation, B.Z., Y.M. and Y.L.; writing—original draft preparation, J.G., B.Z., J.L. and Y.L.; writing—review and editing, J.G., Y.L., B.Z. and Y.C.; visualization, J.L. and Y.M.; supervision, J.L., Y.M. and Y.C.; project administration, J.G., Y.L. and Y.C.; funding acquisition, J.G. and Y.L. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to the requirements of the Ministry of Public Security.

Conflicts of Interest

The authors declare no conflict of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures

Figure 1. The RPAM overall structure.

Figure 2. The simplified observed area.

Figure 3. The different capabilities of perception.

Figure 4. The performance of the driver’s head pose recognition.

Figure 5. The performance of LiDAR detection.

Figure 6. The modified test vehicle.

Figure 7. The user interface for the RPAM system.

Figure 8. The result of the real-world perception ability test.

Figure 9. The result of the real-world control ability test.

References

1. Pang, Z.; Chen, Z.; Lu, J.; Zhang, M.; Feng, X.; Chen, Y.; Yang, S.; Cao, Y. A Survey of Decision-Making Safety Assessment Methods for Autonomous Vehicles. IEEE Intell. Transp. Syst. Mag.; 2023; pp. 2-31. [DOI: https://dx.doi.org/10.1109/MITS.2023.3292511]

2. ISSUES, MOTOR VEHICLE SAFETY ISSUES Improper Driving and Road Rage. Available online: https://injuryfacts.nsc.org/motor-vehicle/motor-vehicle-safety-issues/improper-driving-and-road-rage/ (accessed on 28 September 2023).

3. Lu, J.; Peng, Z.; Yang, S.; Ma, Y.; Wang, R.; Pang, Z.; Feng, X.; Chen, Y.; Cao, Y. A review of sensory interactions between autonomous vehicles and drivers. J. Syst. Archit.; 2023; 141, 102932. [DOI: https://dx.doi.org/10.1016/j.sysarc.2023.102932]

4. Lu, J.; Yang, S.; Shi, R.; Chen, Y.; Cao, Y. Safety Co-Pilot: A System for Autonomous Vehicle to Make Decision Safer and Smarter. Proceedings of the 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC); Macau, China, 8–12 October 2022; IEEE: Piscataway, NJ, USA, 2022; pp. 1809-1816.

5. Warsame, A.M. Reforms in Assessment as a Key Driver of Quality Education in Somalia. East Afr. J. Educ. Stud.; 2023; 6, pp. 224-244. [DOI: https://dx.doi.org/10.37284/eajes.6.1.1124]

6. Toma, M.I.; Rothkrantz, L.J.; Antonya, C. Car driver skills assessment based on driving postures recognition. Proceedings of the 2012 IEEE 3rd International Conference on Cognitive Infocommunications (CogInfoCom); Kosice, Slovakia, 2–5 December 2012; IEEE: Piscataway, NJ, USA, 2012; pp. 439-446.

7. Martinussen, L.M.; Møller, M.; Prato, C.G. Assessing the relationship between the Driver Behavior Questionnaire and the Driver Skill Inventory: Revealing sub-groups of drivers. Transp. Res. Part F Traffic Psychol. Behav.; 2014; 26, pp. 82-91. [DOI: https://dx.doi.org/10.1016/j.trf.2014.06.008]

8. Maillot, P.G. A new, fast method for 2D polygon clipping: Analysis and software implementation. ACM Trans. Graph. (TOG); 1992; 11, pp. 276-290. [DOI: https://dx.doi.org/10.1145/130881.130894]

9. Solntsev, I.A.; Chintagunta, A.; Markesteijn, A.P.; Karabasov, S.A. CABARET on rotating meshes. Appl. Math. Comput.; 2023; 446, 127871. [DOI: https://dx.doi.org/10.1016/j.amc.2023.127871]

10. Baydas, S.; Karakas, B. Defining a curve as a Bezier curve. J. Taibah Univ. Sci.; 2019; 13, pp. 522-528. [DOI: https://dx.doi.org/10.1080/16583655.2019.1601913]

11. Choi, J.W.; Curry, R.; Elkaim, G. Path planning based on bézier curve for autonomous ground vehicles. Proceedings of the Advances in Electrical and Electronics Engineering-IAENG Special Edition of the World Congress on Engineering and Computer Science 2008; San Francisco, CA, USA, 22–24 October 2008; IEEE: Piscataway, NJ, USA, 2008; pp. 158-166.

12. Lee, J.S.; Lee, M.S.; Cho, I.S. Changes in Maritime Traffic Patterns According to Installation of Floating LiDAR using Spatial Analysis. IEEE Access; 2023; 11, pp. 74784-74795. [DOI: https://dx.doi.org/10.1109/ACCESS.2023.3296795]

13. Chai, T.; Draxler, R.R. Root mean square error (RMSE) or mean absolute error (MAE). Geosci. Model Dev. Discuss.; 2014; 7, pp. 1525-1534.

14. Hodson, T.O. Root-mean-square error (RMSE) or mean absolute error (MAE): When to use them or not. Geosci. Model Dev.; 2022; 15, pp. 5481-5487. [DOI: https://dx.doi.org/10.5194/gmd-15-5481-2022]

15. Lu, J.; Yang, S.; Ma, Y.; Shi, R.; Peng, Z.; Pang, Z.; Chen, Y.; Feng, X.; Wang, R.; Cao, R. et al. Modeling Driver’s Real-Time Confidence in Autonomous Vehicles. Appl. Sci.; 2023; 13, 4099. [DOI: https://dx.doi.org/10.3390/app13074099]

16. Ataş, K.; Vural, R.A. Detection of driver distraction using yolov5 network. Proceedings of the 2021 2nd Global Conference for Advancement in Technology (GCAT); Bangalore, India, 1–3 October 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 1-6.

17. Zhang, D.; Li, J.; Shan, Z. Implementation of Dlib deep learning face recognition technology. Proceedings of the 2020 International Conference on Robots & Intelligent System (ICRIS); Sanya, China, 7–8 November 2020; IEEE: Piscataway, NJ, USA, 2020; pp. 88-91.

18. Rocca, F.; Mancas, M.; Gosselin, B. Head pose estimation by perspective-n-point solution based on 2D markerless face tracking. Proceedings of the Intelligent Technologies for Interactive Entertainment: 6th International Conference, INTETAIN 2014, Proceedings 6; Chicago, IL, USA, 9–11 July 2014; Springer: Berlin/Heidelberg, Germany, 2014; pp. 67-76.

19. Wu, J.; Huang, S.; Yang, Y.; Zhang, B. Evaluation of 3D LiDAR SLAM algorithms based on the KITTI dataset. J. Supercomput.; 2023; 79, pp. 15760-15772. [DOI: https://dx.doi.org/10.1007/s11227-023-05267-3]

20. Geometry E. Available online: https://www.licarco.com/geometry-e (accessed on 28 September 2023).