The Entropy-Cost Function Evaluation Method for Unmanned Ground Vehicles

Abstract

To improve the technology of unmanned ground vehicles, it is necessary to conduct a proper evaluation on various technologies. Previous evaluation methods are mainly based on completion of the task; this may mislead most of teams of unmanned ground vehicles using a conservative strategy during the evaluation. In this paper, a new evaluation method is proposed. Based on typical working conditions including intersection, car-following, and obstacle-avoiding, the new evaluation indicator system is established, and the entropy-cost function method is applied to the comprehensive evaluation of unmanned ground vehicles. As reported in a numerical example, the proposed evaluation method can get a quantitative result that authentically reflects the intelligent behavior level of unmanned ground vehicles.

Academic Editor:Xiaosong Hu

Department of Transportation Engineering, Beijing Institute of Technology, Beijing 100081, China

Received 12 June 2015; Accepted 7 October 2015; 20 October 2015

This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

1. Introduction

Based on the preestablished evaluation system, the technology of intelligent vehicles abroad has developed rapidly in recent years. Obviously an excellent evaluation system can guide participants to improve the performance of intelligent vehicles in evaluation/test. For example, none unmanned vehicles finished the entire race in the US intelligent vehicle competitions DARPA 2004 [1]. Guided by the rules/evaluation rules of Grand Challenge, the teams improved their unmanned vehicles and five teams accomplished the entire distance in 2005 competition [2]. In the history of technology development, evaluation system has played an important role.

The 2nd DARPA Grand Challenge simply used the "the number of finishing the races" and "the total number of gates through" to rank the teams [3]. The 3rd DARPA Grand Challenge, Urban Challenge, required that all participants must obey traffic rules [4]. "Future Challenge 2009" was held in June 2009 [5]; the race used the third-party testing to evaluate the implementation efficiency of intelligent vehicles in competitions. This attempt has promoted and facilitated the innovation and development of vehicle verification platform [6]. "Future Challenge 2012" competition was held in November 2012, Chifeng City, Inner Mongolia [7]. The race mainly tested the research level of cognitive computing about visual and auditory information [8]. U-Turn evaluation environment is accepted for more competitions in recent years, but the evaluation environment cannot take more uncertain factors into consideration [9, 10].

Most of existing foreign evaluation methods of domestic and abroad intelligent vehicle competitions used the mission-driven evaluation approach, which has an obvious shortcoming, leading many teams to adopt a conservative strategy [11]. For example, in order to avoid the violation of traffic regulations, many teams chose to park and wait [12] rather than interact with the environment as manned vehicles do. Even though they can get high score, it is a departure from the purpose of technological development [13]. As for domestic methods, the evaluation system is obviously not comprehensive; many factors are not considered in evaluation/test. For example, the papers [7-10] just focus on dynamic driving abilities and ignore static conditions, interactive abilities with environment. In the future, intelligent vehicles will interact with more people and cars in complex scenes [14-16]; thus the evaluation method needs improvement. The method must take more comprehensive evaluation factors into account and reflect the performance of unmanned vehicles objectively.

This paper concentrates on proposing a novel evaluation method for intelligent vehicles that is based on information entropy and cost function. The information entropy checks all evaluation indicators' weight from information amount aspect to handle uncertainty problems in evaluation/test; the cost function checks the intelligent level of each of abilities to get specific evaluation score. The evaluation method also concludes a new rigorous evaluation indicator system based on typical working conditions; each typical working condition of intelligent vehicles is subdivided into different physical indicators to reflect real situation. Not only the completion time of indicators but also all the details are taken into account, even down to the completion quality and veracity of the various secondary indicators. The new evaluation method refines three aspects of evaluation process: evaluation indicator system establishment, indicators' weight arrangement, and evaluation score classification to guarantee the objectivity, comprehensiveness, and scientificalness of evaluation results. The logic and construction of the paper are shown in Figure 1.

Figure 1: The logic and construction of the paper.

[figure omitted; refer to PDF]

The final evaluation results can show the intelligent level of unmanned vehicles and its weakness and then guide the participating vehicles to move in the right direction and goals of high-tech development.

2. Foundation of the Evaluation Indicator System

As described above, most of the practical evaluation activities were mission-driven, which resulted in the incomprehensive evaluation indicator system. Thus the evaluation results were partial. Most participants could get higher score just because of their own research superiorities. It is not fair and objective. That means a proper selection of evaluation indicators is an important part in the evaluation.

Evaluation of unmanned vehicle intelligent behavior is a multilevel comprehensive evaluation problem. Considering the characteristics of unmanned vehicles' data which are scattered and the advantages and disadvantages of traditional evaluation indicators now, expert opinions and analysis of typical working conditions are selected; this can not only make full use of the experts' cognitive knowledge of unmanned vehicles but also avoid missing important indicators. The result is relatively accurate with the rigorous indicators selection process (see Figure 2).

Figure 2: Foundation of the evaluation indicator system.

[figure omitted; refer to PDF]

The paper mainly takes two factors into consideration: objective one and subjective one, which are given in Figure 2. Objective working conditions reflect the external environment; subjective opinions from experts reflect the internal attitude in the evaluation/test.

The typical working conditions are summarized from many intelligent vehicle competitions. In each of intelligent vehicle competitions, the participants will encounter three main working conditions; those working conditions almost conclude all of intelligent driving behaviors in evaluation/test. According to the typical working conditions including intersection, car-following [17], and obstacle-avoiding, the paper analyzes the performance of different situations that can show the real level of unmanned vehicles' intelligence and then summarizes essential features of each typical working condition. Each typical working condition of intelligent vehicles reflects main intelligent behavior; each behavior has its tasks. Evaluation indicator structure is given in Table 1. All indicators are established through considering real situations in the driving process; the physical characteristics can be reflected in the indicator system.

Table 1: The evaluation system of unmanned ground vehicles' intelligent behavior based on typical working conditions.

Evaluation objective	Typical intelligent behaviors	Evaluation auxiliary indicators
Multi-indicator evaluation system for unmanned vehicles	Intersection behavior	Parking precisionRestart abilitySpeed capabilityBraking deceleration
Obstacle-avoiding behavior	Early warning Avoidance in right anglePath replanning
Car-following behavior	StimulationSafe distanceSpeed optimization

The indicator called "Parking precision" tests the ability to park the vehicle at the right place and the right time when facing intersection. "Restart ability" tests the ability to identify complex intersection conditions and participate in the traffic intelligently. "Speed capability" reflects the performance on the speed controlling, acceleration, and deceleration. "Braking deceleration" tests braking quality. "Early warning" tests video detection system and the ability to identify different obstacles. "Avoidance in right angle" tests whether the vehicle can use less space to avoid the obstacles. "Path replanning" can guarantee the vehicle will not abort the original destination. "Stimulation" tests the ability to calculate the fore car's real-time speed, thus taking actions timely. "Safe distance" evaluates the stationary and the following model of the intelligent vehicle [18, 19]. "Speed optimization" tests the ability to control distance through keeping optimal speed.

3. Information Theory and Entropy for Indicators' Weight

This paper introduces knowledge of information theory to determine the weight distribution parameters. In 1948, in the paper "A Mathematical Theory of Communication," Shannon used Probability and Statistics approach to the study of communication systems, revealing that the object of communication system is information and then proposed the concept of entropy through describing the information quantitatively.

Based on Shannon's theory, the recipient cannot predict the message that will be received; therefore, uncertainty exists objectively because of the message's randomness. Delivery of messages is a process to eliminate the uncertainty of information, a process from uncertainty to certainty. Uncertainty in information theory is a kind of inherent property existing in working process, so the concept of entropy can be applied to the evaluation of intelligent behaviors of unmanned ground vehicles. Information entropy can determine the key indicators in terms of the amount of information to calculate the indicator weight parameters, and the entropy of information can be calculated as follows: [figure omitted; refer to PDF]

3.1. Judgment Matrix

To judge all factors with 1~9 scaling method to determine each indicator's importance, experts' suggestion is summarized to construct the judgment matrix [figure omitted; refer to PDF] of fractional values, and evaluation indicator system has [figure omitted; refer to PDF] aspects behaviors, [figure omitted; refer to PDF] objects, [figure omitted; refer to PDF] auxiliary indicators, [figure omitted; refer to PDF] , and [figure omitted; refer to PDF] .

3.2. Matrix Data

According to the linear proportional relationship, the decision matrix [figure omitted; refer to PDF] is reformed as standardization to get standardized matrix [figure omitted; refer to PDF] and then normalize [figure omitted; refer to PDF] : [figure omitted; refer to PDF]

3.3. Information Entropy

Calculation of [figure omitted; refer to PDF] indicator's information entropy is as follows: [figure omitted; refer to PDF]

3.4. Information Utility Value

The real parameters that reflect indicator weights are the values of information utility; the values can be calculated by the following formula: [figure omitted; refer to PDF]

3.5. Total Indicator Weight

The importance weight of all factors of this layer corresponding to the above layer of a certain factor that can be calculated using the result of all single hierarchical sorts at the same hierarchy is as follows: [figure omitted; refer to PDF] [figure omitted; refer to PDF] is objective weight of each indicator.

4. Cost Function for Score Classification

According to the 2007 DARPA Urban Challenge's low-speed, low-density traffic environment issues, scholars at Carnegie Mellon University proposed a kind of robust highway autonomous driving technology that is combined with the cost evaluation method [20]. The cost function method can also be used in Chinese intelligent vehicle competitions to quantify the indicators. The cost function method is bound to promote competition vehicles toward the direction of the minimum cost function in the process of technology development, thus leading unmanned ground vehicles to finish the task with high-quality. The technological level of Chinese unmanned vehicles on its natural environment perception and decision-making capacity is gradually improving.

Taking the indicator "Path replanning" as a research object, unmanned vehicles need to consider the process cost control that means more planning time and more useless operating range deserve more penalty and higher process cost. The cost can be calculated by the following formula: [figure omitted; refer to PDF]

To make sure of the comprehensive consideration for all indicators and the full use of evaluation information, the paper adopts a weighted average algorithm to get final evaluation results by decomposing calculation. As for typical intelligent behaviors, the paper presents each cost function as [figure omitted; refer to PDF] , [figure omitted; refer to PDF] , and [figure omitted; refer to PDF] . [figure omitted; refer to PDF] can be calculated as follows: [figure omitted; refer to PDF]

[figure omitted; refer to PDF] presents the cost of [figure omitted; refer to PDF] indicator of intersection behavior; [figure omitted; refer to PDF] presents the weight of [figure omitted; refer to PDF] indicator of intersection behavior; [figure omitted; refer to PDF] presents the maximum value of the cost values; [figure omitted; refer to PDF] is the penalty factor. The final cost of unmanned vehicles can be calculated as follows: [figure omitted; refer to PDF]

5. The Entropy-Cost Function Evaluation Method

5.1. Weight Distribution of Each Indicator

According to the evaluation indicators for unmanned vehicle intelligent behavior comprehensive evaluation model, the judgment matrix of each level is constructed following the 1~9 scaling methods by expert group composed by research members in related fields. [figure omitted; refer to PDF] , calculated by formulae (2)-(5), is the weight in accordance with the overall goal layer of 10 evaluation indicators. According to the additivity of information entropy, the weight of typical working conditions' behavior can be calculated through accumulating affiliated indicators directly as follows.

Typical working conditions' behavior weight is as follows: [figure omitted; refer to PDF]

Affiliated indicators weight is as follows: [figure omitted; refer to PDF]

Then the level of indicators' weight coefficients is determined, and the evaluation results will be more objective because of different treatments of indicators.

5.2. Comprehensive Score Based on Cost Function

The weight of each indicator in the evaluation system has been calculated in Section 5.1, and the cost combined with weight will be the final value that represents the comprehensive score of unmanned vehicles. In the testing, the paper uses ranking method to define the basic cost value; the real performance will be quantified in the corresponding rank. Taking "Parking precision" as a research object, the cost can be represented as follows: [figure omitted; refer to PDF]

Since the data are rough, the score can just estimate a general intelligent level of unmanned vehicles; two teams with almost same score must make further comparison about more details. The rest of the indicators' cost value can be calculated identically, and the first class indicators' cost is [figure omitted; refer to PDF]

6. Unmanned Vehicle Intelligent Behavior Evaluation

6.1. Each Indicator's Score

The score of each indicator is given based on the comparison between optimal performances and minimum acceptable performances from manned driving conditions roughly; the gap will be divided into several score ranks (see Figure 3).

Figure 3: Calculation of each indicator's score.

[figure omitted; refer to PDF]

The method for calculating score is a transition from manned driving to driverless. As unmanned vehicles' development direction is human-like capacity in driving, the method is accordingly beneficial for technical updating.

6.2. Each Indicator's Cost Value

Also taking the indicator "Path replanning" as the research object, some data in "Future Challenge 2012" competition are shown in Table 2.

Table 2: The "Path replanning" data.

The competing teams	A	B	C	D	E	F
Planning time (s)	13	18	10	13	11	15
Operating range (m)	8.3	7.4	9.2	6.5	10.9	None
Cost value	40	60	20	20	60	100

Table 2 shows that car F's data are invalid because of the lane departure in this test. The score is corresponding to the planning time and operating range, optimal condition in manned driving cannot be quantified with 7-9 (s) and 5-7 (m), and minimum acceptable condition is 18-20 (s) and 15-17 (m). The threshold value of score classification is based on human experience, so the data can only be processed roughly to get the score-rank of performance.

Analyzing the data of 10 indicators of team A in the competition, "Parking precision" is corresponding to parking time and the distance between the front of the vehicle and the stop line; "Restart ability," restart time and acceleration; "Speed capability," horsepower; "Braking deceleration," deceleration and braking distance; "Early warning," lead time and accuracy rate; "Avoidance in right angle," offset distance and maximum offset angle; "Stimulation," time gap in gear shift; "Safe distance," reasonable distance; "Speed optimization," optimal speed and adjustment time. The indicators' data and the corresponding score are listed in Table 3.

Table 3: The data and score of team A in the competition.

Indicators	[figure omitted; refer to PDF]	[figure omitted; refer to PDF]	[figure omitted; refer to PDF]	[figure omitted; refer to PDF]	[figure omitted; refer to PDF]	[figure omitted; refer to PDF]	[figure omitted; refer to PDF]	[figure omitted; refer to PDF]	[figure omitted; refer to PDF]	[figure omitted; refer to PDF]
Data-1	0.7	4.5	59	12.3	30	1.3	13	7	3.6	36
Unit-1	m	s	h	m	s	m	s	s	M	km/h
Data-2	13	1.5		2	89.3	53	8.3			18
Unit-2	s	m/s2		m/s2	%	°	m			s
Cost	20	20	40	20	20	60	40	80	20	40

6.3. Total Cost Value

Calculating the data from Table 3 to get A's total cost value as described in Section 5.2, the cost values of three typical intelligent behaviors are [figure omitted; refer to PDF]

The total cost is [figure omitted; refer to PDF] .

The result shows that team A is at 2nd (20-40) level in the competition.

7. Conclusion

In this paper, the evaluation of unmanned ground vehicles is studied. Based on the typical working conditions of unmanned ground vehicles, a multilevel indicators evaluation system is established. Because the uncertainty is intrinsic property of each evaluation process, information entropy is applied to quantify the weight of each indicator, and each factor matches different weight coefficients to highlight the importance of the evaluation factor. Then entropy-cost function evaluation method is proposed to evaluate team A's unmanned vehicle in "Future Challenge 2012" competition. From the quantitative results, the teams can learn the vehicle's intelligent level generally and find their technical shortcomings in some specific indicators; thus the team will get the right development direction.

The first issue to be developed is more detailed quantification of indicators' score. The technology development is from manned driving to driverless, but manned driving behaviors are also difficult to quantify. Therefore, the cost function should be reformed for more precise cost value through both qualitative and quantitative manned driving empirical data.

Acknowledgments

This project was supported by the National Natural Science Foundation of China (no. 90920304 and no. 91120010) and the Fundamental Research Foundation of Beijing Institute of Technology (no. 20120342026).

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

References

[1] R. Behringer, S. Sundareswaran, B. Gregory, R. Elsley, W. Guthmiller, R. Daily, D. Bevly, "The DARPA a grand challenge-development of an autonomous vehicle," in Proceedings of the IEEE Intelligent Vehicles Symposium, pp. 226-231, Parma, Italy, June 2004.

[2] DARPA http://www.darpa.mil/

[3] E. Krotkov, S. Fish, L. Jackel, B. McBride, M. Perschbacher, J. Pippine, "The DARPA PerceptOR evaluation experiments," Autonomous Robots , vol. 22, no. 1, pp. 19-35, 2007.

[4] C. Urmson, J. Anhalt, D. Bagnell, C. Baker, R. Bittner, M. N. Clark, J. Dolan, D. Duggins, T. Galatali, C. Geyer, M. Gittleman, S. Harbaugh, M. Hebert, T. M. Howard, S. Kolski, A. Kelly, M. Likhachev, M. McNaughton, N. Miller, K. Peterson, B. Pilnick, R. Rajkumar, P. Rybski, B. Salesky, Y.-W. Seo, S. Singh, J. Snider, A. Stentz, W. Whittaker, Z. Wolkowicki, J. Ziglar, H. Bae, T. Brown, D. Demitrish, B. Litkouhi, J. Nickolaou, V. Sadekar, W. Zhang, J. Struble, M. Taylor, M. Darms, D. Ferguson, "Autonomous driving in urban environments: boss and the urban challenge," Journal of Field Robotics , vol. 25, no. 8, pp. 425-466, 2008.

[5] Future Challenge, http://ccvai.xjtu.edu.cn/news

[6] G. Xiong, X. Zhao, H. Liu, S. Wu, J. Gong, H. Zhang, H. Tan, H. Chen, "Research on the quantitative evaluation system for unmanned ground vehicles," in Proceedings of the IEEE Intelligent Vehicles Symposium (IV '10), pp. 523-527, IEEE, San Diego, Calif, USA, June 2010.

[7] X. Gao Research on Information Processing of Intelligent Behavior and Evaluation of Unmanned Vehicle Based on Typical Working Condition , Beijing Institute of Technology, 2012.

[8] S. X. Liu Research on Evaluation of Unmanned Vehicle Intelligent U-Turn Behavior , Beijing Institute of Technology, Beijing, China, 2014.

[9] X. Zhang, Y. N. Zhao, L. Gao, D. H. Hao, "Evaluation framework and method of the intelligent behaviors of unmanned ground vehicles based on AHP scheme," Applied Mechanics and Materials , vol. 721, pp. 476-480, 2015.

[10] X. Zhang Research on Evaluation of Intelligent Behaviors through Specials Areas for Unmanned Ground Vehicles , Beijing Institute of Technology, 2015.

[11] J. Wei, J. M. Dolan, "A robust autonomous freeway driving algorithm," in Proceedings of the IEEE 2009 Intelligent Vehicles Symposium, pp. 1015-1020, Xi'an, China, June 2009.

[12] M. Bertozzi, A. Broggi, A. Fascioli, "Vision-based intelligent vehicles: state of the art and perspectives," Robotics and Autonomous Systems , vol. 32, no. 1, pp. 1-16, 2000.

[13] J. Leonard, J. How, S. Teller, M. Berger, S. Campbell, G. Fiore, L. Fletcher, E. Frazzoli, A. Huang, S. Karaman, O. Koch, Y. Kuwata, D. Moore, E. Olson, S. Peters, J. Teo, R. Truax, M. Walter, D. Barrett, A. Epstein, K. Maheloni, K. Moyer, T. Jones, R. Buckley, M. Antone, R. Galejs, S. Krishnamurthy, J. Williams, "A perception-driven autonomous urban vehicle," Journal of Field Robotics , vol. 25, no. 10, pp. 727-774, 2008.

[14] X. Hu, N. Murgovski, L. M. Johannesson, B. Egardt, "Comparison of three electrochemical energy buffers applied to a hybrid bus powertrain with simultaneous optimal sizing and energy management," IEEE Transactions on Intelligent Transportation Systems , vol. 15, no. 3, pp. 1193-1205, 2014.

[15] X. Hu, N. Murgovski, L. M. Johannesson, B. Egardt, "Optimal dimensioning and power management of a fuel cell/battery hybrid bus via convex programming," IEEE/ASME Transactions on Mechatronics , vol. 20, no. 1, pp. 457-468, 2015.

[16] X. Hu, L. Johannesson, N. Murgovski, B. Egardt, "Longevity-conscious dimensioning and power management of the hybrid energy storage system in a fuel cell hybrid electric bus," Applied Energy , vol. 137, pp. 913-924, 2015.

[17] S. Li, K. Li, J. Wang, X. Lian, H. Ukawa, D. Bai, "Modeling of heavy-duty truck driver's car-following characteristics and its comparative verification," International Journal of Automotive Technology , vol. 11, no. 1, pp. 81-87, 2010.

[18] B. Ran, H. Tan, J. Feng, W. Wang, Y. Cheng, P. Jin, "Estimating missing traffic volume using low multilinear rank tensor completion," Journal of Intelligent Transportation Systems: Technology, Planning, and Operations , 2015.

[19] H. Tan, G. Feng, J. Feng, W. Wang, Y.-J. Zhang, F. Li, "A tensor-based method for missing traffic data completion," Transportation Research Part C: Emerging Technologies , vol. 28, pp. 15-27, 2013.

[20] J. Wei, J. M. Dolan, B. Litkouhi, "A prediction- and cost function-based algorithm for robust autonomous freeway driving," in Proceedings of the IEEE Intelligent Vehicles Symposium (IV '10), pp. 512-517, IEEE, San Diego, Calif, USA, June 2010.