Leveraging UAVs to Enable Dynamic and Smart Aerial Infrastructure for ITS and Smart Cities: An Overview

Abstract

Micro-unmanned aerial vehicles (UAVs), also known as drones, have been recognized as an emerging technology offering a plethora of applications touching various aspects of our lives, such as surveillance, agriculture, entertainment, and intelligent transportation systems (ITS). Furthermore, due to their low cost and ability to be fitted with transmitters, cameras, and other on-board sensors, UAVs can be seen as potential flying Internet-of-things (IoT) devices interconnecting with their environment and allowing for more mobile flexibility in the network. This paper overviews the beneficial applications that UAVs can offer to smart cities, and particularly to ITS, while highlighting the main challenges that can be encountered. Afterward, it proposes several potential solutions to organize the operation of UAV swarms, while addressing one of their main issues: their battery-limited capacity. Finally, open research areas that should be undertaken to strengthen the case for UAVs to become part of the smart infrastructure for futuristic cities are discussed.

1. Introduction

The proliferation of internet and communication technologies is driving significant social and economic changes, fostering the transition of smart cities into the future. As the number of mobile devices and embedded computers grow, and as they contribute to even faster-growing data sets, new technologies are emerging to the forefront of extending wireless network connectivity through fifth-generation (5G) cellular broadband networks and the Internet of things (IoT) [1]. Aerial and spectrum regulations have evolved to accommodate multirotor unmanned aerial vehicles (UAVs) for commercial and private use. As a result, UAVs will indeed become critical enablers of many emerging applications touching various aspects of our lives, including but not limited to: pollution monitoring [2,3], relaying data transmissions [4], acting as network access points [5], surveillance [6,7,8], good delivery [9,10], search and rescue [11], archeology and architecture [12], and intelligent transportation systems (ITS) [13]. UAVs can also play a key role in many applications in rural areas with limited access to the connected ground infrastructure. In fact, UAVs can be employed in smart farming to monitor crops and map the field [14]. They can also be used as remote sensors to detect and classify trees [15].

In smart cities, the main goal of ITS is to mitigate and improve traffic-related issues and challenges, such as traffic congestion, user mobility, and safety. Moreover, ITS implementation’s benefits include saving the city’s energy and enhancing the sustainability level. Therefore, ITS require the collaboration of different stakeholders and the deployment of multiple technologies to bring optimal benefits for smart cities. Moreover, they necessitate the effective exploitation of the available data shared across the city by a variety of road users and sensors. This includes the interactions among vehicles (vehicle-to-vehicle or V2V), with the road infrastructure (vehicle-to-infrastructure, V2I), and with other users/devices (vehicle-to-anything, V2X). Thus, the data and information exchanged across the infrastructure of ITS can play an essential function in providing reliable information for better decision-making and enhanced operation [16].

UAVs can play many essential roles that can contribute to enhancing the mobility and transportation services in smart cities. In fact, the array of mounted equipment, e.g., onboard sensors, cameras, and communication interfaces, that UAVs can support would allow them to be an instrumental player in ITS, performing a long-term collection of data or real-time monitoring of the traffic network while connected to the network at large. UAVs hold many significant advantages over more conventional ground and mobile vehicles. Their flying capabilities give UAVs free mobility in the three spatial dimensions while providing portable connectivity that fixed-ground infrastructure cannot match. The flight capability of UAVs also allows for airborne networks to operate. By operating well above the ground level, UAVs can have more reliable communication links with other devices on the ground, leading to more direct connections with higher line-of-sight probability less hampered by interference [17].

Furthermore, groups of UAVs can collectively provide additional connectivity, adjusting to demand fluctuations and system disruptions. They can reorganize themselves on the fly, forming a dynamic Internet of drones (IoD) network [18]. The coordinated mobility of the UAVs in the swarm that allows a flexible network topology gives infrastructure operators additional oversight and control to improve the quality of experience of their ITS applications [19]. However, this flexibility demands some work in the organization of the airspace [20]. Indeed, efficient UAV management is required to ensure the effective completion of the different assigned missions. Moreover, it is necessary to take into account the limited capabilities of UAVs in accomplishing long-duration missions due to energy limitations from being powered by batteries. Consequently, UAVs must interrupt their operations to replenish their batteries [21]. Therefore, managing large-scale UAV fleets must consider these limitations while keeping the primary application objectives in mind.

The culmination of this work has led to the exploration of a UAV-dominated infrastructure of ITS and its potential applications in smart cities. Moving forward, this paper overviews work related to managing and planning for UAVs in ITS to highlight their viability in these systems while also recognizing relevant challenges. Afterward, it presents selected contributions to mitigate some of the most severe challenges, e.g., energy limitation, to maximize the effectiveness of UAVs in the prior-mentioned applications. Furthermore, this paper presents future research directions that would widen the pool of UAV-based applications as well as improve the reliability of these flying IoT devices in future ITS and smart city systems.

2. UAV Applications for Smart Cities and ITS

In this section, selected applications from the literature to highlight the niche benefit of UAVs in ITS are investigated. We conclude with a list of challenges for future UAV applications pertaining to smart cities. Figure 1 illustrates a few examples of the applications of UAV-based ITS. Examples include communication applications such as delivery, traffic monitoring, surveillance, and media and entertainment, each requiring different quality of service and delay tolerance levels. The applications are categorized based on two main factors: delay tolerance and bandwidth demand. Different challenges can be encountered for each of these applications according to their objectives and constraints.

Several smart city applications require near-zero latency to ensure reliable and smooth operation. For instance, when a flying RSU is made available to report the traffic situation in a certain area, it is mandatory to relay the collected data to the ITS’s control system in real time [13]. Another interesting example would be the security and surveillance applications in public spaces where UAVs fitted with cameras can act as literal “eyes in the sky” allowing for the monitoring of suspicious actors. Such applications require continuous transmission of high-resolution videos over the network. For this reason, UAVs will need effective wireless technologies and data-routing strategies to stream their data toward the central command systems [6]. In addition to the technology, it is worth optimizing the data routes taken by UAVs in order to develop an architecture that minimizes latency across transmissions through the UAV swarm network [5]. Utilizing UAVs to form a flying ad hoc network (FANET) to transfer data [22] allows for a better control of the transmission channel. A significant disadvantage of utilizing a FANET is the extensive scheduling, control, and communication framework to ensure reliable operation. UAVs must be able to position themselves within each other’s transmission ranges to form a multihop network. Consequently, advanced routing solutions are required while considering the mobility of UAVs necessary.

Delay-tolerant applications for UAVs do not require the real-time exchange of data. On the contrary, they tolerate inevitable data transmission and collection delays as long as these delays do not impact the application’s performance. In that case, the UAVs can store the data onboard and deliver it to the central sink after a while. For example, we can cite the applications that primarily revolve around long-term data collection in hard-to-reach locations, such as pollution monitoring at high altitudes where UAVs are equipped with low-cost sensors to form a flying mobile wireless sensor network (WSN), capable of taking air quality readings [2]. In such cases, it is recommended to optimize the data collection process, which includes, for instance, the UAV trajectories and the clustering of source nodes. Another application would be the use of UAVs to inspect transportation infrastructure and identify roads requiring maintenance. In that case, an optimized tour for the UAVs needs to be designed while considering its energy limitation and maximizing the coverage efficiency.

Heavy-bandwidth applications revolve around transmitting more data, such as high-definition video transmission. A safety-related role of UAVs in ITS is in collision monitoring and prevention systems [13] that will aid a quick decision-making and provide proper levels of information to make commutes safer and more efficient. When a collision is detected, UAVs may be deployed to provide emergency personnel a first look at the severity of an accident, providing information detailing the magnitude of the accident. This information allows for a measured response by authorities, allowing them to respond with an optimal amount of personnel to preserve readiness and reliability for other disturbances that may arise. UAVs can additionally provide a mode of communication for people in an accident, especially if they are incapable of accessing their mobile devices and are still conscious. Beyond accident monitoring systems, UAVs can aerially survey roadways, providing an avenue to warn approaching vehicles about hazards impeding roadways to provide drivers additional information to reroute their trip and hence reducing congestion. However, these systems have one major drawback: multirotor UAVs envisioned for these tasks would be too small to effectively remove debris from the road in the case of impediments or rescue collision victims. Hence, they are, for now, only considered in a data-powered support role. Communication aspects of this application also require large amounts of continuous bandwidth that may be interfered with in poor weather, which could also be a factor in accidents.

While UAVs have many practical advantages, there are numerous challenges to overcome before they are fully viable as instruments of future smart city systems. The main challenge facing the utilization of UAVs across all applications is managing energy consumption efficiently. UAVs typically run on a limited-charge battery, limiting the effective range of the UAV’s flight, the range of onboard sensors, and the number of onboard technologies that may be present. Another potential challenge is spectrum scarcity due to (1) a massive growth of predicted connected devices and (2) UAV control transmissions interfering with the actual network communications’ traffic [1]. The authors of [23] proposed a system for partitioning the UAV-to-UAV (U2U) communications from UAV-to-infrastructure (U2I) or end-user communications, ensuring that the control system does not adversely reduce the performance of the communication system at large. Management issues of the UAVs in transit are an issue: complex systems must be implemented to coordinate the UAV fleet, plan the UAV trajectory/missions, and ensure collisions are minimal or nonexistent. The main challenge of energy limitations, as well as the mobility and coordination challenges, is addressed by multiple studies on solving open-planning problems for UAV operations management [24,25,26]. Afterward, a focus on possible new challenges and potential work to mitigate them are discussed in the perspectives and future research considerations.

Table 1 examines some of the recent studies that utilize UAVs to solve some challenges of ITS in smart cities. The main challenges that are tackled can be summarized in the following areas: (1) traffic monitoring, (2) personal and goods mobility, (3) information exchange, (4) network and communication, (5) data processing and computation, and (6) security and privacy. In addition, this table provides the methods used in these studies to solve the investigated issues. Through these selected references, we can confirm that the applications of UAVs for ITS are very broad and varied, with different objectives. Diverse technical methods and algorithms address ITS-related challenges, including optimization, algorithm design, game theory, and artificial intelligence.

3. Potential Solutions to UAV Shortcomings as ITS Infrastructure

This section discusses some potential solutions that contribute to managing the flying UAV fleet while satisfying the applications’ objectives of ITS and dealing with the different challenges that can be encountered.

3.1. Charging Station Placement

Due to the limited flying range of UAVs caused by the battery limitation and the need to return to charging stations to replenish the batteries frequently, in [36], a joint RSU–UAV charging station planning approach for ITS, where the objective was to find the best geographical locations for the charging stations that would complement gaps in the RSU coverage, was proposed. Besides the motivation to fill gaps in the RSU coverage, we aimed to consider the joint planning of RSUs and UAV base stations in tandem, as this still needed to be thoroughly explored in the literature. To this end, we combined RSU planning considerations such as formulating the problem as a constrained combinatorial maximization problem [37] with regards to the UAV base station planning discussed in [26,38] and estimating traffic conditions based on the average behavior [39], while considering how to incorporate financial and energy-efficiency considerations as well.

The placement decisions were based on average road network statistics, which measure the accident and incident histories in the area of interest’s road segments and intersections. The charging stations had to be located in regions where UAVs can monitor the most important points of interest. A few conditions were taken into account. (i) The UAV had to operate within a fixed range from the base station based on relationships defined in [26]. (ii) We considered the long-run operational characteristics of the UAV based on how they were scheduled, as a UAV could spend fractions of the time period either charging, flying to a task, or performing a monitoring task, and the UAVs were scheduled in a way that avoided the depletion of the battery while in transit or operation. (iii) The UAVs that operated from the base station covered a circular subarea of the base station’s coverage area, thus leading to a long-run coverage probability that could be considered when planning alongside RSUs. (iv) There were amortized capital costs and recurring operational costs that could be discounted onto a series of uniform cash flows, allowing for a time-independent consideration in the trade-offs of costs when considering a budget amortized over the same set of period increments. (v) The standardization of the cash flows of the costs allowed for the consideration of how green energy could offset the operational costs of the UAVs and RSUs. In [36], both an exact method approach based on a novel mixed-integer programming (MIP) problem formulation and a heuristic method were employed to determine optimized UAV charging station and RSU locations for realistic traffic network maps based on the aforementioned conditions.

We demonstrate the joint UAV and RSU coverage that was calculated by the MIP-based technique in peak traffic time in Figure 2. The positioning of UAVs was done so that the coverage was maximized within the allocated budget. The RSUs and UAV base stations’ range overlapped because multiple RSUs deactivated to preserve energy during less active hours. It is also worth noting that optimizing coverage efficiency did not imply maximizing the number of covered points. Instead, the algorithms prioritized sites of interest with greater fitness levels, where the fitness was specified as a utility function in the range $[0,1]$. These values would depend on criteria that ITS operators stated to the system, such as traffic density, accident rate, etc. [36]. The remaining charging stations were then distributed to places with lower fitness levels or multiple junctions and road segments. Hence, the points of interest (e.g., intersections) characterized by a high fitness score were given a higher priority to be covered by a charging station. The fitness was designed to avoid the redundant placement of charging stations and RSUs to improve the coverage efficiency of the deployed infrastructure [36].

Figure 3 depicts the effect the number of candidate locations had on the overall effectiveness of the joint RSU/UAV coverage using the optimal approach and a heuristic approach. The exact approach, denoted by opt_CE corresponded to the solution obtained by solving the optimal MIP, which is, in practice, very complex. Therefore, we designed a heuristic algorithm to optimize the planning. The heuristic approach had different levels of increments, denoted by, e.g., 2_CE, 5_CE, and 10_CE in the figure (i.e., dividing the power budget into several possible values), achieving a trade-off between complexity and coverage efficiency [36]. The heuristic increments corresponded to how the algorithm incremented the reduction in the power for a potential unit (charging station or RSU). The larger values were essentially larger step sizes for reducing power. Hence, 2_CE corresponded to a decrease of 2% from the maximum per step size (50 total potential settings in power to the base station) and 50_CE corresponded to 100%, 50%, or not set/installed. The smaller values had more degrees of freedom when iterating for a solution, at the cost of requiring more steps to find a solution. Therefore, in terms of coverage efficiency, the algorithm tended to perform better with small increments. The case of 1_CE corresponded to the highest complexity level of the algorithm. However, the algorithm failed to achieve the effective coverage efficiency due to the fact that the maximum number of iterations had been reached and the algorithm failed to achieve a near-optimal solution. Hence, 2_CE achieved the best coverage efficiency in our setting. Figure 3 also shows that as the number of candidate points (out of the around 215 total points of interest in the problem scope) increased, the overall coverage efficiency increased, but only to a certain point. This was likely due to the fact that beyond a certain threshold, the likelihood of an effective candidate location being contained in the set of all candidate locations increased rapidly. It could be observed that beyond the consideration of one-quarter of all points, there was no immense benefit to consider any more, as there was no notable increase in coverage efficiency with a drastic increase in complexity beyond that point.

To summarize, the optimized placement of UAV charging stations in urban areas must be carefully carried out to increase the coverage efficiency of the road network while taking into account the need for UAVs to go back to recharge their batteries regularly. Planning UAV charging stations is the first step for any large-scale aerial infrastructure in smart cities. The optimized locations of the charging stations can positively impact the performance of the UAV operation, help save energy by reducing redundant flying time, and guarantee the smooth operation of the fleet.

3.2. Tour Planning for Flying IoT Gateways

UAVs, an excellent alternative to crewed aircraft with advantages in size variety and movement agility, have begun to play a significant role in enabling IoT systems to be utilized as wireless and mobile gateways. However, despite the advantages of deploying these flying units, various challenges must be solved to maintain an efficient operation. One of UAVs’ primary obstacles, restricting their use and full adoption in smart cities, is their energy-related issues. Due to limited battery energy, UAVs are constantly jeopardized by mission failure, potentially destroying these devices. As a result, an efficient scheduling framework must be built to create an activity plan that ensures successful and safe UAV mission execution. The activity plan typically consists of a plan that retains the locations and the order of the missions to be covered by the UAVs, as well as the duration to reach the charging station, taking into account the features of the UAVs, such as the limited battery capacity and the mission’s priorities.

Over the past few years, many relevant and recent contributions have been made to research projects concerning UAV scheduling. In [40], the authors solved a mixed-integer linear programming (MILP) problem using an approximation algorithm and a fast heuristics-based solution to address the path-planning problem for a single UAV, making sure that the flying unit covered each target at least once while taking into consideration its limited battery. In [41], the authors tackled an MILP problem by introducing a generic scheduling strategy to handle a fleet of UAVs to cover temporally and spatially dispersed events. In [42], the work presented a study about flight scheduling and trajectory control for UAV-based wireless networks. With an interest in optimizing the UAVs’ flight time, trajectory, and power consumption, the authors proposed a scheduling algorithm for UAVs serving a set of ground users. Finally, in [43], the authors introduced the application of model-based reinforcement learning (RL) to empower UAVs with the capability to find a route home when their battery life was constrained.

The majority of the solutions, however, are either based on MILP or metaheuristic solutions, such as in [40,41], which are computationally demanding. Furthermore, a limited number of papers have addressed every aspect of the UAV scheduling problem. In other words, each solution is tailored to a given application, and some critical variables, such as time duration of events, unexpected events, and event prioritization, are disregarded when modeling the route-scheduling problem. As a result, in [44], a general spatiotemporal scheduling technique for autonomous UAVs based on an RL algorithm, was developed. The framework provided adequate intelligence to the UAV to autonomously create an activity plan that achieved the maximum number of missions scattered across a region, with duration and priority ranks, while taking the resource-constrained nature of the flying units into account. Furthermore, the suggested technique let the UAV change its prescheduled plan for unexpected activities while in operation.

The activity plan pursued by the UAV, denoted as $d1$, is shown in Figure 4. The goal was to design an activity schedule that permitted the UAV to serve as many events as possible while considering its energy constraints, such as the requirement to recharge the battery. Given these limits and the flight period required to travel from one event to the next, the UAV covered about 28% of the total events. The UAV learned to cover missions while staying within the battery’s limitations. Furthermore, a chronological division was employed in this framework (horizontal dotted red lines). The UAV created the activity schedule for the following period towards the conclusion of each session. Such an approach allowed the UAV to change its path in the event of an unforeseen incident. When an unexpected incident was set as a higher-priority event, as illustrated in Figure 5, the UAV autonomously departed its current mission midway to cover the higher-priority event, which is colored in green. The results demonstrated that UAVs could make in-flight scheduling decisions using adaptive RL models to define their activity plans and cover numerous predicted and unexpected occurrences.

3.3. Urban Navigation for UAVs

As previously discussed, UAVs are “game-changing” devices that have changed the view of everyday activities, let alone smart city activities in general. UAVs are easy to deploy, and thanks to their versatility and flexibility, these flying IoT devices are able to perform challenging and remote tasks, offering a bird’s-eye view. However, UAVs’ path planning is considered one of the major concerns that must be addressed to improve their navigation, particularly in urban areas. A critical point to take into consideration while deploying these flying units in urban environments is that they are vulnerable and exposed to a considerable risk of collisions with environmental obstacles. The routing and navigation of UAVs are therefore indispensable in order to significantly improve the operation of UAVs, decrease risks, and increase efficiency.

In [45], the authors proposed a deep RL-based solution to solve the problem of UAV navigation in a large-scale obstacle-constrained environment using GPS signals and sensory information. In [46], the authors proposed a goal-oriented routing algorithm based on Q-learning to enable UAVs to execute tasks in a given mission area while navigating obstacles. However, the presented models used a simplified 2D navigation model, where the UAV lost one of its most significant key points, the third dimension. Hence, the UAVs were unable to change their altitude to cross over obstacles.

In [47], the authors presented a Q-learning-based approach to address the autonomous scheduling problem of UAVs. Q-learning was also employed to establish obstacle-aware navigation in [48]. However, the authors utilized discrete actions (i.e., the system was represented as a grid world with a reduced UAV action space), which could greatly affect the performance of these flying units when dealing with real-world situations.

Therefore, in [49], we proposed a UAV path-planning framework using the deep RL approach. The framework enabled the UAV to autonomously navigate obstacles in order to reach spatially scattered moving or static targets. Unlike other works, the UAV was trained in a three-dimensional (3D) environment with a high matching degree to the real world, using a continuous action space.

Figure 6 depicts instances of UAV trajectories, shown as red dots, used to achieve its target utilizing the autonomous navigation system. In Figure 6a, the UAV successfully arrived at its target while avoiding the obstacles depicted by a gray polygon with varying heights. The UAV ringed the barrier, obstacle 2 ($obs2$), on its route to the destination since it could not fly above the building because the maximum height and flying altitude were both set to their highest (i.e., normalized value of one). In the scenario illustrated in Figure 6b, the UAV passed over obstacle 2 $obs2$ to reach the destination quickly.

Optimizing the navigation of UAVs in urban environments helps reduce propulsion energy consumption, which constitutes the main energy-hungry component in UAVs. Therefore, the main challenges in designing RL-based navigation paths are to devise a reward function that guarantees the fastest navigation toward the destination while avoiding collision with existing obstacles, i.e., static and mobile obstacles. Additionally, UAV motion control aspects must be considered when designing RL for real-world UAV navigation. This should consider the physical limits on velocity and acceleration and the requirements for main stability and safety. Finally, robust RL models need to be designed to cope with novel or unexpected system perturbation. Hence, the RL model needs to be sensitive to changes in the environment (i.e., minor changes such as moving obstacles and wind effects) as well as the dynamics of the UAVs (e.g., different speed profiles and acceleration modes).

3.4. Spatiotemporal UAV Scheduling

In many applications, the separate design of the optimized scheduling and the navigation of UAVs is insufficient to achieve the desired objectives. In many cases, the joint consideration of all the system’s parameters and inputs is mandatory to achieve effective UAV operation in urban environments. As an example, the flying time between two events can dynamically change, unlike in the open-space case where only direct routes are considered. In urban environments, the scheduling of UAVs depends on the possible nondirect trajectories that can be deduced from the 3D environments. Moreover, when a fleet of swarms is operated, coordination among UAVs needs to be ensured such that redundant use of UAVs is avoided. For instance, UAVs should not operate on the same event at the same time. The scheduler needs to guarantee that UAVs can substitute for each other to cover an event. The coordination needs to enclose the battery status of each flying unit so as to guarantee their safe operation and avoid crashes.

In [41], developed a mixed-integer linear program (MILP) that provided optimal spatiotemporal scheduling for UAVs. Thus, each UAV with a specified time window had to finish multiple missions, such as monitoring distinct areas for different periods and acting as temporary RSUs at various intersections. The UAV fleet operator determined the missions’ locations, start times, and duration. The MILP’s objective was to generate a precise activity plan for the next period (hours or days) so that all missions were met with the least quantity of resources, i.e., the least energy and number of units utilized. The UAVs began their missions at a charging station, to which they could return at any moment to recharge their batteries. To avoid the simultaneous execution of the same tasks by two or more UAVs, parallel and collision-free coverage was guaranteed. Furthermore, the architecture allowed for the replacement of one UAV with its counterparts. As a result, if one UAV’s energy was depleted, another UAV would take its position without interfering with the mission goals.

For tractability and clarity, Figure 7 illustrates a simplified scheduling scenario using two UAVs aiming at completing four missions. Two missions were selected to be relatively long such that the UAV battery, even full, would not be enough for a UAV to complete the task alone. Two charging stations were available in the area. The MILP was solved using off-the-shelf software to find the optimal activity plan for each flying unit [41]. The long missions 2 and 3 required both UAVs to cover them consecutively. UAV 2, the closest to mission 3, started the mission, allowing UAV 1 to reload its battery to pursue the task sufficiently. UAV 1, which initially took off from charging station 1, landed on charging station 2 to continue covering mission 4. A similar remark can be made for UAV 1. Optimized scheduling allowed a better fleet management by using the minimum resources.

4. Perspectives and Future Research Directions

The various studies discussed earlier provide an initial set of a full suite of methods to enable dynamic aerial IoT infrastructures for smart cities and ITS. With innovative methods for placing UAV charging stations, scheduling missions, routing data, and minimizing energy consumption across all, smart city and ITS planners have a strong basis for a toolkit that will aid in the design of temporary flying IoT devices that will help support ground-based infrastructure. Although a considerable amount of efforts have been made to mitigate some of the energy-related challenges that apply to UAVs for ITS, various concerns remain open research problems when making a UAV-dominant system adequately robust to be implemented in real-world applications.

4.1. Front-End Intelligent Drones

Supporting UAVs by embedded processors will make them viable contributors to edge-computing platforms. In order to maximize the use of the uptime of UAVs on scheduled missions, distributed computing jobs could be assigned to the extra cache space on the onboard microprocessors. This allocation of jobs would reduce the quantity of data transmitted to the central control system for trip planning and other jobs, reducing the transmission traffic on the spectrum. In another direction, cooperative processing needs to be leveraged to promote the independence of UAVs from the central stations and hence, increase their autonomy. In addition to the distributed processing discussed earlier, cooperative processing procedures, e.g., based on federated learning, can be devised. Multiple UAVs can cooperate together in collecting data, sharing the data, or sharing the AI models processing the data. Hence, UAVs can improve the accuracy of their decision-making, increase the coverage of their operation, and make the aerial infrastructure more scalable and not limited to a single unit.

4.2. Autonomous UAVs and Collision-Free Navigation

One way to make the system more robust is to add a degree of autonomy to the operations of UAVs, either when operating individually or in a group. For example, in the case of a connection loss to the central control, or for UAV missions involving traveling far distances, UAVs must be capable of autonomously planning their trip and returning home. Future research can focus on computer vision applications to utilize onboard cameras to mitigate collisions and accomplish their mission (e.g., find sensors). UAVs could also be trained on data generated on how a human operator would navigate obstacles or avoid collisions. In an online-learning pipeline, the human operators would act as the training data inputs in a controlled scenario. Over time, the UAV could be tested in controlled scenarios, or multiple UAV-control algorithms could be pitted together in an agent-based simulation to see how they behave around each other.

Machine-learning UAV autonomy is only one piece of the puzzle to developing anticollision frameworks for swarms. Complex navigation rules based on existing flight control systems for human air traffic can be put in place on UAVs. Agent-based simulation can be utilized to test the hard-coded rules in the UAV to see how decisions made by individual UAVs propagate in a swarm, assuming a movement based on the optimization frameworks discussed in the previous section. Researchers should also consider joint frameworks considering a multimodal concert of UAVs and ground transmitters working in harmony to take advantage of the strengths of both to mask their weaknesses. Efforts should also focus on fast solution methods; as the size of the UAV fleet grows, the complexity of scheduling problems will grow exponentially due to the NP-hard computational complexity of MIP scheduling problems.

4.3. Security and Privacy

With the growing concern about the privacy and security of data communications, these issues should be of the utmost priority due to the higher risks of employing autonomous UAVs in critical applications. In addition to developing and strengthening encryption transmission protocols, extending physical layer security techniques for UAV communication should be investigated. In all, “secure by default” approaches must be adopted to build networks from the ground up, focusing on security instead of adding security layers as issues arise. Beyond the security and privacy of actual transmissions, privacy concerns regarding the navigation of UAVs must be considered when devising their mission scheduling frameworks, especially for use in residential areas. UAVs need to mitigate security attacks and ensure the smooth and safe operation of the fleet. Anti-intrusion UAV systems for communication networks, as well as for detecting unwanted devices, need to be developed to address the security and privacy challenges of these vulnerable aerial infrastructures.

4.4. Noise and Environmental Considerations

In addition to electromagnetic signals resulting from their data communication, multirotor UAVs contribute a non-negligible amount of noise pollution from the action of their propeller blades. Moreover, these effects should be factored in with the growing number of UAVs expected to be deployed and the large amounts of data traffic to be exchanged with 5G and 6G. Specifically, two fundamental research directions can be identified: (1) a technical analysis of noise propagation models to predict the noise levels as a function of the number of drones in the area and their operation parameters, e.g., flying altitudes; (2) the combined effects of multiple electromagnetic radiations with high-energy millimeter-wave spectrum.

4.5. Multitasking UAVs

Thanks to their flexibility, UAVs can execute multiple missions in parallel especially when they are equipped with different types of sensors, such as camera, LiDAR, and GNSS. The UAVs operating essentially for ITS missions can be exploited for secondary missions. For example, while flying between two events, or while hovering in a certain location. UAVs can be exploited for other public service activities: they can operate in law enforcement, damaged infrastructure monitoring, and air pollution measurement. Advanced autonomous operation of multitasking UAVs need to be investigated and devised while taking into account the objectives of the different missions, their priorities, and the imposed restrictions such as time limit, no-drone zones, and energy budget.

5. Conclusions

In this article, the potential of UAVs in leveraging smart cities and ITS was investigated. First, a high-level overview of promising applications of UAVs in smart cities was provided. Next, a set of solutions to enable the effective deployment of UAVs and their smart operation to act as an aerial infrastructure for the applications of ITS was discussed. It was shown that many challenges need to be addressed to improve the operation of UAVs in an urban environment: including charging station planning, autonomous navigation and mission scheduling, and fleet coordination. The paper also highlighted some potential future research directions to help usher in the benefits of UAVs in future smart cities.

Author Contributions

Conceptualization, H.G., M.C.L. and O.B.; methodology, H.G., M.C.L. and O.B.; software, O.B. and H.G.; validation, H.G., M.C.L., O.B. and A.K.; formal analysis, H.G., M.C.L. and O.B.; investigation, H.G., M.C.L. and O.B.; resources, Y.M.; data curation, H.G., M.C.L. and O.B.; writing—original draft preparation, Y.M., H.G., M.C.L., O.B. and A.K.; writing—review and editing, H.G. and A.K.; visualization, M.C.L., O.B. and A.K.; supervision, H.G. and Y.M.; project administration, Y.M.; funding acquisition, Y.M. and A.K. All authors have read and agreed to the published version of the manuscript.

Conflicts of Interest

The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Table

Figure 1. Highlighting diverse UAV applications for smart cities. An increase on the x-axis indicates higher-bandwidth applications, and an increase in the y-axis represents a higher delay tolerance. The size of the circles reflects the size of the UAV swarm needed to provide robust service.

Figure 2. A part of Hoboken, New Jersey, USA, is shown on the map above, which displays the coverage by the integrated RSU/UAV infrastructure during peak hours. The light gray highlights denote the RSU coverage (red inverted triangles), and the dark gray areas convey the UAV base station coverage (black triangles). All other points’ colors signify their “fitness”, a standardized utility value, with a value of one being a pivotal point to cover; otherwise, a zero value indicates an insignificant point.

Figure 3. Coverage efficiency vs. maximum number of candidate locations for the exact approach (blue line with “x” markers) along with various heuristic increment values; a larger increment value corresponds to a faster convergence at the cost of a lower solution granularity.

Figure 4. Illustration of the maximization of event coverage efficiency with an RL algorithm (i.e., Q-learning model) to optimize a UAV’s schedule. (a) Autonomous restricted battery scheduling (4000 mAh); the blue bottom line represents the charging station event available along the operation time; the other blue lines on the red peaks represent the event time duration. (b) UAV battery fluctuations during task execution. The solid blue lines represent the UAV battery capacity boundaries, while the blue dashed line denotes the maximum energy required in each period to cover the most distant reachable event. The latter energy depends on the flight distance separating the charging station and the farthest event occurring during that period.

Figure 5. (a) Autonomous scheduling of a restricted battery capacity of 4000 mAh in the case of an unforeseen incident, and (b) the evolution of the UAV battery during mission execution. The green line conveys an unexpected event.

Figure 6. The UAV generates autonomous pathways to its destination. The red dots denote the UAV path, the green dots symbolize the target, and the gray boxes represent the obstacles. (a) Environment 1 and (b) environment 2. The height of each obstacle is indicated above it. The z-values indicate the altitude of the UAV at that position.

Figure 7. Spatiotemporal scheduling using two UAVs to accomplish four missions. Two charging stations (DS) are made available in the area of interest. Mission and charging station locations (Left), UAV scheduling (Right); labeled solid lines represent the trajectories taken by the UAVs. Numerical order: UAV 1, alphabetical order: UAV 2.

Table 1

Recent studies addressing smart cities’ challenges using UAVs for ITS.

Reference	Title	Application Domain	Employed Algorithm
[27]	An UAV-assisted VANET architecture for intelligent transportation system in smart cities.	Communication and data exchange	Simulation of ad hoc network architecture
[28]	Caching and computation offloading in high altitude platform station (HAPS) assisted intelligent transportation systems.	Data processing and edge computing	Computation framework for ITS (high-altitude platform station)
[29]	AoI optimization in the UAV-aided traffic monitoring network under attack: A stackelberg game viewpoint.	Traffic monitoring	Game theory: Stackelberg game
[30]	Toward Smart Traffic Management With 3D Placement Optimization in UAV-Assisted NOMA IIoT Networks.	Traffic management	Improved adaptive whale optimization algorithm
[31]	Joint Channel Allocation and Data Delivery for UAV-Assisted Cooperative Transportation Communications in Post-Disaster Networks	Emergency situation	Game theory: Stackelberg game
[32]	Stochastic Task Scheduling in UAV-Based Intelligent On-Demand Meal Delivery System	Goods delivery	Iterated heuristic framework, stochastic event scheduling
[33]	Decentralized multi-agent path finding for UAV traffic management	Traffic management	novel multiagent path finding: (a) prioritization approach and (b) pairwise negotiation approach
[34]	Throughput Maximization for RIS-UAV Relaying Communications	Data transfer	Formulate nonconvex optimization problem with three subproblems: (a) passive beamforming optimization, (b) trajectory optimization, and (c) power allocation optimization
[35]	FRCNN-Based Reinforcement Learning for Real-Time Vehicle Detection, Tracking and Geolocation from UAS	Surveillance	Adaptive filtering, top-hat and bottom-hat transformations, Kanade–Lucas–Tomasi trackers, density-based spatial clustering of applications with noise (DBSCAN) clustering, efficient reinforcement connecting algorithm, and fast regional convolutional neural network (Fast-RCNN)

References

1. Al-Fuqaha, A.; Guizani, M.; Mohammadi, M.; Aledhari, M.; Ayyash, M. Internet of Things: A Survey on Enabling Technologies, Protocols, and Applications. IEEE Commun. Surv. Tutor.; 2015; 17, pp. 2347-2376. [DOI: https://dx.doi.org/10.1109/COMST.2015.2444095]

2. Bolla, G.M.; Casagrande, M.; Comazzetto, A.; Dal Moro, R.; Destro, M.; Fantin, E.; Colombatti, G.; Aboudan, A.; Lorenzini, E.C. ARIA: Air Pollutants Monitoring Using UAVs. Proceedings of the 2018 5th IEEE International Workshop on Metrology for AeroSpace (MetroAeroSpace); Rome, Italy, 20–22 June 2018; pp. 225-229. [DOI: https://dx.doi.org/10.1109/MetroAeroSpace.2018.8453584]

3. Jońca, J.; Pawnuk, M.; Bezyk, Y.; Arsen, A.; Sówka, I. Drone-Assisted Monitoring of Atmospheric Pollution—A Comprehensive Review. Sustainability; 2022; 14, 11516. [DOI: https://dx.doi.org/10.3390/su141811516]

4. Mozaffari, M.; Saad, W.; Bennis, M.; Debbah, M. Mobile Unmanned Aerial Vehicles (UAVs) for Energy-Efficient Internet of Things Communications. IEEE Trans. Wirel. Commun.; 2017; 16, pp. 7574-7589. [DOI: https://dx.doi.org/10.1109/TWC.2017.2751045]

5. Zhang, Q.; Jiang, M.; Feng, Z.; Li, W.; Zhang, W.; Pan, M. IoT Enabled UAV: Network Architecture and Routing Algorithm. IEEE Internet Things J.; 2019; 6, pp. 3727-3742. [DOI: https://dx.doi.org/10.1109/JIOT.2018.2890428]

6. Motlagh, N.H.; Bagaa, M.; Taleb, T. UAV-Based IoT Platform: A Crowd Surveillance Use Case. IEEE Commun. Mag.; 2017; 55, pp. 128-134. [DOI: https://dx.doi.org/10.1109/MCOM.2017.1600587CM]

7. Hildmann, H.; Kovacs, E. Using unmanned aerial vehicles (UAVs) as mobile sensing platforms (MSPs) for disaster response, civil security and public safety. Drones; 2019; 3, 59. [DOI: https://dx.doi.org/10.3390/drones3030059]

8. Yang, F.; Sakti, S.; Wu, Y.; Nakamura, S. A framework for knowing who is doing what in aerial surveillance videos. IEEE Access; 2019; 7, pp. 93315-93325. [DOI: https://dx.doi.org/10.1109/ACCESS.2019.2924188]

9. Mohamed, N.; Al-Jaroodi, J.; Jawhar, I.; Idries, A.; Mohammed, F. Unmanned aerial vehicles applications in future smart cities. Technol. Forecast. Soc. Chang.; 2020; 153, 119293. [DOI: https://dx.doi.org/10.1016/j.techfore.2018.05.004]

10. Niglio, F.; Comite, P.; Cannas, A.; Pirri, A.; Tortora, G. Preliminary Clinical Validation of a Drone-Based Delivery System in Urban Scenarios Using a Smart Capsule for Blood. Drones; 2022; 6, 195. [DOI: https://dx.doi.org/10.3390/drones6080195]

11. Molina, P.; Colomina, I.; Victoria, T.; Skaloud, J.; Kornus, W.; Prades, R.; Aguilera, C. Searching lost people with UAVs: The system and results of the close-search project. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.; 2012; 39, pp. 441-446. [DOI: https://dx.doi.org/10.5194/isprsarchives-XXXIX-B1-441-2012]

12. Saleri, R.; Cappellini, V.; Nony, N.; De Luca, L.; Pierrot-Deseilligny, M.; Bardiere, E.; Campi, M. UAV photogrammetry for archaeological survey: The Theaters area of Pompeii. Proceedings of the 2013 Digital heritage international congress (DigitalHeritage); Marseille, France, 28 October–1 November 2013; Volume 2, pp. 497-502.

13. Menouar, H.; Guvenc, I.; Akkaya, K.; Uluagac, A.S.; Kadri, A.; Tuncer, A. UAV-Enabled Intelligent Transportation Systems for the Smart City: Applications and Challenges. IEEE Commun. Mag.; 2017; 55, pp. 22-28. [DOI: https://dx.doi.org/10.1109/MCOM.2017.1600238CM]

14. Caruso, A.; Chessa, S.; Escolar, S.; Barba, J.; López, J.C. Collection of Data with Drones in Precision Agriculture: Analytical Model and LoRa Case Study. IEEE Internet Things J.; 2021; 8, pp. 16692-16704. [DOI: https://dx.doi.org/10.1109/JIOT.2021.3075561]

15. Nevalainen, O.; Honkavaara, E.; Tuominen, S.; Viljanen, N.; Hakala, T.; Yu, X.; Hyyppä, J.; Saari, H.; Pölönen, I.; Imai, N.N. et al. Individual Tree Detection and Classification with UAV-Based Photogrammetric Point Clouds and Hyperspectral Imaging. Remote Sens.; 2017; 9, 185. [DOI: https://dx.doi.org/10.3390/rs9030185]

16. Chen, Z.G.; Zhan, Z.H.; Kwong, S.; Zhang, J. Evolutionary computation for intelligent transportation in smart cities: A survey. IEEE Comput. Intell. Mag.; 2022; 17, pp. 83-102. [DOI: https://dx.doi.org/10.1109/MCI.2022.3155330]

17. Bithas, P.S.; Michailidis, E.T.; Nomikos, N.; Vouyioukas, D.; Kanatas, A.G. A survey on machine-learning techniques for UAV-based communications. Sensors; 2019; 19, 5170. [DOI: https://dx.doi.org/10.3390/s19235170] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31779133]

18. Abualigah, L.; Diabat, A.; Sumari, P.; Gandomi, A.H. Applications, deployments, and integration of internet of drones (iod): A review. IEEE Sens. J.; 2021; 21, pp. 25532-25546. [DOI: https://dx.doi.org/10.1109/JSEN.2021.3114266]

19. Sayeed, M.A.; Kumar, R. An efficient mobility model for improving transmissions in multi-UAVs enabled WSNs. Drones; 2018; 2, 31. [DOI: https://dx.doi.org/10.3390/drones2030031]

20. Gharibi, M.; Boutaba, R.; Waslander, S.L. Internet of Drones. IEEE Access; 2016; 4, pp. 1148-1162. [DOI: https://dx.doi.org/10.1109/ACCESS.2016.2537208]

21. Guetta, Y.; Shapiro, A. On-Board Physical Battery Replacement System and Procedure for Drones During Flight. IEEE Robot. Autom. Lett.; 2022; 7, pp. 9755-9762. [DOI: https://dx.doi.org/10.1109/LRA.2022.3190077]

22. Zeng, Y.; Zhang, R.; Lim, T.J. Wireless communications with unmanned aerial vehicles: Opportunities and challenges. IEEE Commun. Mag.; 2016; 54, pp. 36-42. [DOI: https://dx.doi.org/10.1109/MCOM.2016.7470933]

23. Hellaoui, H.; Bekkouche, O.; Bagaa, M.; Taleb, T. Aerial Control System for Spectrum Efficiency in UAV-to-Cellular Communications. IEEE Commun. Mag.; 2018; 56, pp. 108-113. [DOI: https://dx.doi.org/10.1109/MCOM.2018.1800078]

24. Ghorbel, M.B.; Rodríguez-Duarte, D.; Ghazzai, H.; Hossain, M.J.; Menouar, H. Joint Position and Travel Path Optimization for Energy Efficient Wireless Data Gathering Using Unmanned Aerial Vehicles. IEEE Trans. Veh. Technol.; 2019; 68, pp. 2165-2175. [DOI: https://dx.doi.org/10.1109/TVT.2019.2893374]

25. Ghazzai, H.; Feidi, A.; Menouar, H.; Ammari, M.L. An exploratory search strategy for data routing in flying ad hoc networks. Proceedings of the 2017 IEEE 28th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC); Montreal, QC, Canada, 8–13 October 2017; pp. 1-7. [DOI: https://dx.doi.org/10.1109/PIMRC.2017.8292474]

26. Ghazzai, H.; Menouar, H.; Kadri, A. On the Placement of UAV Docking Stations for Future Intelligent Transportation Systems. Proceedings of the 2017 IEEE 85th Vehicular Technology Conference (VTC Spring); Sydney, Australia, 4–7 June 2017; pp. 1-6. [DOI: https://dx.doi.org/10.1109/VTCSpring.2017.8108676]

27. Raza, A.; Bukhari, S.H.R.; Aadil, F.; Iqbal, Z. An UAV-assisted VANET architecture for intelligent transportation system in smart cities. Int. J. Distrib. Sens. Netw.; 2021; 17, 15501477211031750. [DOI: https://dx.doi.org/10.1177/15501477211031750]

28. Ren, Q.; Abbasi, O.; Kurt, G.K.; Yanikomeroglu, H.; Chen, J. Caching and computation offloading in high altitude platform station (HAPS) assisted intelligent transportation systems. IEEE Trans. Wirel. Commun.; 2022; 21, pp. 9010-9024. [DOI: https://dx.doi.org/10.1109/TWC.2022.3171824]

29. Yang, Y.; Wang, W.; Liu, L.; Dev, K.; Qureshi, N.M.F. AoI optimization in the UAV-aided traffic monitoring network under attack: A stackelberg game viewpoint. IEEE Trans. Intell. Transp. Syst.; 2022; 24, pp. 932-941. [DOI: https://dx.doi.org/10.1109/TITS.2022.3157394]

30. Adam, A.B.; Muthanna, M.S.A.; Muthanna, A.; Nguyen, T.N.; Abd El-Latif, A.A. Toward Smart Traffic Management with 3D Placement Optimization in UAV-Assisted NOMA IIoT Networks. IEEE Trans. Intell. Transp. Syst.; 2022; pp. 1-11. [DOI: https://dx.doi.org/10.1109/TITS.2022.3182651]

31. Dai, M.; Luan, T.H.; Su, Z.; Zhang, N.; Xu, Q.; Li, R. Joint Channel Allocation and Data Delivery for UAV-Assisted Cooperative Transportation Communications in Post-Disaster Networks. IEEE Trans. Intell. Transp. Syst.; 2022; 23, pp. 16676-16689. [DOI: https://dx.doi.org/10.1109/TITS.2022.3178789]

32. Huang, H.; Hu, C.; Zhu, J.; Wu, M.; Malekian, R. Stochastic Task Scheduling in UAV-Based Intelligent On-Demand Meal Delivery System. IEEE Trans. Intell. Transp. Syst.; 2021; 23, pp. 13040-13054. [DOI: https://dx.doi.org/10.1109/TITS.2021.3119343]

33. Ho, F.; Geraldes, R.; Gonçalves, A.; Rigault, B.; Sportich, B.; Kubo, D.; Cavazza, M.; Prendinger, H. Decentralized multi-agent path finding for UAV traffic management. IEEE Trans. Intell. Transp. Syst.; 2020; 23, pp. 997-1008. [DOI: https://dx.doi.org/10.1109/TITS.2020.3019397]

34. Liu, X.; Yu, Y.; Li, F.; Durrani, T.S. Throughput Maximization for RIS-UAV Relaying Communications. IEEE Trans. Intell. Transp. Syst.; 2022; 23, pp. 19569-19574. [DOI: https://dx.doi.org/10.1109/TITS.2022.3161698]

35. Singh, C.H.; Mishra, V.; Jain, K.; Shukla, A.K. FRCNN-Based Reinforcement Learning for Real-Time Vehicle Detection, Tracking and Geolocation from UAS. Drones; 2022; 6, 406. [DOI: https://dx.doi.org/10.3390/drones6120406]

36. Lucic, M.C.; Ghazzai, H.; Massoud, Y. A Generalized Dynamic Planning Framework for Green UAV-Assisted Intelligent Transportation System Infrastructure. IEEE Syst. J.; 2020; 14, pp. 4786-4797. [DOI: https://dx.doi.org/10.1109/JSYST.2020.2969372]

37. Wang, Z.; Zheng, J.; Wu, Y.; Mitton, N. A centrality-based RSU deployment approach for vehicular ad hoc networks. Proceedings of the 2017 IEEE International Conference on Communications (ICC); Paris, France, 21–25 May 2017; pp. 1-5. [DOI: https://dx.doi.org/10.1109/ICC.2017.7996986]

38. Ghazzai, H.; Menouar, H.; Kadri, A.; Massoud, Y. Future UAV-Based ITS: A Comprehensive Scheduling Framework. IEEE Access; 2019; 7, pp. 75678-75695. [DOI: https://dx.doi.org/10.1109/ACCESS.2019.2921269]

39. Balouchzahi, N.M.; Fathy, M.; Akbari, A. Optimal road side units placement model based on binary integer programming for efficient traffic information advertisement and discovery in vehicular environment. IET Intell. Transp. Syst.; 2015; 9, pp. 851-861. [DOI: https://dx.doi.org/10.1049/iet-its.2014.0051]

40. Sundar, K.; Rathinam, S. Algorithms for Routing an Unmanned Aerial Vehicle in the Presence of Refueling Depots. IEEE Trans. Autom. Sci. Eng.; 2014; 11, pp. 287-294. [DOI: https://dx.doi.org/10.1109/TASE.2013.2279544]

41. Ghazzai, H.; Kadri, A.; Ben Ghorbel, M.; Menouar, H.; Massoud, Y. A Generic Spatiotemporal UAV Scheduling Framework for Multi-Event Applications. IEEE Access; 2018; 7, pp. 215-229. [DOI: https://dx.doi.org/10.1109/ACCESS.2018.2885321]

42. Nguyen, M.T.; Le, L.B. Flight Scheduling and Trajectory Control in UAV-Based Wireless Networks. Proceedings of the IEEE Wireless Communications and Networking Conference (WCNC’20); Seoul, Republic of Korea, 25–28 May 2020; pp. 1-6.

43. Imanberdiyev, N.; Fu, C.; Kayacan, E.; Chen, I. Autonomous navigation of UAV by using real-time model-based reinforcement learning. Proceedings of the 14th International Conference on Control, Automation, Robotics and Vision (ICARCV’16); Phuket, Thailand, 13–15 November 2016; pp. 1-6. [DOI: https://dx.doi.org/10.1109/ICARCV.2016.7838739]

44. Bouhamed, O.; Ghazzai, H.; Besbes, H.; Massoud, Y. A Generic Spatiotemporal Scheduling for Autonomous UAVs: A Reinforcement Learning-Based Approach. IEEE Open J. Veh. Technol.; 2020; 1, pp. 93-106. [DOI: https://dx.doi.org/10.1109/OJVT.2020.2979559]

45. Wang, C.; Wang, J.; Zhang, X.; Zhang, X. Autonomous navigation of UAV in large-scale unknown complex environment with deep reinforcement learning. Proceedings of the IEEE Global Conference on Signal and Information Processing (GlobalSIP’17); Montreal, QC, Canada, 14–16 November 2016; pp. 858-862.

46. Bouhamed, O.; Ghazzai, H.; Besbes, H.; Massoud, Y. Q-learning based routing scheduling for a multi-task autonomous agent. Proceedings of the 2019 IEEE 62nd International Midwest Symposium on Circuits and Systems (MWSCAS); Dallas, TX, USA, 4–7 August 2019; pp. 634-637.

47. Yan, C.; Xiang, X. A Path Planning Algorithm for UAV Based on Improved Q-Learning. Proceedings of the 2nd International Conference on Robotics and Automation Sciences (ICRAS’18); Wuhan, China, 23–25 June 2018; [DOI: https://dx.doi.org/10.1109/ICRAS.2018.8443226]

48. Sichkar, V.N. Reinforcement Learning Algorithms in Global Path Planning for Mobile Robot. Proceedings of the International Conference on Industrial Engineering, Applications and Manufacturing (ICIEAM’19); Sochi, Russia, 25–29 March 2019; pp. 1-5. [DOI: https://dx.doi.org/10.1109/ICIEAM.2019.8742915]

49. Bouhamed, O.; Ghazzai, H.; Besbes, H.; Massoud, Y. A UAV-Assisted Data Collection for Wireless Sensor Networks: Autonomous Navigation and Scheduling. IEEE Access; 2020; 8, pp. 110446-110460. [DOI: https://dx.doi.org/10.1109/ACCESS.2020.3002538]