A Federated Reinforcement Learning Framework via a Committee Mechanism for Resource Management in 5G Networks

Abstract

This paper proposes a novel decentralized federated reinforcement learning (DFRL) framework that integrates deep reinforcement learning (DRL) with decentralized federated learning (DFL). The DFRL framework boosts efficient virtual instance scaling in Mobile Edge Computing (MEC) environments for 5G core network automation. It enables multiple MECs to collaboratively optimize resource allocation without centralized data sharing. In this framework, DRL agents in each MEC make local scaling decisions and exchange model parameters with other MECs, rather than sharing raw data. To enhance robustness against malicious server attacks, we employ a committee mechanism that monitors the DFL process and ensures reliable aggregation of local gradients. Extensive simulations were conducted to evaluate the proposed framework, demonstrating its ability to maintain cost-effective resource usage while significantly reducing blocking rates across diverse traffic conditions. Furthermore, the framework demonstrated strong resilience against adversarial MEC nodes, ensuring reliable operation and efficient resource management. These results validate the framework’s effectiveness in adaptive and efficient resource management, particularly in dynamic and varied network scenarios.

1. Introduction

With the advent of 5G and the forthcoming 6G networks, there is a vision to create multipurpose and intelligent networks that accommodate both traditional end users and various industry sectors. Advanced virtualization technologies in the mobile core network facilitate network virtualization by changing network elements and functions from hardware-based systems to software-based containers. This transformation offers the necessary flexibility for dynamic resource allocation. Correspondingly, the Third Generation Partnership Project (3GPP), which standardizes mobile communication systems, has outlined the 5G core framework. This framework includes various network functions based on a service-based architecture, where each element is virtualized and runs within virtual machines (VMs) or containers as virtualized network functions (VNF) [1,2].

In this paradigm shift, we have encountered a rising demand for 5G networks, necessitating more efficient management of computational resources within mobile core networks. In other words, instances of network functions must be precisely scaled to accommodate varying customer demands, by either initiating new instances or terminating them based on system load [3]. For example, during peak times with high customer demand, additional instances should be deployed to meet the needs, whereas during off-peak times, instances should be reduced to conserve resources. However, dynamically adjusting computational resources by predicting future demand based on past patterns remains a challenging task.

Numerous research efforts have been directed towards efficiently scaling virtual instances in the 5G core network [4,5,6]. In practice, Kubernetes, a popular container orchestrator in practice, has become a leading cloud manager in recent years, offering Vertical and Horizontal Pod Autoscaling (VPA, HPA) [7,8]. Default VPA in Kubernetes involves adjusting the resources allocated to containers by terminating and redeploying them with the updated resources. Due to this limitation, it has attracted less interest from researchers. In contrast, Default HPA in Kubernetes involves adjusting the number of replicas of the same pod to distribute the load. The default HPA relies on manually configuring threshold values, including CPU utilization, and setting the minimum and maximum number of pods. But these scaling approaches are not adaptable to varying workloads. To address dynamic service demands, various methods described in Section 2 utilizing each base technology have been proposed. Recently, the use of deep reinforcement learning (DRL) has gained significant attention in tackling this problem since DRL is a great tool for decision-making in complex and dynamic environments. Moreover, it can outperform traditional approaches when a (near)-optimal solution is unavailable via a data-driven approach. Hence, such DRL-based approaches can adaptively manage core network resources according to customer demand, addressing both service requirements and resource optimization [9]. Moreover, the integration of the Network Data Analytics Function (NWDAF), which is standardized in 3GPP, and DRL presents a promising avenue for enhancing network optimization and management where the NWDAF, a crucial component of 5G networks, excels in collecting and analyzing data about network performance, state, and user experience [10,11].

To the best of our knowledge, most DRL-based virtual instance scaling in the 5G core network relies on centralized learning. In this approach, state-related information from various network functions in Multi-access Edge Computing (MEC) servers is gathered in a centralized server to train the agent. MEC is a crucial component of 5G and 6G networks, bringing computing capabilities closer to end-users, thereby reducing latency and improving service quality. However, this centralized DRL architecture raises privacy concerns due to the presence of sensitive information. This issue becomes more serious when MEC servers are operated by different service providers, supporting multi-tenants. Conversely, if the DRL agent relies on local MEC server information, there is difficulty in applying trained models to different regions with varying customer demand patterns, resulting in different statistical distributions of input data [12].

To address the aforementioned problem, we suggest using decentralized federated learning (DFL) instead of the traditional DRL-based virtual instance scaling in a centralized manner. DFL enables decentralized training of agents by allowing each agent in MEC servers to train their models locally, without sending data to a central server, thereby preserving data privacy. Therefore, this paper proposes a decentralized federated reinforcement learning (DFRL) framework that leverages the combination of DRL and DFL to address these challenges effectively, aiming for adaptive and efficient virtual instance scaling within MEC environments for 5G core network automation. Moreover, since existing DFL frameworks are vulnerable to Byzantine clients and malicious server attacks [13]—where Byzantine clients may upload incorrect gradients, negatively impacting the global model’s performance and potentially causing training failure—we incorporate a committee mechanism into our proposed DFRL framework. This mechanism addresses Byzantine attacks on MEC servers. A subset of participating MEC servers is appointed as committee members, responsible for monitoring the overall DFL process and ensuring the reliable aggregation of local gradients.

In summary, our proposed framework combines DRL and DFL to address the challenges of resource management in MEC environments while ensuring robust data privacy and protection against Byzantine attacks. Our contributions include developing algorithms for effective DRL training within a DFL paradigm, designing a system architecture to support the DFRL framework with the committee mechanism considering 3GPP standardization, and demonstrating the framework’s efficacy through comprehensive simulations. The detailed contributions of this paper are summarized as follows:

• We explore the effectiveness of NWDAF for gathering states essential for the DRL scheme. Using NWDAF, situated within the 5G core network, can achieve an essential goal in data analysis and long-term management. To design a virtual UPF instances scaling module, we should monitor all the information of the core network such as the number of instances, for better scaling decision [3,14]. AI-based solutions addressing data-centric challenges should be rooted in NWDAF.

• We propose an innovative DFRL framework that combines DRL with DFL to efficiently and adaptively manage UPF instance scaling in MEC environments for 5G core networks. In the proposed framework, DRL agents in each MEC make scaling decisions locally and share model parameters with other MEC servers, rather than sharing their raw data. We utilize the FedAvg [15] algorithm to aggregate their models.

• Despite the strengths of DFL and DRL, inherent limitations exist, particularly concerning data privacy. To tackle potential attacks from malicious MECs and aggressive request distributions, we adopt the incorporation of committee-based mechanisms as a defense strategy. It treats the MEC core network as a node, and in each epoch in the DFL process, some nodes turn into committee members and they are responsible for monitoring the entire DFL training process and ensuring the reliable aggregation of the local gradients.

• We evaluated the proposed framework to determine its effectiveness in maintaining scaling performance under various types of arrival rates. The results indicated that the trained DFRL agent could quickly adjust the number of pods and reduce the blocking rate even when faced with the arrival rates of the evaluation data distribution. Additionally, we conducted experiments by providing the agent with arrival rate data that follow a completely new distribution, different from the traditional Poisson distribution. Moreover, the agent was adjusted to prevent excessive pod scaling in the MEC core network. These findings underscore the adaptability and robustness of the proposed framework in dynamic and varied network conditions.

2. Related Work

In the context of 5G and 6G networks, the autoscaling of virtual instances is crucial for efficient resource management. Based on [16], we categorize application scaling theories into non-RL-based theories and RL- and DRL-based theories.

2.1. Non-Reinforcement Learning Based Scaling Methods

Threshold-based Autoscaling. Threshold-based autoscaling involves setting predefined thresholds for resource utilization metrics such as CPU, memory, and network usage. This method is straightforward to implement and widely used in cloud environments. F. Paraiso et al. [17] presented a component-based Platform as a Service, soCloud, which manages portability, elasticity, provisioning, and high availability. C. Kan [18] proposed DoCloud, which automatically controls Docker container to scale a Web application’s resources. L. Baresi et al. [19] presents a reactive autoscaling policy which consists of a gray-box discrete-time feedback controller to allow containerized applications to scale their resources. However, it has several limitations. First, predefined thresholds may not adapt well to varying workloads, leading to suboptimal scaling decisions. Additionally, to avoid performance degradation, thresholds are often set conservatively, which can result in over-provisioning and increased costs. Furthermore, this method lacks the flexibility to handle complex and dynamic workload patterns effectively.

Queueing Theory-based Autoscaling. Queueing theory-based autoscaling applies queueing models to predict resource demands and manage workloads, helping in efficient resource distribution based on expected traffic patterns. Q. Zhang et al. [20] proposed a regression-based approximation of the CPU request and applied it in the queueing network. A. Ali-Eldin et al. [21] concentrated on horizontal elasticity, the power to add or remove resources allocated to a service in the cloud. They proposed two adaptive controllers based on queueing theory, which predict the future request rate of a service. E. B. Lakew et al. [22] focus on their performance models which forecast response latency. Queueing models rely on assumptions about traffic patterns that may not always hold true, leading to inaccuracies. Developing accurate queueing models for complex systems can be challenging and computationally expensive. Additionally, as the system grows, maintaining and updating the queueing models can become increasingly difficult.

Time Series Analysis-based Autoscaling. Time series analysis-based autoscaling uses historical data to predict future resource requirements and adjust resources proactively. This method can effectively prevent performance degradation by anticipating demand. P.D. Kaur et al. [23] presented a Quality of Service (QoS)-Aware Resource Elasticity (QRE) framework for dynamically scaling cloud resources to host application components while considering user, application, and system resource heterogeneity and ensuring compliance with user-agreed QoS criteria, with effectiveness demonstrated through experiments on Amazon EC2. H. Fernandez et al. [24] presented an advanced autoscaling system that addresses the limitations of current provisioning systems by leveraging heterogeneous resources and multiple QoS levels, selecting scaling plans based on workload and customer requirements, and demonstrating significant reductions in QoS violations in experiments on public and private infrastructures. E. Kassela et al. [25] propose extending TIRAMOLA, a cloud-enabled framework for the automated resizing of NoSQL clusters, to identify and adapt to different workload types using user-defined policies, enhancing its accuracy in scaling decisions and demonstrating its effectiveness through extensive evaluation on an HBase cluster. However, its effectiveness is highly dependent on the availability of accurate and extensive historical data. Accurate predictions require a large amount of historical data, which may not always be available. Furthermore, time series models may not adapt quickly to sudden and unexpected changes in workload patterns, reducing their reliability in highly dynamic environments. Continuous data collection and analysis also introduce significant computational overhead.

2.2. Reinforcement Learning-Based Scaling Methods

Recently, various machine learning techniques, especially RL and DRL, were considered for scaling application resources. RL and DRL methods use machine learning algorithms to learn optimal scaling policies through interaction with the environment, adapting to changing conditions dynamically. These methods have the potential to provide highly adaptive and efficient scaling solutions. J. Rao et al. [26] proposed VCONF, an RL-based approach for automating VM configuration, addressing the challenges of dynamic resource reconfiguration due to changing application demands and resource supply. X. Dutreilh et al. [27] proposed an approach to improve resource dimensioning in cloud computing by integrating RL with techniques for better initialization, convergence speedup, and performance model change detection, and demonstrated its implementation as an automated workflow in a real cloud controller to balance performance and cost-effectiveness. E. Barrett et al. [28] proposed using a temporal difference RL algorithm, specifically Q-learning, within a decision–theoretic framework such as Markov decision processes, to determine the optimal scaling policies for dynamically scaling applications on IaaS clouds, addressing performance interference and resource allocation challenges, and introduce a novel parallel Q-learning approach to mitigate the curse of dimensionality and reduce learning time. L. Schuler et al. [29] propose an RL approach to optimize request-based auto-scaling in serverless computing, demonstrating improved performance over default configurations by learning effective scaling policies for varying workloads. A.A Khaleq et al. [30] introduce an intelligent autoscaling system for microservices in the cloud using an RL technique, which dynamically identifies autoscaling thresholds and enhances response times while maintaining QoS constraints with minimal user intervention. W.F. Villota-Jacome et al. [31] propose SARA and DSARA, RL- and DRL-based mechanisms for intelligent admission control and resource allocation in 5G core network slicing, optimizing resource utilization and service provider profit while meeting the QoS requirements of diverse 5G use cases.

However, they also present considerable challenges. RL and DRL models require extensive training, which can be time-consuming and computationally expensive. These models may take a long time to converge to optimal solutions, especially in complex environments. Moreover, RL and DRL systems can be unstable if not properly tuned, potentially leading to erratic scaling decisions. Nevertheless, traditional centralized DRL approaches require the aggregation of vast amounts of data, including sensitive information from various network functions, raising significant privacy concerns. To address this, we employ DFL over MEC, allowing agents to train collaboratively without compromising privacy. Additionally, to enhance robustness during the DFL process, we implement a committee mechanism.

3. Proposed Scaling Method

In this section, we describe our proposed framework that enables MEC in various states to adaptively scale UPF instances using RL. Specifically, adhering to the 3GPP standardization, our framework utilizes the NWDAF to collect data and train the model. Each RL agent within the MEC trains the global RL model using an FL process without a centralized server, which we call decentralized federated reinforcement learning (DFRL). Additionally, to address privacy concerns, particularly regarding potential malicious attacks, the proposed framework incorporates a committee-based defense mechanism.

3.1. Overall Architecture

Figure 1 describes the overall scaling method of this architecture where various MECs are deployed to service a 5G network. As following the 3GPP standard of 5G explained in [3], the User Equipment (UE) connects to the Data Network (DN) in a 5G system via a Protocol Data Unit (PDU) session. This process involves the UE first connecting to a gNB in the Radio Access Network (RAN), which then routes the connection through the transport network to the 5G core, eventually reaching the DN. The 5G core comprises several network functions following a service-based architecture. Key functions include the Access and Mobility Management Function (AMF), which handles UE authentication and access control, the Session Management Function (SMF), which manages PDU session states, and the UPF, which is responsible for data forwarding and handling QoS, among other tasks. The control plane is managed by AMF and SMF, while the UPF operates in the user plane, ensuring efficient data routing and processing. In this paper, we call the UPF instance a UPF pod, or simply pod. If there are too few UPF pods, the QoS deteriorates due to insufficient handling capacity for new PDU sessions. Conversely, too many idle UPF pods waste resources and increase operational costs. Thus, the primary objective of scaling UPF pods is to optimize resource usage by adjusting the number of pods based on PDU session demands. Correspondingly, the system aims to balance between maintaining QoS and minimizing resource costs by dynamically scaling the number of UPF pods. Constraints include minimum and maximum limits on the number of UPF pods and the maximum number of sessions a single pod can handle. We assume that UPF pods can initialize immediately, which means the booting time converges to 0.

To efficiently design scaling policies through interaction with the environment and adapt to changing dynamic conditions, we utilize DRL-based autoscaling, as described in [3]. Our architecture, compliant with 3GPP standards, employs the NWDAF to collect UE communication data from UE instances as state information, which are then used to train the RL agent where Data Management Function (DMF), Analytics Function (AnLF), Policy Management Function (PMF) modules in NWDAF are considered. To leverage diverse state information without compromising privacy on MEC servers, we also implement a decentralized federated learning (DFL) process. This allows for the collaborative training of DRL agents across MEC servers without relying on a centralized aggregation server. Additionally, to ensure the reliability of MEC and prevent potential malicious attacks from Byzantine servers, we propose a committee mechanism to select trustworthy MECs.

3.2. Decentralized Federated Reinforcement Learning (DFRL) Framework

Here, we assume that there are K MEC servers, denoted as a set $\mathbf{M}=\{m_1,m_2,\cdots m_K\}$ with the core network with multiple UPF instances which connect UE and DN. The list of notations is in List of Notations. For each type of PDU session, we assume that a minimum of $D^{\min }$ pods are initiated, and up to $D^{\max }$ pods can be started. Each pod can manage a maximum of $L^{\mathrm{sess}}$ sessions simultaneously. Let $d_{k,t}^{\mathrm{on}}$ represent the number of active pods in MEC k at round t. Thus, it holds that $D^{\min }\le d_{k,t}^{\mathrm{on}}\le D^{\max }$, and the system can handle up to $D^{\max }{L}^{\mathrm{sess}}$ sessions. Let $l_{k,t}^{\mathrm{sess}}$ denote the number of activating sessions in the system at time round t. Therefore, the system can hold $0\le l_{k,t}^{\mathrm{sess}}\le D^{\max }{L}^{\mathrm{sess}}$ sessions. The following steps summarize the DFRL training process of our proposed system architecture:

1. Initialization: Each MEC server initializes their hyperparameters of a DRL agent randomly and resets their environment as in [3]. Then, the system randomly selects the committee MEC set ${\mathbf{S}}_0^c$ using the proposed committee mechanism, which will be explained later. Other MEC servers become the training MEC set ${\mathbf{S}}_0^b$. Before implementing a DRL algorithm, it is necessary to represent the issue as a Markov decision process (MDP) problem. This involves defining the state space $\mathcal{S}$, the action space $\mathcal{A}$, and the reward function $r:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to \mathbb{R}$. Additionally, the complete MDP definition requires the state transition probability $\mathsf{p}:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to [0,1]$ and the discount factor $\gamma \in [0,1]$. We use a model-free reinforcement learning method. In the MDP framework, an agent interacts with the environment defined by the MDP. At each decision round t, it observes the state $s_t\in \mathcal{S}$ and, according to its policy ${\pi }_t:\mathcal{S}\to \mathcal{A}$, takes an action $a_t\in \mathcal{A}$. As a result, the agent receives a reward $r_t$ and observes the next state at the subsequent round. The state at round t should include all necessary information for optimal scaling decisions. In our case

   (1)$s_{k,t}=\{d_{k,t}^{\mathrm{on}},l_{k,t}^{\mathrm{sess}},{\widehat{\lambda }}_{k,t}\}$

   where ${\widehat{\lambda }}_{k,t}$ represents the measured arrival rate of MEC k since the previous decision at round t. This information is collected by DMF in NWDAF [32,33]. The action space includes three possible actions: start a new pod, terminate an existing pod, or take no action. The agent may start new pods if there is available capacity in the cluster, i.e., $d_{k,t}^{\mathrm{on}}<D^{\max }$, and the agent may only terminate pods if $d_{k,t}^{\mathrm{on}}>D^{\min }$. Termination is considered graceful, meaning the pod waits for all its PDU sessions to close before shutting down and will not accept new sessions during this time. Each agent selects an action through its stochastic policy ${\pi }_t$ in order to maximize their cumulative reward:

   (2)$a_{k,t}\sim {\pi }_{k,t}\left(a_{k,t}\right|s_{k,t})\mathrm{where}{a}_{k,t}\in \mathcal{A}=\{‘\mathrm{pod}\mathrm{start}’,‘\mathrm{pod}\mathrm{terminate}’,‘\mathrm{no}\mathrm{action}’\}.$

   The reward function is defined as follows:

   (3)$r_{k,t}=\left\{\begin{array}{cc}-\kappa {\widehat{b}}_{k,t}\hfill & \mathrm{if}{\widehat{b}}_{k,t}>b_{th}\hfill \\ -d_{k,t-1}^{\mathrm{on}}\hfill & \mathrm{if}{\widehat{b}}_{k,t}\le b_{th}\hfill \end{array}\right..$

   where ${\widehat{b}}_{k,t}$ is the measured blocking rate since the last decision in MEC k at round t and $b_{th}$ is the blocking rate threshold as defined by the QoS level, and $\kappa$ is reward multiplier variable. In the proposed reward structure, ${\widehat{b}}_{k,t}$ represents the UE request processing performance, which can be considered as the QoS of the MEC, while $d_{k,t-1}^{\mathrm{on}}$ denotes the number of currently active pods, which is proportional to the operational cost of the MEC. At each round t, the goal of each RL agent is to maximize its reward. If ${\widehat{b}}_{k,t}$ exceeds the threshold due to an excessive number of requests (the first case of the reward function), the agent receives a significant negative reward proportional to the value of ${\widehat{b}}_{k,t}$. This penalty encourages the agent to reduce the blocking rate by providing additional pods, thereby maintaining the desired level of QoS. In contrast, if ${\widehat{b}}_{k,t}$ remains below the threshold (the second case of the reward function), the agent receives a significant negative reward based on the value of $d_{k,t-1}^{\mathrm{on}}$. This penalty motivates the agent to minimize the number of unnecessary active pods, thus avoiding resource waste and minimizing the operational cost of the MEC. An LF module in NWDAF [32,33] analyzes UE communication data from the DMF module and refines them as ’state-action-reward’ data for training RL agent. The scalar coefficient $\kappa$ scales the blocking rate to align it numerically with the $d_{k,t}^{\mathrm{on}}$ value. The logic behind this reward function is to minimize the blocking rate if it exceeds the threshold and minimize the number of pods if it does not. We set $\kappa =15$ with many trial-and-error and experiments.

2. Local RL Agent Training and Update: In each round, each MEC trains its local RL agent with its UE communication data collected from DMF in NWDAF to maximize the long-term cumulated reward $V^{\pi }\left(s\right)$ with state s, defined as

   (4)$V^{\pi }\left(s\right)={\mathbb{E}}_{\pi }\left[\sum _{i=0}^{\infty }{\gamma }^i{r}_{t+i}|s_t=s\right].$

   The optimal policy ${\pi }^{*}$ is independent of the starting state.

   After the round, each MEC adjusts its scaling strategy with its own RL model by utilizing PMF in NWDAF [32,33]. The PPO algorithm is an actor–critic algorithm [34]. It employs a parameterized policy $\pi (s,\theta )$ as the actor to choose actions, where $\theta$ represents the parameter vector. Additionally, the algorithm approximates the value function with $V(s,\omega )$, which is parameterized by the $\omega$ vector. This value function is utilized to compute the advantage $A^{GAE}$ using the Generalized Advantage Estimation (GAE) method [35]. In PPO, the actor and critic networks are optimized with distinct objectives to achieve stable and efficient learning. Specifically, the actor network is responsible for learning the policy, which determines the actions the agent should take in different states. The purpose of training the actor network is to improve the agent’s decision-making process by gradually adjusting the policy to maximize expected rewards. In this process, PPO uses a clipping mechanism [36] to ensure that policy updates remain within a controlled range, preventing overly large changes that could destabilize learning. On the other hand, the critic network estimates the value function, which predicts the expected cumulative reward for a given state. The critic’s objective is to minimize the prediction error between the estimated value and the actual rewards received, improving the accuracy of the value function over time. By optimizing both the actor and critic together, PPO ensures that the agent not only improves its action selection (through the actor) but also gains a better understanding of the long-term outcomes of its actions (through the critic). This dual optimization enables the agent to learn stable and effective policies while balancing exploration and exploitation in dynamic environments. Although the values in the state are practically constrained by the maximum number of pods $D^{\max }$ and the maximum arrival rate ${\lambda }^{\max }$, the state space can become so large that it is impractical to store the policy or value function in a computer’s memory. To address this, a neural network with 1 hidden layer consisting of 50 hidden nodes was used to approximate the policy $\pi$ and the value function V, and is summarized in Figure 2 and Table 1. In this context, $\theta$ and $\omega$ denote the parameter sets of each actor and critic networks, and $\mathbf{w}$ denotes a tuple with those two model parameters $(\theta ,\omega )$.

   During each round, each MEC $m_k$ trains its local neural model as an RL agent using its UE communication data to minimize the local objective function $F_k\left({\mathbf{w}}_k\right)$ with parameter ${\mathbf{w}}_k$, which is defined as follows:

   (5)$F_k\left({\mathbf{w}}_k\right)={\displaystyle \frac{1}{N_k}}\sum _{i=1}^{N_k}f({\mathbf{w}}_k;{\xi }_{k,i}),$

   where $N_k=\left|{\mathcal{D}}_k\right|$ are the amount of UE communication data collected at MEC $m_k$ and ${\xi }_{k,i}=(s_{k,i},a_{k,i},\pi (·|s_{k,i},{\mathbf{w}}_{k,i}),r_{k,i},s_{k,t+1})$ are the sample data at time i in MEC $m_k$ and $f({\mathbf{w}}_k;{\xi }_{k,i})$ is the loss function of basic PPO algorithm [36] which is optimized to minimize the discrepancy between policy improvement and value function prediction using Mean Squared Error (MSE). It should be noted that various loss functions commonly used in federated learning (FL) can be represented as $F_k\left({\mathbf{w}}_k\right)$. In this study, the Stochastic Gradient Descent (SGD) method is chosen for the loss function as it is widely adopted in FL, as mentioned in [37]. The time complexity of local training and updates for the PPO-based RL agent are related to the complexity of the policy and value networks [38]. The time complexity of the PPO algorithm increases linearly with the size of the action space and the number of nodes in the hidden layer. In our case, as shown in Table 1, the number of nodes in the network and the size of the action space are small, so the time required for a single update is negligible.

3. Local Agent Scoring and Selection: Training agents transmit their local gradients to each committee MEC, which then evaluates these gradients based on a predetermined scoring system. Only those gradients that meet the criteria of the established selection strategy are utilized to construct the global gradient. We will describe the details of the committee mechanism later in Section 3.3.

4. Aggregation: The selected MEC becomes an aggregation set ${\mathbf{S}}_t^a$, and transmits the local model parameters from the agent and updates the global model parameter $\overline{\mathbf{w}}$. The widely adopted FedAvg [15] aggregation method is used, formulated as an optimization problem:

   (6)$\underset{\overline{\mathbf{w}}}{\min }\left(F\left(\overline{\mathbf{w}}\right)={\displaystyle \frac{N_k}{N}}\sum _{k=1}^K{F}_k\left({\mathbf{w}}_k\right)\right),$

   where N is the total number of UE communication data across all MEC servers. The aggregation set broadcasts the updated global model parameters back to $m\in \mathbf{M}$.

5. Iteration: This distributed local training and RL agent model parameter aggregation process is repeated until the RL agent thought to be fully trained to adaptive scale pods or iteration round t is reached at maximum communication round T. Excessive training can lead to overfitting, which significantly degrades the performance of the proposed algorithm. While [3] addressed the volatility of the PPO algorithm by integrating it with a machine learning model, we mitigate this variability through the application of FL. By setting an appropriate value T, FL allows us to employ an optimally converged model before the significant performance degradation associated with overfitting occurs, enabling a more efficient training process.

The following algorithm provides a detailed overview of our proposed process, including the DFRL step and committee step.

3.3. Committee-Based Architecture

In FL, workers typically do not share their training data with others. Consequently, some workers may be malicious, altering parameters (such as weights or gradients) in their models to undermine the training accuracy of the global model. This type of behavior is commonly referred to as a Byzantine attack [39]. Recent studies reveal that standard FL is highly susceptible to Byzantine attacks by faulty or malicious clients [40]. Even a single attacker can drastically reduce model accuracy from 100% to 0%.

In this section, as in Figure 3, we propose the committee mechanism for the DFRL framework, which is composed of the scoring system, election strategy, selection strategy, and the committee consensus protocol. The committee mechanism strengthens the system’s resilience by blocking malicious MEC from impacting the training process. By ranking MEC servers based on their scores and choosing those around the middle or higher positions, the committee reduces the influence of malicious MEC servers which usually have lower scores [13].

The following steps summarize the committee mechanism process:

• Global model download: Each MEC server $m\in {\mathbf{S}}_t^b$ downloads a global model $\overline{{\mathbf{w}}_t}$.

• Local training: Each $m\in {\mathbf{S}}_t^b\cup {\mathbf{S}}_t^c$ trains its local model to minimize its local objective function with equation 6. After local training, each MEC computes its local gradient ${\widehat{g}}_{k,t}$:

  (7)${\widehat{g}}_{k,t}=g_k({\mathbf{w}}_{kt},{\mathcal{B}}_{k,t})={\displaystyle \frac{1}{|{\mathcal{B}}_{k,t}|}}\sum _{\xi \in {\mathcal{B}}_{k,i}^t}\nabla f({\mathbf{w}}_{k,t},\xi ),$

  where ${\mathcal{B}}_{k,t}$ is mini-batch for t iterations of SGD, which is randomly sampled from ${\mathcal{D}}_k$.

• Scoring model: The scoring system’s main purpose is to differentiate between honest and malicious gradients by calculating the Euclidean distance. Honest gradients tend to have a lower Euclidean distance compared to the distance between an honest gradient and a malicious one. Consequently, MEC servers uploading honest gradients receive higher scores, while those uploading malicious gradients receive lower scores. Assume the local gradient on the k-th training MEC at round t is denoted as ${\widehat{g}}_{k,t}=g_k({\mathbf{w}}_{k,t};{\mathcal{B}}_{k,t})$, and the local gradient on the c-th committee MEC at round t is ${\widehat{g}}_{c,t}=g_c({\mathbf{w}}_{c,t};{\mathcal{B}}_{c,t})$. The score ${\mathcal{P}}_k^c$ of the k-th training MEC, assigned by the c-th committee MEC, is calculated as follows:

  (8)$p_k^c={\displaystyle \frac{1}{\parallel {\widehat{g}}_{k,t}-{\widehat{g}}_{c,t}{\parallel }_2^2}}.$

  The final score of the k-th training MEC is defined as follows:

  (9)$P_k={\displaystyle \frac{1}{\frac{1}{\left|{\mathbf{S}}_c\right|}{\sum }_{c\in {\mathbf{S}}_c^t}{\parallel {\widehat{g}}_{k,t}-{\widehat{g}}_{c,t}\parallel }_2^2}}=\left|{\mathbf{S}}_c\right|\sum _{c\in {\mathbf{S}}_c}{\displaystyle \frac{1}{p_k^c}}.$

  The scoring principle is based on the Euclidean distance between local gradients ${\widehat{g}}_{,t}$ and ${\widehat{g}}_{c,t}$. Typically, Byzantine attackers replace some local gradients with malicious ones, which increases the Euclidean distance between these and the honest gradients. Thus, when the proportion of malicious MEC is tolerable, their scores will be lower than those of the honest MEC. In non-attack scenarios, the score indicates the degree of MEC heterogeneity, with a higher score representing higher heterogeneity.

• Aggregate model: Committee MEC servers select local gradients with higher scores to construct the global gradient. Gradients similar to the committee gradients in Euclidean space participate in constructing the global gradient. We sort the local gradients by their scores and accept the top $\alpha$% of them, focusing on robustness. This approach ensures a more robust aggregation process by choosing gradients close to those of the committee, assuming the committee is honest. However, this makes it hard for MEC servers with significant differences to be selected, potentially lowering performance.

• Committee voting: This ensures the honesty of committee set ${\mathbf{S}}_t^c$. Committee members reach a consensus on their decisions, which relies on the majority. Malicious MEC servers in the committee can interfere with decision-making, so it is crucial to have more honest than malicious members. Malicious MEC servers generally have low scores, so to prevent malicious members from being included as committee members, the committee members should be selected primarily from those with high scores. However, if the committee is composed solely of the highest-scoring members, the global model’s learning direction would be overly influenced by the initial committee members. This aligns with the intuition that committee members should represent the majority. Thus, we sort training MEC servers by their scores and select those in the middle $\beta$% score position as new committee MEC servers.

The following pseudocode represents a committee-based algorithm.

The overall time complexity of our framework (Algorithm 1) is O (DFRL process) + O (Algorithm 2). As previously noted, the complexity of the RL agent is negligible due to the model size in the proposed scenario. Therefore, the time complexity of the DFRL process is determined by T and K, while the complexity of Algorithm 2 is determined solely by K. Thus, the time complexity of the entire framework can be expressed as $O(TK+K^2)$.

Algorithm 1 DFRL training loop

1:. for $m_{1,2,\cdots ,k}\in \mathbf{M}$ do 2:.       initialize local RL agent and environment. 3:. end for 4:. for $t=0$ to T do 5:.       for $m_{1,2,\cdots ,k}\in \mathbf{M}$ do 6:.           download global model parameter ${\overline{\mathbf{w}}}_{t-1}$. 7:.           Get action from agent: $a_{k,t}\leftarrow$ Sample ${\pi }_{k,t}(s_{k,t},\overline{{\mathbf{w}}_{t-1}})$. 8:.           Execute action $a_{k,t}$ to scale the cluster in MEC $m_k$. 9:.           Observe the new state $s_{k,t+1}$ and performance measures during DFL round. 10:.         Compute reward $r_{k,t}$ from the measurements using Equation (3). 11:.         agent update local RL agent using PPO update Algorithm with Equation (6). 12:.     end for 13:.     get new aggregation set ${\mathbf{S}}_t^a$ with Algorithm 2. 14:.     aggregate all ${\mathbf{w}}_t$ in aggregation set $S_t^a$ and generate new global model ${\overline{\mathbf{w}}}_t$ using FedAvg [15]. 15:.     $t\leftarrow t+1$ 16:. end for

Algorithm 2 Committee mechanism

Input: t, ${\mathbf{w}}_t$ from all MECs $m\in \mathbf{M}$   Output: new aggregation MEC set ${\mathbf{S}}_{t+1}^a$, new committee MEC set ${\mathbf{S}}_{t+1}^c$ 1:. for $c\in {\mathbf{S}}_t^c$ do 2:.       for $k\in {\mathbf{S}}_t^b$ do 3:.             Committee c broadcasts score $p_k^c$ to other committee MECs. 4:.       end for 5:. end for 6:. for $c\in {\mathbf{S}}_t^c$ do 7:.       for $k\in {\mathbf{S}}_t^b$ do 8:.             Committee c calculates the total score $P_k$ of MEC k. 9:.       end for 10:. end for 11:. Choose MEC servers $a\in {\mathbf{S}}_{t+1}^a$ with the top $\alpha$% of their score among ${\mathbf{S}}_t^b$. 12:. Choose MEC servers $c\in {\mathbf{S}}_{t+1}^c$ with the middle $\beta$% of their score among ${\mathbf{S}}_t^b$. 13:. Return ${\mathbf{S}}_{t+1}^a$.

4. Evaluation

In this section, we evaluated the effectiveness of the proposed framework through simulation-based experiments. First, we implemented the benchmark of the proposed framework and conducted evaluations by comparing the scaling performance according to the varying arrival rate. Then, we conducted experimental results on the defense performance against Byzantine attacks from malicious servers.

4.1. Environment Setting

We slightly adjust the simulator in [3], which has a multi-distributed MEC servers serving 5G network using Python 3.9.7. In our experiments, the simulation program was developed using Python. To implement the RL agents, we utilized PyTorch 2.1.1, which allows us to construct neural network model and train the RL agents within the system. The RL environment was managed using Gymnasium 0.29.1, a robust framework for RL environments that provided the necessary tools to simulate dynamic network conditions and interactions between the agents and the environment. For more detailed hyperparameters of our simulator are listed in Table 2. There are K MEC servers and each MEC server has a core network with multiple nodes that have PDU sessions which make data transmission occur between UE and DN. When UE arrives, a PDU session is initiated if the system has available capacity; otherwise, the request is blocked. This simulator is equipped to simulate the each arrival of UE as a Poisson process with a specific rate at a given time, but the exact arrival of UE can not be known in practical scenarios. Here, we assume that the arrival rate of UE makes a periodic function:

(10)${\lambda }_{k,t}^{\mathrm{train}}=A_k+B_{k}sin\left({\displaystyle \frac{1}{C_k}}\right)\pi t,$

with random value $A_k,B_k,C_k$ that $0\le B_k\le A_k\le 250$ represent the amplitude and the mean of arrival rate, $1\le C_k\le 10$ represents the period. More detailed parameters of our simulation are shown in Table 2.

4.2. Convergence Analysis

In this subsection, we evaluate the convergence of our proposed DFRL framework. We designed our multi-MEC simulator illustrated in Section 4.1. All MEC servers have one DRL agent and train their agent via Algorithm 1. At each round, we collect rewards from all the DRL agents and average them to show the convergence.

In Figure 4, at the early stage of evaluation, the average rewards of our framework fluctuate significantly because the agents are still in the exploration phase, and there are insufficient data in the experience replay buffers for effective training. In this phase, the DRL agent can not provide the scaling action properly, leading to too many or too less pod scales, resulting in a poor reward. As the number of training episodes increases, the agents’ average rewards begin to gradually improve. Ultimately, our algorithm reached a stable positive reward and the fluctuation was significantly down. Thus, we conclude that our trained agents can provide scaling action properly while maintaining a request blocking rate under threshold value, with no wasted core network resources.

To evaluate our framework, we adapt an equation from [3] which presents mobile user traffic [41] as the evaluation arrival rate:

(11)$\begin{array}{cc}\hfill {\lambda }_t^{\mathrm{eval}}& =330.07620+171.10476sin\left({\displaystyle \frac{1}{12}}\pi t+3.08\right)\hfill \\ \hfill & +100.19048sin\left({\displaystyle \frac{1}{6}}\pi t+2.08\right)\hfill \\ \hfill & +31.77143sin\left({\displaystyle \frac{1}{4}}\pi t+1.14\right)\hfill \end{array}$

and the value of ${\lambda }_t^{\mathrm{eval}}$ is represented in Figure 5.

The goal of the RL agent in the proposed framework is to minimize cost while maintaining the system’s QoS. As mentioned before, the reward function simultaneously considers the number of active pods and the blocking rate, and in an MEC system, we remind that keeping the request-blocking rate low is synonymous with maintaining the system’s QoS, and minimizing the number of activated pods corresponds to minimizing the system’s cost. Figure 6 illustrates the real-world performance of the proposed framework in MEC environments. Figure 6a shows the scaling of pods in response to request ${\lambda }^{\mathrm{eval}}$. Compared with Figure 5, we can observe that the proposed framework adjusts the number of pods according to the incoming requests. Figure 6b depicts the blocking rate of requests. With the exception of hour 5 and hour 30, when the number of requests was too low to determine precise scaling actions, the blocking rate is maintained below the blocking rate threshold $b_{\mathrm{th}}$ for nearly all times. Therefore, it is evident that the proposed framework faithfully achieves the goal of minimizing cost while maintaining QoS in the MEC environment.

4.3. Different Distribution of Arrival Rate

In the previous section, we assume that the arrival rate of UE follows a periodic function. However, in reality, the assumption that the arrival rate of UE follows a periodic function may not always hold [42]. To explore the impact of different distributions on the accuracy of our DFRL framework, we perform a sensitivity analysis using simulations with different distributions of arrival rates. We use alternative distributions, which are gamma distribution with shape parameter 3 and 5, and uniform distribution. To evaluate the performance in comparison, we assess the results of the traditional DRL method without the application of the FL technique. In our setup, the DRL agent operates in a single MEC, and we create two different benchmarks based on the amount of data used for training: (1) data collected from only one MEC (i.e., local training) and (2) data aggregated from all K MECs (i.e., centralized training). The simulated average rewards for different distributions of arrival rates are shown in Table 3 and depicted in Figure 7.

For the DRL agent trained using data from only a single MEC, the average reward is observed to be very low. This indicates that the agent is unable to appropriately select scaling actions based on the measured arrival rates, leading to inadequate adjustments in the number of pods. As a result, the system either experiences a high blocking rate due to an insufficient number of pods or faces overprovisioning when there are too many pods. The agent remains in the exploration phase of DRL, demonstrating insufficient training due to the lack of enough data, and ultimately shows poor scaling performance. In contrast, the DRL agent trained with the aggregated training data from all MECs (i.e., centralized training) achieved the highest average reward. This suggests that the agent learned from a much wider range of scenarios compared to when it was trained with data from a single MEC, gaining the ability to respond adaptively and flexibly to diverse situations. However, since this approach requires aggregating data from all MEC servers in one location for training, it can potentially cause data privacy concerns and create networking and computing bottlenecks at the centralized server. Therefore, while this benchmark may provide an upper bound on performance, it is not scalable or practical for real-world applications. In comparison, our framework demonstrated scaling performance similar to that of the DRL agent trained with aggregated data from all MEC servers across various arrival rate distributions. By utilizing distributed learning, our framework avoids the need to centralize data, thereby mitigating the previously mentioned data privacy issues and computing/networking bottlenecks while still achieving appropriate scaling actions. These results suggest that our framework is a more practical and effective choice than the traditional DRL methods in a multi-MEC scenario.

4.4. Simulation with Malicious MEC Servers

In this subsection, we test the robustness of our committee-based algorithm for Byzantine attacks from malicious MEC servers compared with the DFRL framework with no Byzantine defenses. In our simulator, to imitate the back-gradient attack, malicious MEC servers share their local agent parameter adding significant Gaussian noise which has the purpose of reducing the global model performance. We consider three hyperparameters in committee mechanism, $\alpha$, $\beta$, $ϵ$ is a portion of each aggregator MEC servers, committee MEC servers, and malicious MEC servers, respectively. We divide this evaluation into three sub-tests, and for each sub-test, we set two hyperparameters with a fixed value and vary the other hyperparameter as follows:

1. varying committee ratio $\beta =\{10,20,30,40,50\}$, fixed $\alpha =40$, $ϵ=10$.

2. varying aggregator ratio $\alpha =\{10,20,30,40,50\}$, fixed $\beta =40$, $ϵ=10$.

3. varying malicious ratio $ϵ=\{10,20,30,40,50\}$, fixed $\alpha =40$, $\beta =40$.

In sub-tests, we average the rewards from all MEC servers as scaling performance with our committee-based DFRL framework. Also, we run sub-test 3 with a DFRL framework with no committee mechanism and observed the results of the non-committee framework. We run each sub-test 10 times and average those results, which are illustrated in Figure 8 and Figure 9.

Figure 8a demonstrates that increasing the number of committee members by enough can positively impact global performance. Expanding the number of committee members can enhance the robustness of the committee and prevent the training and aggregation stages from being dominated by a small number of members.

Figure 8b shows that appropriately increasing the number of aggregator members can also positively impact global performance. The condition for being selected as an aggregator member is to receive high scores from committee members. In a typical flow where a normal committee is formed, honest members are more likely to receive higher scores from the committee members compared to malicious members, increasing their chances of being selected as aggregator members. By increasing the number of aggregator members, more honest members are likely to be chosen, thereby enhancing the robustness of the global performance.

Figure 9 shows that if the number of malicious members is high, global performance significantly deteriorates in most situations. Increasing the number of committee members raises the likelihood of malicious members being included in the committee. This results in honest members receiving lower scores from the committee, preventing them from being selected as aggregator members. The probability of forming an aggregator composed of malicious members increases, leading to a significant drop in global performance. Similarly, increasing the number of aggregator members can also raise the likelihood of malicious members being included among them, potentially causing further degradation in global performance. Finally, the result shows that the proposed committee mechanism is more robust than a non-committee framework with every $ϵ$ value. The non-committee-based frameworks can not protect their global performance when the malicious ratio $ϵ$ meets only 20%. However, our DFRL framework with a committee-based mechanism can maintain their global performance even $ϵ$ is more than 30%. In the experimental results, it can be observed that the global performance significantly decreases when the proportion of malicious entities exceeds 40%. This decline is attributed to the system being entirely compromised by the attack, preventing it from taking appropriate scaling actions.

5. Conclusions

In this paper, we presented a DFRL framework enhanced with a committee mechanism, aiming for robust and efficient virtual instance scaling in MEC environments for 5G core network automation. Our experimental results showed that the proposed framework outperforms traditional approaches by maintaining high network performance and achieving resilience under various operational conditions, including different traffic patterns and the presence of adversarial MECs. Qualitatively, the committee mechanism offers a significant advantage: it proactively identifies and excludes malicious MEC participants, thereby enhancing both security and model stability. This self-regulating feature sets our framework apart from others that either lack strong adversary detection capabilities or struggle to maintain performance consistency in the presence of security threats. By minimizing the likelihood of harmful decisions in the DRL-based decision-making process, our framework introduces an additional layer of control over actions driven by probabilistic policies, though the complete elimination of undesirable actions remains a challenge. Furthermore, the framework demonstrates scalability across a variety of network configurations and operational environments, emphasizing its versatility and potential applicability in future 5G and beyond network deployments. While our results already showcase clear performance improvements, future research should focus on refining the probabilistic aspects of the DRL component to further reduce risks associated with adversarial attacks and stochastic behaviors. This ongoing evolution holds the potential to provide even greater robustness and security for MEC-based automation.

Author Contributions

J.J.: conceptualization, data curation, formal analysis, methodology, software, validation, visualization, writing—original draft. J.L.: conceptualization, methodology, validation, writing—review and editing. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data are contained within the article.

Conflicts of Interest

The authors declare no conflicts of interest.

List of Notations

Notations for the environment
K	number of MEC servers
$D^{\min }$	minimum number of pods
$L^{\mathrm{sess}}$	maximum number of PDU sessions per pod
t	current episode round
$\mathbf{M}$	Whole MEC servers set
$D^{\max }$	maximum number of pods
$b_{th}$	blocking rate threshold
T	maximum number of episode round
Notations for the observation
$d_{k,t}^{\mathrm{on}}$	number of running pods in MEC k at round t
${\widehat{\lambda }}_{k,t}$	measured arrival rate in MEC k at round t
$l_{k,t}^{\mathrm{sess}}$	number of running sessions in MEC k at round t
${\widehat{b}}_{k,t}$	measured blocking rate in MEC k at round t
Notations for the MDP
$\mathcal{S}$	state space set of MDP
$\mathcal{A}$	action space set of MDP
${\pi }_{k,t}$, ${\pi }^{*}$	policy function in MEC k at round t, optimal policy
$\mathsf{p}$	transition probability
$\kappa$	reward multiplier
$s_{k,t}$	state in MEC k at round t
$a_{k,t}$	action in MEC k at round t
$r_{k,t}$	reward in MEC k at round t
$\gamma$	discounting factor
V	value function
Notations for the committee mechanism
${\mathbf{S}}_t^a$	aggregation MEC set
${\mathbf{S}}_t^c$	committee MEC set
${\mathbf{w}}_{k,t}$	model parameters in MEC k at round t
$f(\mathbf{w};\xi )$	Overall loss function of RL
${\widehat{g}}_{k,t}$	local gradient in MEC k at round t
$p_k^c$	score of MEC k calculated from committee c
${\mathbf{S}}_t^b$	training MEC set
$\theta$, $\omega$	parameter vectors of policy and value function
$\overline{\mathbf{w}}$	global model parameter
$F\left(\mathbf{w}\right)$	Local objective function
${\mathcal{B}}_k$	randomly sampled mini-batch from ${\mathcal{D}}_k$
$P_k$	total score of MEC k

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Overall architecture of the proposed framework.

Figure 2. The neural network for the policy [Forumla omitted. See PDF.] takes the state as input and produces the action probabilities for that specific state. The connections correspond to the weights in [Forumla omitted. See PDF.], and the nodes in the hidden layer apply a non-linear activation function.

Figure 3. Proposed committee mechanism.

Figure 4. Average reward with proposed DFRL framework.

Figure 5. The value of [Forumla omitted. See PDF.].

Figure 6. Scaling performance of proposed DFRL framework. (a) Pod count [Forumla omitted. See PDF.]. (b) Blocking rate [Forumla omitted. See PDF.].

Figure 7. Mean reward results using a different distribution of arrival rate.

Figure 8. Boxplot showing the distribution of mean rewards across different committee and aggregator ratio. The boxes represent the interquartile range (IQR), with the median indicated by the horizontal line inside each box. Whiskers extend to 1.5 times the IQR. (a) Different committee ratio. (b) Different aggregator ratio.

Figure 9. Boxplot showing the distribution of mean rewards across different malicious MEC ratios compared with committee-based and non-committee-based DFRL frameworks.

Table 1

DRL hyperparameters.

neural network hidden layers	1
node count in each hidden layer	50
learning rate	0.0001
epochs	5
batch size	32
PPO clipping parameter	0.1
GAE parameter	0.9
activation function	ReLU

Table 2

Environment hyperparameter setting.

number of MECs ($K=\left|\mathbf{M}\right|$)	50
minimum number of Pods in MEC ($D^{\min }$)	2
maximum number of Pods in MEC ($D^{\max }$)	100
PDU session service rate	1/s
requests blocking rate threshold ($b_{th}$)	0.01
scaling decision time interval ($\Delta t$)	1s

Table 3

Mean reward results using a different distribution of arrival rate.

Distribution	DRL (1 MEC)	DRL (K MECs)	DFRL (K MECs)
${\lambda }_t^{\mathrm{eval}}$ (Equation (11))	−11.58974	−1.32819	−2.30002
gamma distribution ($\Gamma$ with shape 3)	−10.02404	−1.12015	−2.07609
gamma distribution ($\Gamma$ with shape 5)	−9.99494	−1.02000	−1.90180
uniform distribution ($\lambda =250$/s)	−9.65637	−0.88085	−1.75417

References

1. Lee, J.; Solat, F.; Kim, T.Y.; Poor, H.V. Federated Learning-Empowered Mobile Network Management for 5G and Beyond Networks: From Access to Core. IEEE Commun. Surv. Tutorials; 2024; 26, pp. 2176-2212. [DOI: https://dx.doi.org/10.1109/COMST.2024.3352910]

2. Rotter, C.; Van Do, T. A Queueing Model for Threshold-Based Scaling of UPF Instances in 5G Core. IEEE Access; 2021; 9, pp. 81443-81453. [DOI: https://dx.doi.org/10.1109/ACCESS.2021.3085955]

3. Nguyen, H.T.; Van Do, T.; Rotter, C. Scaling UPF Instances in 5G/6G Core With Deep Reinforcement Learning. IEEE Access; 2021; 9, pp. 165892-165906. [DOI: https://dx.doi.org/10.1109/ACCESS.2021.3135315]

4. Al-Dhuraibi, Y.; Paraiso, F.; Djarallah, N.; Merle, P. Elasticity in Cloud Computing: State of the Art and Research Challenges. IEEE Trans. Serv. Comput.; 2018; 11, pp. 430-447. [DOI: https://dx.doi.org/10.1109/TSC.2017.2711009]

5. Dogani, J.; Namvar, R.; Khunjush, F. Auto-scaling techniques in container-based cloud and edge/fog computing: Taxonomy and survey. Comput. Commun.; 2023; 209, pp. 120-150. [DOI: https://dx.doi.org/10.1016/j.comcom.2023.06.010]

6. Lorido-Botran, T.; Miguel-Alonso, J.; Lozano, J.A. A Review of Auto-scaling Techniques for Elastic Applications in Cloud Environments. J. Grid Comput.; 2014; 12, pp. 559-592. [DOI: https://dx.doi.org/10.1007/s10723-014-9314-7]

7. Toka, L.; Dobreff, G.; Fodor, B.; Sonkoly, B. Machine Learning-Based Scaling Management for Kubernetes Edge Clusters. IEEE Trans. Netw. Serv. Manag.; 2021; 18, pp. 958-972. [DOI: https://dx.doi.org/10.1109/TNSM.2021.3052837]

8. Tran, M.N.; Vu, D.D.; Kim, Y. A Survey of Autoscaling in Kubernetes. Proceedings of the 2022 Thirteenth International Conference on Ubiquitous and Future Networks (ICUFN); Barcelona, Spain, 5–8 July 2022; pp. 263-265. [DOI: https://dx.doi.org/10.1109/ICUFN55119.2022.9829572]

9. Meignanamoorthi, D.; Vetriselvi, V. DRL-based customised resource allocation for sub-slices in 6G network slicing. Trans. Emerg. Telecommun. Technol.; 2024; 35, e5016. [DOI: https://dx.doi.org/10.1002/ett.5016]

10. Gkonis, P.; Nomikos, N.; Panagiotis, T.; Sarakis, L.; Xilouris, G.; Masip, X.; Martrat, J. Leveraging Network Data Analytics Function and Machine Learning for Data Collection, Resource Optimization, Security and Privacy in 6G Networks. IEEE Access; 2024; 12, pp. 21320-21336. [DOI: https://dx.doi.org/10.1109/ACCESS.2024.3359992]

11. Kweon, K.; Gutierrez-Estevez, D.; Pujol-Roig, J.; Jeong, S. Automated Multi-service 5G Session Timer via AI-based Network Data Analytics Function. Proceedings of the 2020 IEEE Globecom Workshops (GC Wkshps); Taipei, Taiwan, 7–11 December 2020; pp. 1-6. [DOI: https://dx.doi.org/10.1109/GCWkshps50303.2020.9367529]

12. Lim, D.; Joe, I. A DRL-Based Task Offloading Scheme for Server Decision-Making in Multi-Access Edge Computing. Electronics; 2023; 12, 3882. [DOI: https://dx.doi.org/10.3390/electronics12183882]

13. Che, C.; Li, X.; Chen, C.; He, X.; Zheng, Z. A Decentralized Federated Learning Framework via Committee Mechanism with Convergence Guarantee. IEEE Trans. Parallel Distrib. Syst.; 2022; 33, pp. 4783-4800. [DOI: https://dx.doi.org/10.1109/TPDS.2022.3202887]

14. Mekrache, A.; Boutiba, K.; Ksentini, A. Combining Network Data Analytics Function and Machine Learning for Abnormal Traffic Detection in Beyond 5G. Proceedings of the GLOBECOM 2023—2023 IEEE Global Communications Conference; Kuala Lumpur, Malaysia, 4–8 December 2023; pp. 1204-1209. [DOI: https://dx.doi.org/10.1109/GLOBECOM54140.2023.10436766]

15. McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; y Arcas, B.A. Communication-efficient learning of deep networks from decentralized data. Proceedings of the Artificial Intelligence and Statistics, PMLR; Fort Lauderdale, FL, USA, 20–22 April 2017; pp. 1273-1282.

16. Alharthi, S.; Alshamsi, A.; Alseiari, A.; Alwarafy, A. Auto-Scaling Techniques in Cloud Computing: Issues and Research Directions. Sensors; 2024; 24, 5551. [DOI: https://dx.doi.org/10.3390/s24175551] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/39275461]

17. Paraiso, F.; Merle, P.; Seinturier, L. soCloud: A service-oriented component-based PaaS for managing portability, provisioning, elasticity, and high availability across multiple clouds. Computing; 2016; 98, pp. 539-565. [DOI: https://dx.doi.org/10.1007/s00607-014-0421-x]

18. Kan, C.H. DoCloud: An elastic cloud platform for Web applications based on Docker. Proceedings of the 2016 18th International Conference on Advanced Communication Technology (ICACT); PyeongChang, Republic of Korea, 31 January–3 February 2016; pp. 478-483. [DOI: https://dx.doi.org/10.1109/ICACT.2016.7423440]

19. Baresi, L.; Guinea, S.; Leva, A.; Quattrocchi, G. A discrete-time feedback controller for containerized cloud applications. Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering; New York, NY, USA, 13–18 November 2016; FSE 2016 pp. 217-228. [DOI: https://dx.doi.org/10.1145/2950290.2950328]

20. Zhang, Q.; Cherkasova, L.; Smirni, E. A Regression-Based Analytic Model for Dynamic Resource Provisioning of Multi-Tier Applications. Proceedings of the Fourth International Conference on Autonomic Computing (ICAC’07); Jacksonville, FL, USA, 11–15 June 2007; 27. [DOI: https://dx.doi.org/10.1109/ICAC.2007.1]

21. Ali-Eldin, A.; Tordsson, J.; Elmroth, E. An adaptive hybrid elasticity controller for cloud infrastructures. Proceedings of the 2012 IEEE Network Operations and Management Symposium; Maui, HI, USA, 16–20 April 2012; pp. 204-212. [DOI: https://dx.doi.org/10.1109/NOMS.2012.6211900]

22. Lakew, E.B.; Klein, C.; Hernandez-Rodriguez, F.; Elmroth, E. Towards Faster Response Time Models for Vertical Elasticity. Proceedings of the 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing; London, UK, 8–11 December 2014; pp. 560-565. [DOI: https://dx.doi.org/10.1109/UCC.2014.86]

23. Kaur, P.D.; Chana, I. A resource elasticity framework for QoS-aware execution of cloud applications. Future Gener. Comput. Syst.; 2014; 37, pp. 14-25. [DOI: https://dx.doi.org/10.1016/j.future.2014.02.018]

24. Fernandez, H.; Pierre, G.; Kielmann, T. Autoscaling Web Applications in Heterogeneous Cloud Infrastructures. Proceedings of the 2014 IEEE International Conference on Cloud Engineering; Boston, MA, USA, 11–14 March 2014; pp. 195-204. [DOI: https://dx.doi.org/10.1109/IC2E.2014.25]

25. Kassela, E.; Boumpouka, C.; Konstantinou, I.; Koziris, N. Automated workload-aware elasticity of NoSQL clusters in the cloud. Proceedings of the 2014 IEEE International Conference on Big Data (Big Data); Washington, DC, USA, 27–30 October 2014; pp. 195-200. [DOI: https://dx.doi.org/10.1109/BigData.2014.7004232]

26. Rao, J.; Bu, X.; Xu, C.Z.; Wang, L.; Yin, G. VCONF: A reinforcement learning approach to virtual machines auto-configuration. Proceedings of the 6th International Conference on Autonomic Computing; Barcelona, Spain, 15–19 June 2009; pp. 137-146. [DOI: https://dx.doi.org/10.1145/1555228.1555263]

27. Dutreilh, X.; Kirgizov, S.; Melekhova, O.; Malenfant, J.; Rivierre, N.; Truck, I. Using reinforcement learning for autonomic resource allocation in clouds: Towards a fully automated workflow. Proceedings of the 7th International Conference on Autonomic and Autonomous Systems (ICAS 2011); Venice, Italy, 22–27 May 2011; pp. 67-74.

28. Barrett, E.; Howley, E.; Duggan, J. Applying reinforcement learning towards automating resource allocation and application scalability in the cloud. Concurr. Comput. Pract. Exp.; 2013; 25, pp. 1656-1674. [DOI: https://dx.doi.org/10.1002/cpe.2864]

29. Schuler, L.; Jamil, S.; Kühl, N. AI-based Resource Allocation: Reinforcement Learning for Adaptive Auto-scaling in Serverless Environments. Proceedings of the 2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid); Melbourne, Australia, 10–13 May 2021; pp. 804-811. [DOI: https://dx.doi.org/10.1109/CCGrid51090.2021.00098]

30. Khaleq, A.A.; Ra, I. Intelligent Autoscaling of Microservices in the Cloud for Real-Time Applications. IEEE Access; 2021; 9, pp. 35464-35476. [DOI: https://dx.doi.org/10.1109/ACCESS.2021.3061890]

31. Villota-Jacome, W.F.; Rendon, O.M.C.; da Fonseca, N.L.S. Admission Control for 5G Core Network Slicing Based on Deep Reinforcement Learning. IEEE Syst. J.; 2022; 16, pp. 4686-4697. [DOI: https://dx.doi.org/10.1109/JSYST.2022.3172658]

32. 3GPP. Technical Specification Group Services and System Aspects; Architecture Enhancements for 5G System (5GS) to Support Network Data Analytics Services; Stage 2 (Release 16). Technical Report TS 23.288, 3rd Generation Partnership Project (3GPP), 3rd Generation Partnership Project (3GPP). 2020; Version 16.0.0 Available online: https://www.3gpp.org/ftp/Specs/archive/23_series/23.288/23288-g00.zip (accessed on 6 August 2024).

33. 3GPP. Technical Specification Group Core Network and Terminals; 5G System; Network Data Analytics Services (Release 16). Technical Report TS 29.520, 3rd Generation Partnership Project (3GPP), 3rd Generation Partnership Project (3GPP). 2020; Version 16.0.0 Available online: https://www.3gpp.org/ftp/Specs/archive/29_series/29.520/29520-g00.zip (accessed on 6 August 2024).

34. Sutton, R.S.; Barto, A.G. Reinforcement Learning: An Introduction; MIT Press: Cambridge, MA, USA, 2018.

35. Schulman, J.; Moritz, P.; Levine, S.; Jordan, M.; Abbeel, P. High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv; 2018; arXiv: 1506.02438

36. Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; Klimov, O. Proximal Policy Optimization Algorithms. arXiv; 2017; arXiv: 1707.06347

37. Lim, W.Y.B.; Luong, N.C.; Hoang, D.T.; Jiao, Y.; Liang, Y.C.; Yang, Q.; Niyato, D.; Miao, C. Federated Learning in Mobile Edge Networks: A Comprehensive Survey. IEEE Commun. Surv. Tutorials; 2020; 22, pp. 2031-2063. [DOI: https://dx.doi.org/10.1109/COMST.2020.2986024]

38. Ho, T.M.; Nguyen, K.K.; Cheriet, M. Federated Deep Reinforcement Learning for Task Scheduling in Heterogeneous Autonomous Robotic System. IEEE Trans. Autom. Sci. Eng.; 2024; 21, pp. 528-540. [DOI: https://dx.doi.org/10.1109/TASE.2022.3221352]

39. Wang, Y.; Xia, Y.; Zhan, Y. ELITE: Defending Federated Learning against Byzantine Attacks based on Information Entropy. Proceedings of the 2021 China Automation Congress (CAC); Beijing, China, 22–24 October 2021; pp. 6049-6054. [DOI: https://dx.doi.org/10.1109/CAC53003.2021.9727486]

40. Shi, J.; Wan, W.; Hu, S.; Lu, J.; Zhang, L.Y. Challenges and Approaches for Mitigating Byzantine Attacks in Federated Learning. arXiv; 2022; arXiv: 2112.14468

41. Wang, S.; Zhang, X.; Zhang, J.; Feng, J.; Wang, W.; Xin, K. An Approach for Spatial-Temporal Traffic Modeling in Mobile Cellular Networks. Proceedings of the 2015 27th International Teletraffic Congress; Ghent, Belgium, 8–10 September 2015; pp. 203-209. [DOI: https://dx.doi.org/10.1109/ITC.2015.31]

42. Wang, B.; Irwin, D.; Shenoy, P.; Towsley, D. INVAR: Inversion Aware Resource Provisioning and Workload Scheduling for Edge Computing. Proceedings of the IEEE INFOCOM 2024—IEEE Conference on Computer Communications; Vancouver, BC, Canada, 20–23 May 2024; pp. 1511-1520. [DOI: https://dx.doi.org/10.1109/INFOCOM52122.2024.10621417]