Small-Scale 5G Testbeds for Network Slicing Deployment: A Systematic Review

Abstract

Developing specialized cloud-based and open-source testbeds is a practical approach to investigate network slicing functionalities in the fifth-generation (5G) mobile networks. This paper provides a comprehensive review of most of the existing cost-efficient and small-scale testbeds that partially or fully deploy network slicing. First, we present relevant software packages for the three main functional blocks of the ETSI NFV MANO framework and for emulating the access and core network domains. Second, we define primary and secondary design criteria for deploying network slicing testbeds. These design criteria are later used for comparison between the testbeds. Third, we present the state-of-the-art testbeds, including their design objectives, key technologies, network slicing deployment, and experiments. Next, we evaluate the testbeds according to the defined design criteria and present an in-depth summary table. This assessment concludes with the superiority of some of them over the rest and the most dominant software packages satisfying the ETSI NFV MANO framework. Finally, challenges, potential solutions, and future works of network slicing testbeds are discussed.

This work is licensed under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.

1. Introduction

The fifth-generation (5G) and beyond networks are expected to provide various services compared to the 4G and previous generations of networks. The Quality of Service (QoS) requirements can be quite different in terms of low latency (or even extralow latency), bandwidth, reliability, and availability. Remote surgery, autonomous driving, a massive number of sensors communicating with the network, and video streaming with extrahigh quality are just some of the numerous 5G services. The main concern here is that the physical infrastructure resources are limited and valuable, especially when data traffic demands from different operators increase. Therefore, efficient network sharing [1, 2] is considered as a conventional solution. Through network sharing, multiple operators can share infrastructure resources according to their agreed allocation plans. This approach can help an operator to reduce a significant amount of Capital Expenditure (CAPEX) and Operational Expenditure (OPEX).

As an evolution of network sharing, network slicing brings the flexibility and dynamicity of allocating the required and appropriate amount of physical resources to all service types mentioned above over the same physical infrastructure simultaneously. In fact, network slicing leverages the running of multiple logical networks on top of physical infrastructure. Network Functions (NFs) [3] are constructive operational components (physical networking devices) such as routers, firewalls, and load balancers that have specific functionalities in network infrastructure and hold distinct exterior interfaces for establishing communication between each other. An End-to-End (E2E) network slice [4] is a logical separated (isolated) network, created by chaining NFs, which delivers a particular network service according to QoS requirements via the underlying shared infrastructure in the (Radio) Access Network ((R)AN), Transport Network (TN), and Core Network (CN).

Network Function Virtualization (NFV), Software Defined Networking (SDN), and Cloud computing are considered as the three enabling technologies for implementing network slicing in 5G.

(i) NFV [3] is a network architecture framework where NFs that traditionally used dedicated vendor-specific hardware, so-called Physical NFs (PNFs), are now implemented in software. There are two leading solutions towards softwarized PNFs: (1) Virtualized NFs (VNFs) deployed on virtual machines and (2) Containerized NFs (CNFs) deployed on containers. These VNFs and CNFs, in turn, are then implemented in data centers or on cloud environments that run on top of general-purpose (vendor-neutral) hardware.

(ii) SDN [5] enables programmable and dynamic network configuration by separating the Control Plane (CP) and the Data Plane (DP), where a centralized entity (controller) in the CP configures the forwarding devices in the DP.

(iii) Cloud computing [6] deploys remote network resources in shared pools that can be administered over the Internet. Cloud computing is based on two principal orientations: (1) Cloud-based applications that point to relocating legacy applications, which were established on end-users’ devices or on the companies’ IT infrastructure, to cloud-based servers in order to deliver the applications over web browsers, and (2) Cloud-native applications, which refer to those applications that are particularly created and developed to employ the advantages of the cloud environment such as constant development, modularity, Application Programming Interface (API) integration, and scalability.

As mentioned, one of the 5G objectives is to implement ultralow latency services and to serve many devices with different amounts of computing resources. Multi-access Edge Computing (MEC) [7] is an enhancement of cloud computing that reduces the latency in a mobile network by pushing the processing and computing tasks to the edges of the network (such as base stations) to be closer to the devices with a limited amount of resources. This yields in facilitating the operation of delay-sensitive applications in such devices. These enabling technologies bring flexibility, programmability, and efficiency, but at the cost of higher complexity in operating and managing the 5G networks. The necessity for the Management and Network Orchestration (MANO) framework [8], which performs efficient resource management and orchestration between all network elements in the whole architecture, is undeniable.

Figure 1 illustrates a multi-layered architecture of network slice provisioning in 5G.

(i) In the first layer, there is a shared infrastructure layer, which includes heterogeneous hardware and software resources (base station, compute, storage, and networking) spanning over the RAN, TN, and CN domains to host multiple NFs in the second layer. In fact, these resources are sliced according to various service requirements and then will be allocated to different service types.

(ii) In the second layer, there are various NFs (PNFs, VNFs, and CNFs) with certain capabilities, belonging to different network domains. This layer encapsulates the essential configuration and managing operations of the NFs to provide different service types in the third layer.

(iii) In the third layer, according to service specifications, particular PNFs, VNFs, and/or CNFs (from the second layer) are chained in an explicit order with the appropriate amount of resources (from the first layer) to grant a distinctive service instance. The uniqueness of a service instance in this layer has a straightforward association with the business model, which indicates the reason for creating such a service that will be presented via a slice.

(iv) In the fourth layer, the launched service instances from the previous layer constitute E2E network slices. Hence, controlling and management policies on each of the network slices can be achieved independently via the NFV MANO framework.

(v) The NFV MANO framework is in charge of orchestrating all of the mentioned layers. Basically, the NFV MANO delivers all the monitoring, coordinating, controlling, and managing tasks of the available physical and virtual resources in order to maintain an efficient resource utilization between all types of NFs (PNF, VNF, and CNF) in the whole architecture. This results in producing network services that meet the specific service requirements over distinctive network slices.

[figure omitted; refer to PDF]

Since the introduction of the network slicing concepts and specification by the 3rd Generation Partnership Project (3GPP) [9], network slicing has attracted a lot of attention in the past years. Apart from the theoretical aspects of different ways of achieving the 5G objectives, research communities in academia and industry have followed practical approaches to examine different features of 5G and to evaluate the network performance under various use cases. In this regard, practical research works in the 5G area have developed prototype system implementations of individual parts of the mobile network architecture, which are known as research testbeds. Recently, even more complex network architectures have been deployed on such testbeds that support network slicing. Research testbeds grant the possibility to evaluate and enhance network performance. Besides, while research testbeds keep the cost of network deployment low, their functionalities, with a fair approximation, are comparable to real networks. Such testbeds can usually be implemented on standard PCs or servers with a not very high amount of resources and without the need of purchasing specialized hardware and software. Moreover, the availability of open-source software packages provides opportunities for creating innovative solutions towards 5G [10].

Deploying testbeds with network slicing capabilities is a challenging and error-prone task as it involves development of a network equipped with fundamental enabling technologies and the ability of programming and configuring the physical infrastructure. Depending on the specific service requirements, the physical and virtual components of a network slicing testbed must satisfy performance requests such as the amount of hardware and software resources (CPU, memory), reliability, and failure rates (dependability analysis) [11]. Nevertheless, the complexity of the testbed deployment process sometimes impacts the utilization of open-source solutions and standard PCs.

Although some excellent surveys have been done on different aspects of network slicing such as [4, 12, 13], just a few works focus on network slicing implementations, in particular, [14–17] elaborate collaborative 5G network slicing research projects and the proposed large-scale testbeds as outcomes of these projects. Reference [14] presents a broad study of five main large-scale SDN testbeds by explaining their design purposes, essential technologies, slicing capability, and use cases. Reference [15] investigates the necessity of network slicing for facilitating the implementation of Internet-of-Things (IoT) intelligent applications and smart services. Bonati et al. in [16] describe open source utilities, frameworks, and hardware components that can be used to instantiate softwarized 5G networks. Barakabitze et al. [17] provide a comprehensive review of 5G networks, a tutorial of the 5G network slicing technology enablers including SDN, NFV, MEC, Cloud/Fog computing, network hypervisors, virtual machines, and containers, as well as an overview of collaborative large 5G network slicing implementations. Nonetheless, there is a lack of a comprehensive survey that presents and evaluates small-scale state-of-the-art 5G network slicing implementations. Small-scale network slicing testbeds are important for the research community in several aspects. Small-scale testbeds require a lower deployment budget compared to large-scale testbeds. Besides, small-scale testbeds, with a compact softwarized version of the required entities, are more effortless to deploy and launch than large-scale ones. Further, due to such testbeds’ small scaling, they are more manageable to troubleshoot, and resolving possible issues is faster than large-scale testbeds with multiple involved entities. Eventually, although the number the practical use cases that can be investigated on small-scale testbeds is lower than large-scale testbeds and real networks, small-scale testbeds can afford similar analogous results to large-scale solutions. The aforementioned aspects motivate the work in this paper. We summarize our contributions as follows:

(i) We present the software packages and platforms that fit in the ETSI NFV MANO framework functional blocks for emulating RAN, CN domains, and MANO.

(ii) We define primary and secondary design criteria for network slicing testbeds.

(iii) We provide a detailed study of small-scale state-of-the-art testbeds for deploying network slicing. These testbeds are relatively easy to deploy and usually without requiring a huge financial investment, thus, suitable for university labs.

(iv) We further evaluate the testbeds according to the defined primary and secondary design criteria.

(v) We highlight the typical challenges while deploying such testbeds, and present possible solutions and directions for future work.

The rest of the paper is organized as follows. Section 2 explains the research methodology for this paper. Section 3, firstly, presents the ETSI NFV MANO framework along with possible open-source software solutions for each specific block in this framework, and secondly, outlines the desired criteria for designing network slicing testbeds in 5G. In Section 4, small-scale and cost-efficient state-of-the-art network slicing testbeds are detailed with their specific features. In Section 5, first, we compare the testbeds presented in the previous section, and then, we explain some of the main challenges while deploying such testbeds. Section 6 concludes the paper. Table 1 presents a list of the acronyms used in this paper.

Table 1

List of the used acronyms in this paper.

Abb.	Definition	Abb.	Definition	Abb.	Definition
5G	Fifth generation	4G	Forth generation	3GPP	3rd Generation Partnership Project
AI	Artificial Intelligence	AMF	Access and Mobility Management function	API	Application Programming Interface
BBU	Baseband Unit	CAI	Connected AI	CN	Core Network
CNF	Containerized NF	CP	Control Plane	CAPEX	Capital Expenditure
C-RAN	Cloud-RAN	DC	Data Center	DCAE	Data Collection Analytics & Events
DP	Data Plane	DSAF	Dynamic Slice Allocation Framework	E2E	End-to-End
eMBB	enhanced Mobile Broadband	EPC	Evolved Packet Core	ETSI	European Telecommunications Standards Institute
FCFSFA	First Come First Serve First Available	GUI	Graphical UI	HP LCVNF	High Priority LCVNF
IaaS	Infrastructure-as-a-Service	IIoT	Industrial IoT	IMS	IP Multimedia System
IoT	Internet-of-Things	KPI	Key Performance Indicator	KQI	Key Quality Indicators
L2TP	Layer-2 Tunneling Protocol	LCVNF	Latency Critical VNF	LP LCVNF	Low Piority LCVNF
LTE	Long-Term Evolution	LT VNF	Latency Tolerant VNF	MAC	Medium Access Control
MANO	Management and Network Orchestration	M-CORD	Mobile-Central Office Rearchitected as Datacenter	MEC	Multiaccess Edge Computing
ML	Machine Learning	MME	Mobility Management Entity	MTC	Machine Type Communication
NAS	Network Attached Storage	NBI	Northbound Interface	NF	Network Function
NR	New Radio	NFV	Network Function Virtualization	NFVI	NFV Infrastructure
NFVO	NFV Orchestrator	NIM	Network Infrastructure Manager	NSO	Network Service Orchestrator
OAI	OpenAirInterface	ODL	OpenDayLight	ODTN	Open and Disaggregated Transport Network
OMEC	Open Mobile Evolved Core	ONAP	Open Networking Automation Platform	ONOS	Open Network Operating System
OPEX	Operational Expenditure	OSM	Open Source MANO	OTG	OAI Traffic Generator
OvS	Open virtualSwitch	PaaS	Platform-as-a-Service	PNF	Physical NF
QoE	Quality of Experience	QoS	Quality of Service	RAN	Radio Access Network
RAT	Radio Access Technology	RLC	Radio Link Control	RO	Resource Orchestrator
RRC	Radio Resource Control	RRH	Remote Radio Head	RRM	Radio Resource Management
SA	Service Assurance	SaaS	Software-as-a-Service	SBI	Southbound Interface
SDN	Software Defined Networking	SD-RAN	Software Defined RAN	SEMIoTICS	Smart End-to-end Massive IoT Interoperability, Connectivity, and Security
SLA	Service Level Agreement	SlaaS	Slice-as-a-Service	SRS LTE	Software Radio Systems LTE
TN	Transport Network	UE	User Equipment	UI	User Interface
VDU	Virtual Deployment Unit	VES	Virtual Event Streaming	VIM	Virtualized Infrastructure Manager
VNF	Virtualized NF	VNFFG	VNF Forwarding Graph	VNFM	VNF Manager

2. Research Methodology

Network slicing has become a very hot topic both in academia and industry. This trend has resulted in research on various aspects of network slicing in 5G and a fast-growing number of publications. It is evident that only a portion of these publications introduces implementation solutions for network slicing, i.e., network slicing testbeds. In order to review such publications, we followed a research methodology and defined the procedure to search for related publications, the inclusion and exclusion criteria, and finally, the data collection method to extract pertinent publications. Inclusion and exclusion criteria are used to filter out nonrelevant collected papers. There is also an extra step for quality assessment regarding those publications that pass the inclusion criteria in the final systematization.

In the first step, we identified the databases for searching for potential relevant publications such as (1) ACM Digital Library, (2) IEEE Xplore, (3) Springer Link, (4) ScienceDirect, and (5) arXiv. Next, we started our searching process with relevant keywords to narrow down the selection area of the scientific publications into the network slicing field and, in particular, the deployment of network slicing. We employed some keywords such as <5G testbed>, , , and < network slicing framework>.

In the second step, we defined the inclusion criteria, for the publications resulted from the first step, as follows:

(i) Does the publication present a solution for network slicing deployment?

(ii) How is the solution provided? Which software and hardware components are used?

(iii) Is the presented testbed cost-efficient in terms of equipment and also human resources needed for the tested deployment?

We also defined the exclusion criteria as:

(i) A publication that introduces a large-scale testbed for network slicing, which is not possible to be implemented with a small budget.

(ii) A testbed, which is a result of national or international research projects, and those projects have been finished or are no longer active.

In the third step, the publications that meet the inclusion criteria are assessed for their quality. Following questions are applied for quality assessment:

(i) Can the presented testbed be used to investigate different typical use cases in the 5G network slicing, or is the solution just an initial implementation of network slicing with limited capacity for providing few realistic scenarios?

(ii) Does the publication include comprehensive information for the testbed architecture and deployment? Are there any extra and complementary sources included in the publication, that could help other researchers to deploy a similar testbed or a possible future extension?

In the end, we categorize the testbeds following the primary and secondary criteria defined in Section 3.

3. ETSI NFV MANO Framework and Design Criteria for Network Slicing Testbeds

3.1. ETSI NFV MANO Framework and Different Open-Source Software Solutions

ETSI introduces the NFV MANO architecture [8], which is comprised of three main functional blocks. Figure 2 illustrates these blocks with the reference points that connect them. This figure also summarizes some of the preeminent software solutions for each specific block. We focus on combining these solutions into the presented testbeds in Section 4 instead of explaining each one of these software modules individually.

(i) Virtualized Infrastructure Manager (VIM) performs controlling mechanisms for the NFV Infrastructure (NFVI) resources within an infrastructure provider. VIM is also responsible for receiving fault measurement and performance information of NFVI resources. Consequently, VIM can supervise NFVI resources allocation to the available VNFs. OpenStack [18] and OpenVIM [19] (for VNFs) and Kubernetes [20] (for CNFs) are possible solutions for the VIM section.

(ii) VNF Manager (VNFM) conducts one or several VNFs and does the lifecycle management of VNFs. VNF lifecycle management involves establishing/configuring, preserving, and terminating VNFs.

(iii) NFV Orchestrator (NFVO) implements resource and service orchestration in the network. NFVO is split up into Resource Orchestrator (RO) and Network Service Orchestrator (NSO). First, RO collects the current information regarding possible physical and virtual resources of NFVI through the VIM. Second, NSO applies a complete lifecycle management of multiple network services. In this way, NFVO keeps updating the information about the available VNFs running on top of NFVI. As a result, NFVO can initiate multiple network services. As part of the lifecycle management, NFVO can also terminate a network service whenever no longer a service request is received for that specific service. In several solutions, NFVO and VNFM are integrated into the MANO section. Open Source MANO (OSM) [21], Open Networking Automation Platform (ONAP) [22], OpenBaton [23], Cloudify [24], SONATA [25], and Katana Slice Manager [26] are considered as the leading integrated solutions for the MANO section. Note that OSM can also perform management and orchestration tasks on PNFs.

[figure omitted; refer to PDF]

Regarding VNFs/CNFs, several open-source software solutions can emulate RAN and CN domains:

(i) RAN domain is emulated with Software Radio Systems LTE (srsLTE) [27], OpenAirInterface (OAI) [10, 28], or O-RAN in its Bronze release [29, 30]

(ii) CN domain is realized with OAI, Open5GS (previously known as NextEPC) [31], Open Mobile Evolved Core (OMEC) [32], or free5GC [33]

Then, via chaining these VNFs/CNFs in the RAN and CN by the NFVO, distinguished service instances, so-called network slice subinstances, are formed. Some solutions for the TN domain, such as Open and Disaggregated Transport Network (ODTN) [34], utilize disaggregated optical equipment and open-source software to create a TN slice subinstance. An E2E network slice instance is created by pairing the definite RAN and CN slice subinstances via the TN slice subinstance [35].

3.2. Design Criteria for Network Slicing Testbeds

Multiple features should be taken into consideration when designing a comprehensive testbed of 5G and beyond networks. We identify the key design criteria for creating a 5G testbed that can emulate a real network’s major features and allow us to develop and test new algorithms. They are divided into two groups.

3.2.1. Primary Criteria

These attributes are fundamental for creating a network slicing testbed.

(i) Support of the main enabling technologies. The proposed testbed should be based on SDN, NFV, and cloud computing. Therefore, flexibility and dynamicity in the network are granted. SDN and NFV are complementary, hence, combined with cloud computing pave the way for the paradigms Software-as-a-Service (SaaS), Platform-as-a-Service (Paas), and Infrastructure-as-a-Service (IaaS) [3].

(ii) MANO equipped with dynamic monitoring capability. The testbed should support management, orchestration, programmability, and dynamic monitoring of different network functions, network services, and network slices. Therefore, the role of the MANO entity is essential that is the result of SDN/NFV utilization in the network architecture [36].

(iii) Multi-network domain with partial slicing support. A 5G testbed needs to provide connectivity across all network domains (air interface, (R)AN, TN, and CN) in order to show a practical ability that emulates the main functionalities of the 5G network. Multi-network domain support allows achieving E2E network slicing; however, it is worth noting that network slicing is a capability that can be implemented partially, and testbeds can deploy slicing only in one specific network domain.

(iv) Multi-tenancy support. 5G network is expected first to enable the coexistence of multiple tenants that demand the same network functionalities, and second to administrate the cooperation and interaction between them. This capability represents the so-called multi-tenancy environment, which means that a single instance of the software and its supporting infrastructure serves multiple tenants. Multi-tenancy is one of the main aspects of the 5G networks and should be supported in the testbed implementation.

3.2.2. Secondary Criteria

These attributes add extra features to a network slicing testbed apart from those in the primary group. Testbeds with these extra features broaden the research scope in the network slicing field.

(i) Multi-radio access technologies support. Different Radio Access Technologies (RATs) such as Long-Term Evolution (LTE), WiFi, and 5G New Radio (5G NR) should be deployed on the same platform [37]. Furthermore, Cloud-RAN (C-RAN), as a cloud computing-based architecture, brings cloudification benefits into the RAN domain. C-RAN consists of a cloud-Baseband Unit (BBU) pool and several Remote Radio Heads (RRHs). Since the 5G-RAN domain integrates the mentioned RATs with the corresponding frequency bands and provides them via the cloud, a solid platform should implement these capabilities.

(ii) End-to-End network slicing. The slicing capability should be expanded upon all network domains. An E2E network slice consists of several network slice subnet instances, each belonging to a particular network domain. Therefore, all network slice subnet instances should be provided and chained together to form an E2E network slice.

(iii) Cross-location support. One possible solution for experimenting with more realistic scenarios is deploying testbeds located in two geographical areas. In this case, RAN and CN domains are implemented and launched on two geographically separated infrastructures, and a backbone TN interconnects them. The cross-location capability becomes even more essential when evaluating network performance for providing delay-sensitive services in the 5G network. In real-world use cases, the RAN and CN domains are not necessarily located in the same geographical location, and, as mentioned, MEC is the technology answer to expedite the communication between the RAN and CN domains. Hence, cross-location testbeds facilitate measuring service delay and proposing possible solutions for those services that require low latency.

(iv) Machine Learning(ML)-enabled. 5G testbeds equipped with ML toolkits enable users to design, verify, and operate machine learning models via a supervised user interface. One possible outcome of using ML techniques in network slicing is to predict wireless channel behavior in the RAN domain. As a result, the available radio resources can be scheduled in an optimized way to maximize the resource usage per end-user or slice in the next transmissions.

(v) Open-source. Providing open-source 5G platforms with well-defined interfaces is considered as a huge advantage in deploying 5G testbeds because an open-source testbed can be deployed by other researchers to help foster research and innovation. It helps to reduce the hassle of setting up a working mobile network that on itself is a complicated and error-prone process

These design criteria explained above and outlined in Figure 3 are later used as an assessment for the state-of-the-art testbeds.

[figure omitted; refer to PDF]

The testbed is evaluated in three use cases: (1) real-time monitoring of resource utilization in disaster recovery by installing ShellMon client on IoT gateways; (2) hosting VNF as a docker container when a MEC node becomes overloaded by taking a self-triggered action to relocate to another MEC node (known as VNF migration); (3) modeling wireless channel and scheduling radio resources in RAN domain employing Matlab and using the testbed to perform SDN functionality.

4.20. MEC-Enabled 5G IoT Platform [69, 70]

This work (Figure 23) is a solid proposal for onboarding and scheduling aspects in VNF lifecycle management, and it presents a programmable and flexible MEC-enabled platform for IoT traffic. In this work, VNFs are categorized into Latency Critical VNFs (LCVNFs) and Latency Tolerant VNFs (LTVNFs). As a result, the applications are also divided into (1) real-time, provided by High Priority LCVNFs (HP LCVNFs), with resources in the MEC, (2) near-real-time, provided by Low Priority LCVNFs (LP LCVNFs), and (3) non-real-time, provided by LTVNFs. The LP LCVNFs and LTVNFs are deployed on the cloud instead of MEC since they do not provide real-time applications. The work improves the joint orchestration capability in the NFVO for the MEC and cloud resources for the mentioned VNF types via two methods: (1) an online placement scheme to deliver the required management tasks at the VNF level according to the data traffic and (2) a latency embedding structure that enables VNF migration and scalability to fulfill service requirements in real time. These two methods are accomplished by introducing (1) an algorithm for VNF Forwarding Graph (VNFFG) in chained VNFs for prioritizing delay-sensitive services and (2) a second algorithm for the real-time allocation of the MEC and cloud resources to the VNFs that takes into account scale-in/out features for diverse service requirements. The testbed is deployed on several physical servers for the functionalities of the core (cloud infrastructure and NFVO) and network edge (MEC) with lower computational resources compared to the core. OpenStack, as the VIM with its telemetry feature, conducts data collection, data monitoring for future resource utilization, and placement policy through its compute schedulers. Furthermore, the OSM provides the NFVO functionality in this testbed. There are some hypervisors located at the core and the edge that afford the computing tasks. The testbed is assessed by some autoscaling, VNF placement, and online VNF scheduling scenarios.

[figure omitted; refer to PDF]

4.21. CAI Testbed [71]

This testbed (Figure 24) offers a cost-efficient virtualized and orchestrated 5G mobile network equipped with containers and distinct fronthaul and backhaul topologies. The testbed mainly concentrates on integrating Artificial Intelligence (AI) using Kubeflow tool [72] to the management tasks in the 5G RAN and TN domains in order to optimize network performance. The testbed, called Connected AI (CAI), with the help of Kubernetes as a container-orchestrator, presents a mobile network composed of OvS devices, Ryu as SDN controller, and the OAI FlexRAN controller. CAI expedites the deployment of various network topologies on the fronthaul and backhaul by creating an emulated TN using Mininet. An AI agent takes various actions in the network by employing the information granted via Ryu and OAI FlexRAN controllers to feed ML models in order to implement several slice configurations. CAI builds a containerized implementation of OAI for C-RAN and Free5GC for the CN using Docker. The CAI testbed is evaluated via two use cases: (1) monitoring the amount of allocated radio resource blocks to different slice requests and (2) VNF placement in a cluster of containers by means of the AI agent.

[figure omitted; refer to PDF]

5. Discussion

5.1. Comparison between Different State-of-the-Art Network Slicing Testbeds

In this section, we compare the testbeds according to the design criteria for network slicing testbeds presented in Section 3. Table 2 summarizes the major characteristics of each testbed. The testbeds in Table 2 can be arranged into two categories.

(i) The first category comprises those testbeds that partially achieve some of the primary or secondary attributes of the design criteria for network slicing testbeds. In this regard, the testbeds in [38, 40, 45, 46, 50, 55, 56, 58, 60–62, 68–71] present network slicing in a particular network domain, and they do not realize a complete E2E network slicing. Reference [56] applies network slicing within multiple-VIMs (DCs); however, this implementation is limited to one network domain, and it does not present E2E network slicing, which crosses all network domains (RAN, TN, and CN). Other testbeds such as [47] implements E2E network slicing; however, it does not offer MANO capability, multi-RATs, and multi-tenancy facilities in the architecture. The platform in [41] applies light employment of MANO entity and E2E network slicing in its design.

(ii) The second category encompasses the implementations which satisfy all of the primary and the majority of secondary attributes from the design criteria explained in Section 3. The testbeds such as those in [43, 48, 49, 52, 53, 59, 64, 67] deliver E2E network slicing with MANO privilege in their architectures along with multi-tenancy and multi-RAT support. The testbeds in [59, 64] also incorporate ML-enabled capability in their architectures, and the testbed in [64] is open-source.

Table 2

Comparison of small-scale testbeds for network slicing in 5G, which ✓ denotes supported feature and ✗ denotes unsupported feature.

Testbed

SDN

NFV

Cloud comp.

Multi-domain

Multi-tenancy

MANO

Multi-RATs

E2E slicing

Cross-location

ML-enabled

Open-source

MANO type

1. 5G4IoT [38, 39]

✓

✓

✓

✓

✓

✗

✗

✗

✗

✗

✗

✗

2. 5GTN [40]

✓

✓

✓

✓

✗

✓

✓

✗

✗

✗

✓ (https://5gtn.fi/)

OSM, CloudBand

3. SEMIoTICS [41]

✓

✓

✓

✓

✓

✓

✗

✓

✗

✗

✓ (https://www.semiotics-project.eu/)

OpenStack tacker

4. Mosaic5G [43]

✓

✓

✓

✓

✓

✓

✓

✓

✗

✗

✓ (http://mosaic5g.io/)

JOX

5. Orion [45, 46]

✓

✓

✓

✓

✓

✓

✗

✗

✓

✓

✗

OSM and a customized orchestrator

6. 5G Testbed for NS [47]

✗

✓

✓

✓

✗

✗

✗

✓

✗

✗

✓ (https://github.com/ashxz47)

✗

7. POSENS [48, 49]

✓

✓

✓

✓

✓

✓

✓

✓

✗

✗

✓ (https://github.com/wnlUc3m)

Customized OSM

8. UPC testbed [50]

✓

✓

✗

✓

✓

✗

✓

✗

✗

✓

✗

✗

9. M-CORD based testbed [52, 53]

✓

✓

✓

✓

✓

✓

✓

✓

✗

✗

✓ (https://nick133371.github.io/)

XOS

10. NS for 5G IoT and eMBB [55]

✓

✓

✓

✓

✓

✗

✓

✗

✗

✗

✗

✗

11. Transformable resources slicing testbed [56]

✓

✓

✓

✗(E2E slice traverses over RAN, TN and CN.)

✓

✓

✗

✗

✗

✗

✗

VLSP, Kubernetes, and OpenStack

12. DSAF [58]

✗

✓

✓

✗

✗

✓

✗

✗

✗

✗

✗

Customized python-based orchestrator

13. SliceNet [59]

✓

✓

✓

✓

✓

✓

✓

✓

✗

✓

✗

OSM, OpenBaton

14. IqInf testbed [60]

✓

✓

✗

✓

✗

✗

✓

✗

✗

✗

✗

✗

15. Slice-aware SA testbed [61]

✓

✓

✗

✗

✗

✓

✗

✗

✗

✗

✗

Service assurance integrated with MANO

16. Simula [62, 63]

✓

✓

✓

✓

✓

✓

✓

✗

✗

✗

✓ (https://github.com/simula/5gvinni-oai-ns)

OSM

17. 5GIIK [64, 65]

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓ (https://bit.ly/3rgOgd6)

OSM

18. ONAP based testbed [67]

✓

✓

✓

✓

✓

✓

✓

✓

✗

✗

✗

ONAP

19. BlueArch [68]

✓

✓

✓

✓

✗

✓

✗

✗

✗

✗

✗

Open MANO, RIFT.io

20. MEC IoT platform [69, 70]

✓

✓

✓

✓

✗

✓

✓

✗

✗

✗

✗

OSM

21. CAI [71]

✓

✓

✓

✓

✗

✗

✓

✗

✗

✓

✓ (https://bit.ly/3tXErSX)

✗

5.2. Implementation Challenges for Deploying Network Slicing Testbeds

This section presents some of the current challenges for deploying small-scale network slicing testbeds and summarizes proposed solutions that can slightly mitigate these challenges.

(i) Monitoring frameworks for testbeds. 5G is expected to provide heterogeneous services with distinct QoS requirements via utilizing network slicing. In this regard, the dynamic monitoring of the launched services is essential. This becomes challenging when recognizing the issues of possible performance degradation of the services. In fact, the multi-layered architecture of the 5G network, as shown in Figure 1, causes such challenges. Intelligently identifying such issues requires analyzing multiple possible sources of the problem via particular frameworks to effectively monitor the deployment and performance of services. To partially address this problem, different types of monitoring capabilities are integrated in some of the elaborated testbeds. The testbeds in which OSM acts as an orchestrator in their architectures, such as [40, 45, 46, 59, 62, 64, 70], usually employ the interaction of the system monitoring module (MON) with a monitoring toolkit such as Prometheus [73] for collection of VNFs’ metrics and then utilize Grafana [74] to visualize the collected data. The testbeds with an ONAP orchestrator, such as [67], focus on SLA monitoring by exploiting Data Collection Analytics & Events (DCAE) and Virtual Event Streaming (VES) components. Reference [43] benefits from a monitoring application in the store component. The architecture in [61] offers monitoring functions in each layer of SA and also implements virtual monitoring agents or virtual probes at each point of presence to actively observe network services.

(ii) Cross-location testbeds. Launching testbeds over separate areas impacts the service performance because of delay, jitter, and packet loss. This issue becomes even more challenging when providing delay-sensitive services. Consequently, discovering techniques to enhance service performance in cross-location deployment is exceptionally important. As mentioned in Table 2, the testbeds in [46, 64] deploy a cross-location architecture for C-RAN (RAN and MEC) and CN on two separate cloud-based infrastructures. In these two testbeds, the MANO entity (OSM), with the help of an SDN-assist feature, partially considers this issue by implementing application-aware traffic flow strategies to mitigate the generated latency because of the cross-location architecture, which results in enhancing connection reliability [46, 75].

(iii) C-RAN deployment on testbeds. Implementing C-RAN architecture on a testbed using open-source software packages can be challenging since the interaction between BBU and RRHs entails extremely low latency. Some attempts, such as in [43, 46, 63] resolve this problem by deploying the BBU section with a combination of PNF and VNF. They split the protocol stack of BBU into two sections in their solution instead of launching the BBU completely in a cloud-based environment. In particular, the functionality of the PHY layer of the BBU is split into a lower-PHY as PNF (to run on a physical machine along with RRHs) and higher-PHY as VNF (to run on a cloud infrastructure). In this way, the communication between (lower-PHY layer of) BBU and RRHs fulfills the ultralow delay requirement while keeping the benefit of the cloud-based implementation of (higher-PHY layer of) BBU.

(iv) Resource management on testbeds with limited infrastructure capacity. Resource management is considered as another possible challenge while deploying testbeds on infrastructures with limited physical and/or virtual resources. Since diverse services demand various amounts of networking, computing, and storage resources, it is essential to identify optimized methods to allocate available resources to service instances. To deal with this issue, testbeds that adopt OpenStack as VIM in their infrastructures, such as references [38–41, 45, 46, 62, 64], can enable Telemetry Data Collection to gather event and data for utilization statistics of the infrastructure resources.

(v) Slice isolation on testbeds. The (intra/inter) slice isolation concept is a common concern while implementing network slicing, and it is not limited to research testbeds. It is worth stating that there are some endeavors to tackle the isolation issue. Testbeds, such as those in [43, 45, 46, 55, 59], which utilize FlexRAN in their architectures, present partial slice isolation in the RAN domain. The testbeds in [48, 49] perform isolation in the RAN domain by slicing the protocol stack down to RRC, RLC, and MAC layers. Nevertheless, introducing and realizing efficient and practical techniques to guarantee isolation in network slicing, especially in the RAN domain, is subject of future work. The work presented in [65] is one step towards providing traffic isolation and security isolation in network slicing.

6. Conclusion

Network slicing testbeds with dedicated management and orchestration entities endeavor to outline and emulate trial and real use cases to achieve network slicing. On this basis and according to pioneer technologies, this paper addresses the principal design criteria for creating and deploying experimental environments for network slicing in 5G. After that, the paper explains the most common small-scale state-of-the-art testbeds for network slicing with their characteristics. The presented testbeds are then reviewed and compared via the design criteria, followed by possible challenges while creating such experimental platforms. Although many efforts have been performed to create testbeds for examining and evaluating network performance under various use cases in network slicing, there are still open research questions in this field.

References

[1] A. Antonopoulos, "Bankruptcy problem in network sharing: fundamentals, applications and challenges," IEEE Wireless Communications, vol. 27 no. 4, pp. 81-87, DOI: 10.1109/MWC.001.1900414, 2020.

[2] P. Rost, A. Banchs, I. Berberana, M. Breitbach, M. Doll, H. Droste, C. Mannweiler, M. A. Puente, K. Samdanis, B. Sayadi, "Mobile network architecture evolution toward 5G," IEEE Communications Magazine, vol. 54 no. 5, pp. 84-91, DOI: 10.1109/MCOM.2016.7470940, 2016.

[3] R. Mijumbi, J. Serrat, J. Gorricho, N. Bouten, F. De Turck, R. Boutaba, "Network function virtualization: state-of-the-art and research challenges," IEEE Communications Surveys Tutorials, vol. 18 no. 1, pp. 236-262, DOI: 10.1109/COMST.2015.2477041, 2016.

[4] J. Ordonez-Lucena, P. Ameigeiras, D. Lopez, J. J. Ramos-Munoz, J. Lorca, J. Folgueira, "Network slicing for 5G with SDN/NFV: concepts, architectures, and challenges," IEEE Communications Magazine, vol. 55 no. 5, pp. 80-87, DOI: 10.1109/MCOM.2017.1600935, 2017.

[5] D. Kreutz, F. M. V. Ramos, P. E. Veríssimo, C. E. Rothenberg, S. Azodolmolky, S. Uhlig, "Software-defined networking: a comprehensive survey," Proceedings of the IEEE, vol. 103 no. 1, pp. 14-76, DOI: 10.1109/JPROC.2014.2371999, 2015.

[6] P. M. Mell, T. Grance, The NIST Definition of Cloud Computing,DOI: 10.6028/NIST.SP.800-145, 2011.

[7] Y. Mao, C. You, J. Zhang, K. Huang, K. B. Letaief, "A survey on mobile edge computing: the communication perspective," IEEE Communications Surveys Tutorials, vol. 19 no. 4, pp. 2322-2358, DOI: 10.1109/COMST.2017.2745201, 2017.

[8] ETSI, Network Functions Virtualisation (NFV): Architectural Framework, vol. 2 no. 2, 2013.

[9] 3GPP, Study on Management and Orchestration of Network Slicing for Next Generation Network, 2017.

[10] F. Kaltenberger, A. P. Silva, A. Gosain, L. Wang, T.-T. Nguyen, "Openairinterface: democratizing innovation in the 5G era," Computer Networks, vol. 176, article 107284,DOI: 10.1016/j.comnet.2020.107284, 2020.

[11] Stackify, "The Ultimate Guide to Performance Testing and Software Testing: Testing Types, Performance Testing Steps, Best Practices, and More," . April 2021, https://stackify.com/ultimate-guide-performance-testing-and-softwaretesting/

[12] I. Afolabi, T. Taleb, K. Samdanis, A. Ksentini, H. Flinck, "Network slicing and softwarization: a survey on principles, enabling technologies, and solutions," IEEE Communications Surveys Tutorials, vol. 20 no. 3, pp. 2429-2453, DOI: 10.1109/COMST.2018.2815638, 2018.

[13] X. Foukas, G. Patounas, A. Elmokashfi, M. K. Marina, "Network slicing in 5G: survey and challenges," IEEE Communications Magazine, vol. 55 no. 5, pp. 94-100, DOI: 10.1109/MCOM.2017.1600951, 2017.

[14] T. Huang, F. R. Yu, C. Zhang, J. Liu, J. Zhang, Y. Liu, "A survey on large-scale software defined networking (SDN) testbeds: approaches and challenges," IEEE Communications Surveys Tutorials, vol. 19 no. 2, pp. 891-917, DOI: 10.1109/COMST.2016.2630047, 2017.

[15] L. U. Khan, I. Yaqoob, N. H. Tran, Z. Han, C. S. Hong, "Network slicing: recent advances, taxonomy, requirements, and open research challenges," IEEE Access, vol. 8, pp. 36009-36028, DOI: 10.1109/access.2020.2975072, 2020.

[16] L. Bonati, M. Polese, S. D’Oro, S. Basagni, T. Melodia, "Open, programmable, and virtualized 5G networks: state-of-the-art and the road ahead," 2020. https://arxiv.org/abs/2005.10027

[17] A. A. Barakabitze, A. Ahmad, R. Mijumbi, A. Hines, "5G network slicing using SDN and NFV: a survey of taxonomy, architectures and future challenges," Computer Networks, vol. 167, article 106984, 2020. 10.1016/j.comnet.2019.106984

[18] OpenStack, "OpenStack The Most Widely Deployed Open Source Cloud Software in the World," . April 2021, https://www.openstack.org/

[19] OpenVIM, "Telefónica NFV reference lab," . April 2021, https://github.com/nfvlabs/openvim

[20] Kubernetes, "Kubernetes Production-Grade Container Orchestration," . April 2021, https://kubernetes.io

[21] Open Source MANO (OSM), "OSM Open Source NFV Management and Orchestration (MANO) software stack aligned with ETSI NFV," . April 2021, https://osm.etsi.org

[22] ONAP, "ONAP Open Networking Automation Platform," . April 2021, https://www.onap.org/

[23] OpenBaton, "OpenBaton An extensible and customizable NFV MANO-compliant framework," . April 2021, http://openbaton.org

[24] Cloudify, "Cloudify Multi Cloud Orchestration," . April 2021, https://cloudify.co/

[25] SONATA, "Sonata agile development. testing and orchestration of services in 5g virtualized networks," . April 2021, https://www.sonata-nfv.eu

[26] Katana Wiki Home, "MediaNetworks Laboratory," . April 2021, https://github.com/medianetlab/katana-slice_manager/wiki

[27] I. Gomez-Miguelez, A. Garcia-Saavedra, P. Sutton, P. Serrano, C. Cano, D. Leith, Srslte: An Opensource Platform for Lte Evolution and Experimentation, 2016.

[28] N. Nikaein, M. Marina, S. Manickam, A. Dawson, R. Knopp, C. Bonnet, "OpenAirInterface," English, Computer Communication Review, vol. 44 no. 5, pp. 33-38, DOI: 10.1145/2677046.2677053, 2014.

[29] O-RAN, "O-RAN Operator Defined Open and Intelligent Radio Access Networks," . April 2021, https://www.o-ran.org

[30] J. Murray, J. Huang, Blog, 2020. April 2021, https://www.o-ran.org/blog/2020/6/28/the-2nd-release-of-o-ran-software-bronze-addssupport-for-new-key-elements-of-the-o-ran-architecture-and-updates-to-align-with-the-latest-o-ranspecifications

[31] Open5GS, "Open5GS Open source project of 5GC and EPC," . April 2021, https://open5gs.org/

[32] TECHNICAL STEERING TEAM (TST), OMEC Open Mobile Evolved Core, 2020. April 2021, https://www.opennetworking.org/omec/

[33] free5GC, free5GC Link the World!, . April 2021, https://www.free5gc.org/

[34] Open Networking Foundation (ONF), "ODTN Open and Disaggregated Transport Network," . April 2021, https://www.opennetworking.org/odtn/

[35] D. Gligoroski, K. Kralevska, "Expanded combinatorial designs as tool to model network slicing in 5G," IEEE Access, vol. 7, pp. 54879-54887, DOI: 10.1109/ACCESS.2019.2913185, 2019.

[36] B. Sonkoly, J. Czentye, R. Szabo, D. Jocha, J. Elek, S. Sahhaf, W. Tavernier, F. Risso, "Multi-domain service orchestration over networks and clouds," ACM SIGCOMM Computer Communication Review, vol. 45 no. 4, pp. 377-378, DOI: 10.1145/2829988.2790041, 2015.

[37] C. Kilinc, M. Ericson, P. Rugeland, I. Da Silva, A. Zaidi, O. Aydin, V. Venkatasubramanian, M. C. Filippou, M. Mezzavilla, N. Kuruvatti, J. F. Monserrat, "5G multi-rat integration evaluations using a common PDCP layer," 2017 IEEE 85th Vehicular Technology Conference (VTC Spring), .

[38] B. Dzogovic, V. T. Do, B. Feng, T. van Do, "Building virtualized 5G networks using open source software," 2018 IEEE Symposium on Computer Applications Industrial Electronics (ISCAIE), pp. 360-366, .

[39] B. Dzogovic, B. Santos, V. T. Do, B. Feng, N. Jacot, T. Van Do, "Connecting remote eNodeB with containerized 5G C-RANs in OpenStack cloud," 2019 6th IEEE International Conference on Cyber Security and Cloud Computing (CSCloud)/ 2019 5th IEEE International Conference on Edge Computing and Scalable Cloud (EdgeCom), pp. 14-19, .

[40] M. Arif, O. Liinamaa, I. Ahmad, A. Pouttu, M. Ylianttila, "On the demonstration and evaluation of service-based slices in 5G test network using NFV," 2019 IEEE Wireless Communications and Networking Conference Workshop (WCNCW), .

[41] P. Mekikis, K. Ramantas, A. Antonopoulos, E. Kartsakli, L. Sanabria-Russo, J. Serra, D. Pubill, C. Verikoukis, "NFV-enabled experimental platform for 5G tactile internet support in industrial environments," IEEE Transactions on Industrial Informatics, vol. 16 no. 3, pp. 1895-1903, DOI: 10.1109/TII.2019.2917914, 2020.

[42] SEMIoTICS, "SEMIoTICS Smart End-to-end Massive IoT Interoperability, Connectivity and Security," . April 2021, https://www.semiotics-project.eu/

[43] N. Nikaein, C.-Y. Chang, K. Alexandris, "Mosaic5G," ACM SIGCOMM Computer Communication Review, vol. 48 no. 3, pp. 29-34, DOI: 10.1145/3276799.3276803, 2018.

[44] X. Foukas, N. Nikaein, M. M. Kassem, M. K. Marina, K. Kontovasilis, "Flexran: a flexible and programmable platform for software-defined radio access networks," Proceedings of the 12th International on Conference on Emerging Networking EXperiments and Technologies, ser. CoNEXT ‘16, pp. 427-441, DOI: 10.1145/2999572.2999599, .

[45] X. Foukas, M. K. Marina, K. P. Kontovasilis, "Orion: Ran slicing for a flexible and cost-effective multiservice mobile network architecture," In Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking (MobiCom ’17). Association for Computing Machinery, pp. 127-140, DOI: 10.1145/3117811.3117833, .

[46] X. Foukas, F. Sardis, F. Foster, M. K. Marina, M. A. Lema, M. Dohler, "Experience building a prototype 5G testbed," Proceedings of the Workshop on Experimentation and Measurements in 5G (EM-5G’18). Association for Computing Machinery, pp. 13-18, DOI: 10.1145/3286680.3286683, .

[47] A. Shorov, "5G testbed development for network slicing evaluation," 2019 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus), pp. 39-44, .

[48] G. Garcia-Aviles, M. Gramaglia, P. Serrano, A. Banchs, "POSENS: a practical open source solution for end-to-end network slicing," IEEE Wireless Communications, vol. 25 no. 5, pp. 30-37, DOI: 10.1109/MWC.2018.1800050, 2018.

[49] G. Garcia-Aviles, M. Gramaglia, P. Serrano, F. Gringoli, S. Fuente-Pascual, I. L. Pavon, "Experimenting with open source tools to deploy a multi-service and multi-slice mobile network," Computer Communications, vol. 150,DOI: 10.1016/j.comcom.2019.11.003, 2020.

[50] K. Koutlia, R. Ferrus, E. Coronado Calero, R. Riggio, F. Palacio, A. Umbert, J. Pérez-Romero, "Design and experimental validation of a software-defined radio access network testbed with slicing support," Wireless Communications and Mobile Computing, vol. 2019,DOI: 10.1155/2019/2361352, 2019.

[51] E. Coronado, S. N. Khan, R. Riggio, "5G-EmPOWER: a software-defined networking platform for 5G radio access networks," IEEE Transactions on Network and Service Management, vol. 16 no. 2, pp. 715-728, DOI: 10.1109/TNSM.2019.2908675, 2019.

[52] C. Huang, C. Ho, N. Nikaein, R. Cheng, "Design and prototype of a virtualized 5G infrastructure supporting network slicing," 2018 IEEE 23rd International Conference on Digital Signal Processing (DSP), .

[53] M. T. Abbas, T. A. Khan, A. Mahmood, J. J. D. Rivera, W. Song, "Introducing network slice management inside m-cord-based-5G framework," NOMS 2018-2018 IEEE/IFIP Network Operations and Management Symposium, .

[54] P. Berde, M. Gerola, J. Hart, Y. Higuchi, M. Kobayashi, T. Koide, B. Lantz, B. O’Connor, P. Radoslavov, W. Snow, G. Parulkar, "Onos: towards an open, distributed SDN OS," Proceedings of the Third Workshop on Hot Topics in Software Defined Networking, ser. HotSDN’14,DOI: 10.1145/2620728.2620744, .

[55] S. Costanzo, I. Fajjari, N. Aitsaadi, R. Langar, "Dynamic network slicing for 5G IoT and eMBB services: a new design with prototype and implementation results," 2018 3rd Cloudification of the Internet of Things (CIoT), .

[56] L. A. Freitas, V. G. Braga, S. L. Corrêa, L. Mamatas, C. E. Rothenberg, S. Clayman, K. V. Cardoso, "Slicing and allocation of transformable resources for the deployment of multiple virtualized infrastructure managers (vims)," 4th IEEE Conference on Network Softwarization and Workshops (NetSoft), pp. 424-432, .

[57] S. Clayman, "Network slicing supported by dynamic vim instantatiation," IETF 100, .

[58] D. Sattar, A. Matrawy, "Dsaf: dynamic slice allocation framework for 5G core network," 2019. https://arxiv.org/abs/1905.03873

[59] Q. Wang, J. Alcaraz-Calero, R. Ricart-Sanchez, M. B. Weiss, A. Gavras, N. Nikaein, X. Vasilakos, B. Giacomo, G. Pietro, M. Roddy, M. Healy, P. Walsh, T. Truong, Z. Bozakov, K. Koutsopoulos, P. Neves, C. Patachia-Sultanoiu, M. Iordache, E. Oproiu, I. G. B. Yahia, C. Angelo, C. Zotti, G. Celozzi, D. Morris, R. Figueiredo, D. Lorenz, S. Spadaro, G. Agapiou, A. Aleixo, C. Lomba, "Enable advanced QoS-aware network slicing in 5G networks for slice-based media use cases," IEEE Transactions on Broadcasting, vol. 65 no. 2, pp. 444-453, DOI: 10.1109/TBC.2019.2901402, 2019.

[60] K. Ramantas, E. Kartsakli, M. Irazabal, A. Antonopoulos, C. Verikoukis, "Implementation of an SDN-enabled 5G experimental platform for core and radio access network support," Interactive Mobile Communication, Technologies and Learning, pp. 791-796, DOI: 10.1007/978-3-319-75175-7_77, .

[61] J. Kim, M. Xie, "A study of slice-aware service assurance for network function virtualization," 2019 IEEE Conference on Network Softwarization (NetSoft), pp. 489-497, .

[62] T. Dreibholz, "Flexible 4G/5G testbed setup for mobile edge computing using OpenAirInterface and open source mano," 2020.

[63] A. F. Ocampo, T. Dreibholz, M.-r. Fida, A. Elmokashfi, H. Bryhni, Integrating Cloud-RAN with Packet Core as VNF Using Open Source MANO and OpenAirInterface, 2020.

[64] A. Esmaeily, K. Kralevska, D. Gligoroski, "A cloud-based SDN/NFV testbed for end-to-end network slicing in 4G/5G," 2020 6th IEEE Conference on Network Softwarization (NetSoft), pp. 29-35, .

[65] S. Haga, A. Esmaeily, K. Kralevska, D. Gligoroski, "5G network slice isolation with WireGuard and open source MANO: a VPNaaS Proof-of-Concept," 2020 IEEE Conference on Network Function Virtualization and Software Defined Networks, NFV-SDN 2020, pp. 181-187, DOI: 10.1109/NFV-SDN50289.2020.9289900, .

[66] J. A. Donenfeld, "WireGuard: Next Generation Kernel Network Tunnel," 24th Annual Network and Distributed System Security Symposium, NDSS,DOI: 10.14722/ndss.2017.23160, 2017.

[67] V. Q. Rodriguez, F. Guillemin, A. Boubendir, "5G e2e network slicing management with onap," 2020 23rd Conference on Innovation in Clouds, Internet and Networks and Workshops (ICIN), pp. 87-94, .

[68] S. Ghosh, "Bluearch - an implementation of 5G testbed," Journal of Communications, vol. 14, pp. 1110-1118, DOI: 10.12720/jcm.14.12.1110-1118, 2019.

[69] I. Sarrigiannis, E. Kartsakli, K. Ramantas, A. Antonopoulos, C. Verikoukis, "Application and network VNF migration in a MEC-enabled 5G architecture," 2018 IEEE 23rd International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD),DOI: 10.1109/CAMAD.2018.8514943, .

[70] I. Sarrigiannis, K. Ramantas, E. Kartsakli, P. Mekikis, A. Antonopoulos, C. Verikoukis, "Online VNF lifecycle management in an MEC-enabled 5G IoT architecture," IEEE Internet of Things Journal, vol. 7 no. 5, pp. 4183-4194, DOI: 10.1109/JIOT.2019.2944695, 2020.

[71] C. V. Nahum, L. D. N. M. Pinto, V. B. Tavares, P. Batista, S. Lins, N. Linder, A. Klautau, "Testbed for 5G connected artificial intelligence on virtualized networks," IEEE Access, vol. 8, pp. 223202-223213, DOI: 10.1109/ACCESS.2020.3043876, 2020.

[72] E. Bisong, "Kubeflow and kubeflow pipelines," Building Machine Learning and Deep Learning Models on Google Cloud Platform, pp. 671-685, DOI: 10.1007/978-1-4842-4470-8_46, 2019.

[73] Prometheus, "Prometheus From metrics to insight," . April 2021, https://prometheus.io/

[74] Grafana, "Grafana Labs Your observability wherever you need it," . April 2021, https://grafana.com/

[75] Technical Steering Committee, Osm Release Eight Notes, 2020. April 2021, https://osm.etsi.org/wikipub/images/5/56/OSM_Release_EIGHT_-_Release_Notes.pdf