oneM2M [9] is a global standard for Machine-to-Machine (M2M) communications and IoT platforms. It provides common service functions and uniform APIs via a common framework to support the development of IoT applications and services. The oneM2M architecture consists of two domains: Field Domain and Infrastructure Domain as depicted in Figure 1.

· Field Domain is the domain where sensors, actors, aggregators and gateways are deployed. It is the M2M area network and resides at the edge of the network.

· Infrastructure Domain is the M2M core network and is normally located in a cloud environment where IoT servers and applications reside.

There are 4 different kinds of nodes in the oneM2M network: Application Dedicated Node (ADN), Application Service Node (ASN), Middle Node (MN) and Infrastructure Node (IN). The Field Domain consists of ADN, ASN and MN while IN is located in the Infrastructure Domain. We only apply ASN and IN in our system. A Traffic Generator is designed as the ASN to simulate IoT devices sending traffic. Three different types of ASN devices including video camera, light pole and parking detector are simulated for three IoT services, respectively. On the other hand, virtualized IoT servers are used as INs in the cloud to receive application traffic from Traffic Generator.

We utilize OM2M, which is an open source implementation of oneM2M developed by LAAS-CNRS [10], as our IoT platform. OM2M provides RESTful APIs for creating and managing M2M resources. It offers Common Services Entities (CSEs) to implement horizontal M2M servers, gateways, and devices. We deploy OM2M IN CSEs as VNFs and use these VNFs to construct network slices in NFV to conduct our experiments with three different slicing systems.

NFV MANO (Management and Orchestration) [11] is a framework developed by ETSI (Europe Telecommunication Standards Institute) for the management and orchestration of all virtualized resources including compute, network, storage,

and VNFs. Whenever there is a demand for virtual resources, NFV MANO will coordinate, verify and authorize requests for these resources. It is also responsible for managing the life cycle of VNFs, such as instantiation, scaling, update and termination. In addition, it manages the policy of network services, the collection and transfer of performance measurement, and the allocation of resources related to infrastructures. The NFV MANO framework is adopted in our research to construct the network slicing environment.

As illustrated in Figure 2, NFV MANO consists of three major components.

· NFV Orchestrator (NFVO), which is in charge of the lifecycle of Network Services (NS) and responsible for onboarding Network Service Descriptor (NSD).

· VNF Manager (VNFM), which is responsible for the lifecycle management of the VNFs including VNF scaling out/in and their performance and fault management.

· Virtualized Infrastructure Manager (VIM), which is in charge of allocating and releasing NFV infrastructure (NFVI) including compute, storage, and network resources upon requests of the VNFM and NFVO.

Virtualized Network Functions (VNFs) are the software implementations of network functions such as firewall, load balancer. VNF can improve network scalability and agility and also make better use of network resources. Network Service (NS) consists of one or more network functions, which can include VNFs and Physical Network Functions (PNFs).

The NFV Infrastructure (NFVI) is composed of hardware and software resources that allow VNFs to be deployed, managed and executed. The physical infrastructure includes compute, storage and network. The virtualization layer decouples the virtual resources from the underlying hardware resources. Virtual resources are abstractions of the physical resources through the virtualization layer.

According to 3GPP [1], network slice is an end-to-end network architecture. It consists of multiple network slice subnets. Each network slice subnet represents different components in an end-to-end network such as access network, core network, transport network. NFV MANO, as the key enabling technology of network slicing, maps each network slice subnet to an NS in MANO. Each NS is defined by an NSD that consists of a set of attributes and several constituent descriptors including VNF descriptors (VNFDs), VL descriptors (VLDs), and VNFFG descriptors (VNFFGDs) [12]. The attributes of NSD are used to specify how NS instances should be deployed.

Network slicing enables the operator to divide a physical network into multiple virtual and logically independent end-to-end networks. Each network slice is tailored to fulfill different service requirements, such as delay, bandwidth, security, and reliability to cope with diverse network application scenarios [13]. Mobile operators can use network slicing to provide customized 5G networks to various vertical services based on specific needs of each [14].

We first explain the functional blocks of our three systems, then show the Network Service lifecycle management flows and the system workflows. The general architecture of our systems is illustrated in Figure 3 and our three systems are depicted in Figure 4.

Note that we design three new system components Master Node, Load Balancer and Traffic Generator on top of OpenStack and Tacker open sources in order to complete our systems.

· Master Node is incorporated in VNFM to monitor the CPU status of VNFs on each network slice in order to trigger scale-out or scale-in actions [24]. When the average CPU usage of VNFs exceeds an overload threshold, it will trigger the scale-out of the VNFs on the network slice. On the other hand, if the CPU usage is lower than an underload threshold, it will trigger the scale-in of the VNFs.

· Load Balancer is designed to fairly dispatch the incoming traffic to each VNF [25]. It is used only in the single slice scalable system (see Figure 4(c)) for distributing traffic. We use RabbitMQ [26], an open-source message-broker software implementing Advanced Message Queuing Protocol (AMQP) [27], based on Remote Procedure Call (RPC) to design our load balancer. In our system, Traffic Generator sends HTTP requests to Load Balancer and Load Balancer sends those requests to a load balancing queue. On the other side, each OM2M VNF as a server consumes those requests from the queue and replies a response back to Load Balancer.

· Traffic Generator is a multi-thread program that we design to simulate three types of IoT traffic. It can set the number for each kind of ASN devices and the frequency of sending data. Three types of IoT traffic generated include video, adaptive lighting and smart parking. Each traffic is a stream of HTTP requests.

1) Video: This is to simulate a security surveillance service enabled by the video camera. It provides monitoring services for road traffic and crowd movement. This service has the highest bandwidth demand among all three types of traffic.

2) Adaptive lighting: This is to simulate an adaptive lighting service by the smart street light pole that monitors weather conditions and adapts the brightness of street lighting accordingly based on the inputs from temperature, humidity, air pollution and light sensors.

3) Smart parking: This is to simulate a smart parking service that monitors the availability of parking spaces based on geomagnetic sensors embedded in parking areas. This service has the lowest bandwidth requirement among these three types of IoT services.

Figure 4 shows the differences of network slices in three systems under study. In our experiment, Traffic Generator will simulate the same three types of traffic. But the three systems would handle traffic in different ways.

As depicted in Figure 4(a), the single slice system architecture has only one network slice with one IoT platform as a VNF. The only one VNF needs to handle all types of IoT traffic. Then Figure 4(b) shows the multiple slicing system architecture that has three types of network slices. Each network slice is provisioned with a different customized bandwidth for dealing with a particular type of IoT service. At last, Figure 4(c) shows the design of the single slice scalable system architecture where only one network slice is provisioned but this slice supports scalability. Similar to the single slice system, its network slice needs to handle all three types of IoT traffic. But unlike the single slice system, it is capable of scaling its VNF to multiple instances and thus is equipped with a Load Balancer to evenly distribute IoT traffic to multiple VNF instances. However, because of Load Balancer, IoT traffic must go through an additional VNF, which may result in a longer response time than other systems.

Figure 5 shows the Network Service (NS) lifecycle management flows [28].

There are four phases to run a network slicing system.

In the preparation phase, we set up the environment by first registering an OpenStack VIM to the Tacker MANO system. This is done by setting up an account with authentication URL, username, password, project name and certificate. After registration is complete, we first onboard VNFD and then NSD to NFVO. NFVO will verify the integrity and authenticity of the NSD and check whether the VNFD required by the NSD exists. If acknowledged, it means that the NSD is successfully onboarded.

In the instantiation phase, NFVO receives a request to instantiate an NS. After receiving the request, NFVO validates the request and checks with VNFM whether the VNF instances required exist. If any of them does not exist, NFVO will request VNFM to instantiate it. NFVO will also check with VIM about the availability of required network resources and request the instantiation of virtual resources needed by the NS. Then NFVO will proceed to instantiate the NS by instantiating all its needed VNFs one by one. NFVO will also set up the connectivity among VNFs according to VLDs and VNFFGDs in the NSD. After deploying NS, we limit the bandwidth of each NS according to the attribute in NSD.

In the run-time phase, Master Node will keep monitoring the load status of each VNF and scale out/in VNFs according to the load. When there is only one VNF left, no scale-in action will be triggered.

In the termination phase, NFVO receives a request to terminate an NS instance and requests VNFM to terminate every required VNF in the NS if it is not used by another NS. VIM then deletes the compute, storage and network resources required by the VNFs. After NFVO acknowledges the completion of the Network Service termination, the lifecycle of NS ends in this phase.

The workflow of our system for scaling is shown in Figure 6. This scaling mechanism is only used by the single slice scalable system in our experiment. Note that Master Node will keep monitoring the CPU utilization of each VNF of the NS as Traffic Generator sends requests to the NS. When the average CPU utilization of all VNFs on the network slice exceeds the scale-out threshold, the scale-out action will be triggered. On the other hand, when it is lower than the

scale-in threshold, it will trigger the scale-in action. However, if there is only one VNF left for the NS, the scale-in action will not be triggered. In addition, if there is already an action being executed, the next scale-out or scale-in action will not be triggered until the previous one ends.

Our test environment consists of two servers. Both Tacker and OpenStack are running on these two servers configured as shown in Table 1. Table 2 shows the virtual resource allocation of each VNF in our environment.

In our experiment, we use Traffic Generator to simulate three types of traffic and send the HTTP requests to each VNF on the network slice. For the single slice system, we send the traffic to the OM2M IoT platform directly. For the multiple slicing system, each type of IoT traffic will be sent to the VNF on the corresponding network slice. For the single slice scalable system, we send all HTTP requests to the load balancer on the network slice first; the load balancer then distributes the requests to each VNF.

In order to meet the different requirements of each type of IoT requests we simulate, we set the required bandwidth of each network slice according to the setting in Table 3.

The required bandwidth is set to twice the expected traffic throughput to avoid temporary excessive traffic. For the single slice system and the single slice scalable system, we set the bandwidth to 1400 Kbps. For the multiple slicing system, the bandwidth limits are 1000 Kbps, 300 Kbps and 100 Kbps respectively with a total at 1400 Kbps that is the same as the other two systems for the fairness of comparison. The configuration of Traffic Generator in Table 4 would meet the expected traffic throughput in Table 3.

To test each system, there are three stages in our experiments. The whole process takes a total of 240 seconds. The payload size of each request sent by Traffic Generator will be based on the settings defined in Table 4. For the single slice scalable system, we follow the workflow for scaling as illustrated in Figure 6, and set the scale-out threshold to 50% and the scale-in threshold to 10% as in [29] [30].

· In the first stage, we will follow the configuration in Table 4 to send data to each system for 30 seconds. The requests of each application will be sent with different frequency and payload size.

· In the second stage, we triple the number of user threads as shown in Table 4 and send data for 120 seconds. We simulate higher traffic in this stage. For the scalable system, the scale-out action will be triggered.

· In the final stage, we return to the same configuration as the first stage for 90 seconds. During this stage, the scalable system will trigger the scale-in action back to its original status.

Table 5 shows the throughput of each application type in each stage. These three systems had a similar result of the throughput and reach the values of expected traffic throughput shown in Table 3.

Figure 7 shows the average response times of all applications in each system and Figure 8 shows the total CPU utilizations of three systems. Integrating the information from these two charts, we conclude that the multiple slicing system achieves the best response time at all stages. Also, its total CPU utilization is only slightly higher than the single slice system so it is an acceptable tradeoff. Overall, the performance of the multiple slicing system is better than those of the single slice systems whether it is equipped with scalability or not.

Comparing the single slice system with the single slice scalable system, it is clear that the average response time of the single slicing system with scalability is better than the one without scalability. In the first stage, since the single slice scalable system must go through Load Balancer which is an additional VNF, the response time is longer than the single slice system. However, when the traffic load increases in the second stage, the response time of the single slice scalable system is similar to that of the single slice system. Moreover, the result of the single slice scalable system is even better in the final stage. This is because the single slice scalable system can deal with increasing traffic loads better than the single slice system. However, the total CPU utilization of the single slice scalable

system is always higher than the other two systems due to the overhead of Load Balancer and scalability.

The average response times of each application type in the single slice system over all testing stages are shown in Figure 9. The results show that the response time of each application type under the same system architecture only differs a little bit. Similar results are also exhibited in the other two systems. Figure 10 shows that the multiple slicing system gets the lowest response time regardless of the type of applications. We can thus conclude that the system architectures instead of the application types have deeper impact on the system performance.

According to the above results, we speculate that implementing the horizontal scalability across the multiple slicing system may improve its performance and stability, which will be our future work. The research in [7] proved that network slicing can improve IoT/M2M scalability and fulfill different QoS requirements, which is also proved by our experimental results. However, the network slicing in [8] is based on SDN, while ours is based on NFV that has been adopted by the upcoming 3GPP 5G architecture.

Figure 11 shows the general architecture of the two slicing systems. Each functional block of our systems has been presented before. Also, Tacker which is utilized as NFVO and VNFM has been introduced in Section 3. The only difference is that we now use Kubernetes instead of OpenStack as VIM.

Traffic Generator which is our design will simulate the same three types of traffic as our experiments in OpenStack. As depicted in Figure 12(a), the Single Slice System has only one network slice with one IoT platform as a containerized VNF. The only one network slice will deal with all types of IoT traffic. Figure 12(b) shows the architecture of the multiple slicing system where each network slice would handle a specific type of IoT services.

Our test environment consists of two servers. Tacker and Kubernetes are each running on a server configured as shown in Table 6. Table 7 shows the virtual resource allocation of each containerized VNF in our environment.

In this experiment, we use Traffic Generator to simulate three types of traffic and send the HTTP requests to each containerized VNF on the network slice. For the single slice system, we send all three types of traffic to the OM2M IoT

platform directly. For the multiple slicing system, each type of IoT traffic will be sent to the containerized VNF on the corresponding network slice.

The configuration of Traffic Generator is the same as the one used for Open-Stack as shown in Table 4 of Section 4. It will generate the expected traffic throughput required for each application type in this experiment as shown in Table 8.

To test each system, there are three stages in our experiment. The whole process takes a total of 90 seconds. The payload size of each request sent by Traffic Generator will be based on the settings defined in Table 4 of Section 4.

· In the first stage, we will follow the configuration in Table 4 to send data to each system for 30 seconds. The requests of each application will be sent with different frequency and payload size.

· In the second stage, we triple the number of user threads as shown in Table 4 and send data for 30 seconds. We simulate higher traffic in this stage.

· In the final stage, we return to the same configuration as the one in the first stage for 30 seconds. During this stage, the systems will approach stability.

Because Kubernetes has its own scaling functions and policy for scalability, we only construct a single slice system and a multiple slicing system. Also, the time spent in the experiment for Kubernetes as VIM is different from the previous one for OpenStack as VIM. Since there was no need to do scalability, we shortened the total time of the experiment.

Figure 13 shows the average response times of all applications in each system. It shows that the multiple slicing system achieves better response time at all stages although the response times of the two systems are similar in the first stage and the third stage. When the traffic load increases in the second stage, the response time of the multiple slicing system is half of that of the single slice system.

This result shows that the multiple slicing system has better performance when it encounters high traffic.

On the other hand, as depicted in Figure 14 the CPU utilization of the multiple slicing system is always higher than that of the single slice system in all three stages because it has three network slices to handle different services. Note that the CPU utilizations of both systems are at their peak in the second stage due to the highest traffic load. On the other hand, because the first stage is the warm-up stage, the CPU utilizations of both systems are higher than those in the final stage when the systems become stable.

Integrating the information from these two charts, we conclude that the performance of the multiple slicing system is better in general as its total CPU utilization is only slightly higher than that of the single slice system but it can achieve faster response time than the single slice system.